Original: ALISTA: A NALYTIC W EIGHTS A RE A S G OOD A S L EARNED W EIGHTS IN LISTA
Revised: ALISTA: A NALYTIC W EIGHTS A RE A S G OOD A S L EARNED W EIGHTS IN LISTA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks based on unfolding an iterative algorithm, for example,
Revised: Deep neural networks based on unfolding an iterative algorithm, for example,

Original: LISTA (learned iterative shrinkage thresholding algorithm), have been an empirical success for sparse signal recovery.
Revised: LISTA (learned iterative shrinkage thresholding algorithm), have been an empirical success for sparse signal recovery.

Original: The weights of these neural networks are currently determined by data-driven “black-box” training.
Revised: The weights of these neural networks are currently determined by data-driven “black-box” training.

Original: In this work, we propose Analytic LISTA (ALISTA), where the weight matrix in LISTA is computed as the solution to a data-free optimization problem, leaving only the stepsize and threshold parameters to data-driven learning.
Revised: In this work, we propose Analytic LISTA (ALISTA), where the weight matrix in LISTA is computed as the solution to a data-free optimization problem, leaving only the stepsize and threshold parameters to data-driven learning.

Original: This signiﬁcantly simpliﬁes the training.
Revised: This signiﬁcantly simpliﬁes the training.

Original: Speciﬁcally, the data-free optimization problem is based on coherence minimization.
Revised: Speciﬁcally, the data-free optimization problem is based on coherence minimization.

Original: We show our ALISTA retains the optimal linear convergence proved in (Chen et al., 2018) and has a performance comparable to LISTA.
Revised: We show our ALISTA retains the optimal linear convergence proved in (Chen et al., 2018) and has a performance comparable to LISTA.

Original: Furthermore, we extend ALISTA to convolutional linear operators, again determined in a data-free manner.
Revised: Furthermore, we extend ALISTA to convolutional linear operators, again determined in a data-free manner.

Original: We also propose a feed-forward framework that combines the data-free optimization and ALISTA networks from end to end, one that can be jointly trained to gain robustness to small perturbations in the encoding model.
Revised: We also propose a feed-forward framework that combines the data-free optimization and ALISTA networks from end to end, one that can be jointly trained to gain robustness to small perturbations in the encoding model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Sparse vector recovery, or sparse coding, is a classical problem in source coding, signal reconstruction, pattern recognition and feature selection.
Revised: Sparse vector recovery, or sparse coding, is a classical problem in source coding, signal reconstruction, pattern recognition and feature selection.

Original: There is an unknown sparse vector x ∗ =
Revised: There is an unknown sparse vector x ∗ =

Original: [ x ∗ 1 , · · · , x ∗ M ] T ∈ R M .
Revised: [ x ∗ 1 , · · · , x ∗ M ] T ∈ R M .

Original: We observe it noisy linear measurements:
Revised: We observe it noisy linear measurements:

Original: [Equation]
Revised: [Equation]

Original: (1) where b ∈ R N , D
Revised: (1) where b ∈ R N , D

Original: =
Revised: =

Original: [ d 1 , · · · , d M ] ∈ R N × M is the dictionary , and ε ∈ R N is additive Gaussian white noise.
Revised: [ d 1 , · · · , d M ] ∈ R N × M is the dictionary , and ε ∈ R N is additive Gaussian white noise.

Original: For simplicity, each column of D , named as a dictionary kernel , is normalized, that is, (cid:107) d m (cid:107) 

Revised: For simplicity, each column of D , named as a dictionary kernel , is normalized, that is, (cid:107) d m (cid:107)

Original: = (cid:107) D : ,m (cid:107) 2 = 1 , m = 1 , 2 , · · · , M .
Revised: = (cid:107) D : ,m (cid:107) 2 = 1 , m = 1 , 2 , · · · , M .

Original: Typically, we have N (cid:28) M , so Equation (1) is an under-determined system.
Revised: Typically, we have N (cid:28) M , so Equation (1) is an under-determined system.

Original: However, when x ∗ is sufﬁciently sparse, it can be recovered faithfully.
Revised: However, when x ∗ is sufﬁciently sparse, it can be recovered faithfully.

Original: A popular approach is to solve the LASSO problem below (where λ is a scalar):
Revised: A popular approach is to solve the LASSO problem below (where λ is a scalar):

Original: [Equation]
Revised: [Equation]

Original: using iterative algorithms such as the iterative shrinkage thresholding algorithm (ISTA):
Revised: using iterative algorithms such as the iterative shrinkage thresholding algorithm (ISTA):

Original: [Equation]
Revised: [Equation]

Original: where η θ is the soft-thresholding function 1 and L is usually taken as the largest eigenvalue of D T D .
Revised: where η θ is the soft-thresholding function 1 and L is usually taken as the largest eigenvalue of D T D .

Original: Inspired by ISTA, the authors of (Gregor & LeCun, 2010) proposed to learn the weights in the matrices in ISTA rather than ﬁxing them.
Revised: Inspired by ISTA, the authors of (Gregor & LeCun, 2010) proposed to learn the weights in the matrices in ISTA rather than ﬁxing them.

Original: Their methods is called Learned ISTA (LISTA) and resembles a recurrent neural network (RNN).
Revised: Their methods is called Learned ISTA (LISTA) and resembles a recurrent neural network (RNN).

Original: If the iteration is truncated to K iterations, LISTA becomes a K -layer feed-forward neural network with side connections.
Revised: If the iteration is truncated to K iterations, LISTA becomes a K -layer feed-forward neural network with side connections.

Original: Speciﬁcally, LISTA is:
Revised: Speciﬁcally, LISTA is:

Original: [Equation]
Revised: [Equation]

Original: If we set W ( k )
Revised: If we set W ( k )

Original: [Equation]
Revised: [Equation]

Original: , then LISTA recovers ISTA.
Revised: , then LISTA recovers ISTA.

Original: Given each pair of sparse vector and its noisy measurements ( x ∗ , b ) , applying (4) from some initial point x (0) and using b as the input yields x ( k ) .
Revised: Given each pair of sparse vector and its noisy measurements ( x ∗ , b ) , applying (4) from some initial point x (0) and using b as the input yields x ( k ) .

Original: Our goal is to choose the parameters Θ = { W ( k ) 1 , W ( k ) w , θ ( k ) } k =0 , 1 ,...,K − 1 such that x ( k ) is close to x ∗ for all sparse x ∗ following some distribution P .
Revised: Our goal is to choose the parameters Θ = { W ( k ) 1 , W ( k ) w , θ ( k ) } k =0 , 1 ,...,K − 1 such that x ( k ) is close to x ∗ for all sparse x ∗ following some distribution P .

Original: Therefore, given the distribution P , all parameters in Θ are subject to learning:
Revised: Therefore, given the distribution P , all parameters in Θ are subject to learning:

Original: [Equation]
Revised: [Equation]

Original: This problem is approximately solved over a training dataset { ( x ∗ i , b i ) }
Revised: This problem is approximately solved over a training dataset { ( x ∗ i , b i ) }

Original: Ni =1 sampled from P .
Revised: Ni =1 sampled from P .

Original: Many empirical results, e.g., (Gregor & LeCun, 2010; Sprechmann et al., 2015; Wang et al., 2016), show that a trained K -layer LISTA (with K usually set to 10 ∼ 20 ) or its variants can generalize more than well to unseen samples ( x (cid:48) , b (cid:48) ) from the same distribution and recover x (cid:48) from b (cid:48) to the same accuracy within one or two order-of-magnitude fewer iterations than the original ISTA.
Revised: Many empirical results, e.g., (Gregor & LeCun, 2010; Sprechmann et al., 2015; Wang et al., 2016), show that a trained K -layer LISTA (with K usually set to 10 ∼ 20 ) or its variants can generalize more than well to unseen samples ( x (cid:48) , b (cid:48) ) from the same distribution and recover x (cid:48) from b (cid:48) to the same accuracy within one or two order-of-magnitude fewer iterations than the original ISTA.

Original: Additionally, the accuracies of the outputs { x ( k ) } of the layers k = 1 , .., K gradually improve.
Revised: Additionally, the accuracies of the outputs { x ( k ) } of the layers k = 1 , .., K gradually improve.

Original: However, such networks will generalize worse when the input deviates from the training distribution (e.g., when D varies), in contrast to the classical iterative algorithms such as ISTA that are trainingfree and thus agnostic to the input distribution.
Revised: However, such networks will generalize worse when the input deviates from the training distribution (e.g., when D varies), in contrast to the classical iterative algorithms such as ISTA that are trainingfree and thus agnostic to the input distribution.

Revised: The Analysis-Synthesis model (Rubinstein & Elad, 2014; Yang et al., 2016) could also be viewed as a special LISTA model with only one layer ( K = 1 ).

Original: More recently, the convolutional sparse coding (CSC), an extension of the sparse coding (1), gains increasingly attention in the machine learning area.
Revised: More recently, the convolutional sparse coding (CSC), an extension of the sparse coding (1), gains increasingly attention in the machine learning area.

Original: (Sreter & Giryes, 2018) showed that the CSC could be similarly approximated and accelerated by a LISTA-type feed-forward network.
Revised: (Sreter & Giryes, 2018) showed that the CSC could be similarly approximated and accelerated by a LISTA-type feed-forward network.

Original: (Tolooshams et al., 2018) designed a structure of sparse auto-encoder inspired by multi-layer CSC.
Revised: (Tolooshams et al., 2018) designed a structure of sparse auto-encoder inspired by multi-layer CSC.

Original: (Papyan et al., 2016; Sulam et al., 2017) also revealed CSC as a potentially useful tool for understanding general convolutional neural networks (CNNs).
Revised: (Papyan et al., 2016; Sulam et al., 2017) also revealed CSC as a potentially useful tool for understanding general convolutional neural networks (CNNs).

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Despite the empirical success (Sprechmann et al., 2015; Wang et al., 2016; Zhang & Ghanem, 2018;
Revised: Despite the empirical success (Sprechmann et al., 2015; Wang et al., 2016; Zhang & Ghanem, 2018;

Original: Zhou et al., 2018;
Revised: Zhou et al., 2018;

Original: Ito et al., 2018) in constructing fast trainable regressors for approximating iterative sparse optimization solvers, the theoretical understanding of such approximations remains limited.
Revised: Ito et al., 2018) in constructing fast trainable regressors for approximating iterative sparse optimization solvers, the theoretical understanding of such approximations remains limited.

Original: A handful of recent works have been investigating the theory of LISTA.
Revised: A handful of recent works have been investigating the theory of LISTA.

Original: (Moreau & Bruna, 2017) re-factorized the Gram matrix of dictionary, by trying to nearly diagonalize the Gram matrix with a basis, subject to a small (cid:96) 1 perturbation.
Revised: (Moreau & Bruna, 2017) re-factorized the Gram matrix of dictionary, by trying to nearly diagonalize the Gram matrix with a basis, subject to a small (cid:96) 1 perturbation.

Original: They thus re-parameterized LISTA a new factorized architecture that achieved similar acceleration gain to LISTA, hence ending up with an “indirect” proof.
Revised: They thus re-parameterized LISTA a new factorized architecture that achieved similar acceleration gain to LISTA, hence ending up with an “indirect” proof.

Original: They concluded that LISTA can converge faster than ISTA, but still sublinearly.
Revised: They concluded that LISTA can converge faster than ISTA, but still sublinearly.

Original: (Giryes et al., 2018) interpreted LISTA as a projected gradient descent descent (PGD) where the projection step was inaccurate, which enables a trade-off between approximation error and convergence speed.
Revised: (Giryes et al., 2018) interpreted LISTA as a projected gradient descent descent (PGD) where the projection step was inaccurate, which enables a trade-off between approximation error and convergence speed.

Original: The latest work (Chen et al., 2018) presented the more related results to ours: they introduced necessary conditions for the LISTA weight structure in order to achieve asymptotic linear convergence of
Revised: The latest work (Chen et al., 2018) presented the more related results to ours: they introduced necessary conditions for the LISTA weight structure in order to achieve asymptotic linear convergence of

Original: LISTA, which also proved to be a theoretical convergence rate upper bound .
Revised: LISTA, which also proved to be a theoretical convergence rate upper bound .

Original: They also introduced a thresholding scheme for practically improving the convergence speed.
Revised: They also introduced a thresholding scheme for practically improving the convergence speed.

Original: Note that, none of the above works extended their discussions to CSC and its similar LISTA-type architectures.
Revised: Note that, none of the above works extended their discussions to CSC and its similar LISTA-type architectures.

Original: Several other works examined the theoretical properties of some sibling architectures to LISTA.
Revised: Several other works examined the theoretical properties of some sibling architectures to LISTA.

Original: (Xin et al., 2016) studied the model proposed by (Wang et al., 2016), which unfolded/truncated the iterative hard thresholding (IHT) algorithm instead of ISTA, for approximating the solution to (cid:96) 0 -minimization.
Revised: (Xin et al., 2016) studied the model proposed by (Wang et al., 2016), which unfolded/truncated the iterative hard thresholding (IHT) algorithm instead of ISTA, for approximating the solution to (cid:96) 0 -minimization.

Original: They showed that the learnable fast regressor can obtained a transformed dictionary with improved restricted isometry property (RIP).
Revised: They showed that the learnable fast regressor can obtained a transformed dictionary with improved restricted isometry property (RIP).

Original: However, their discussions are not applicable to LISTA directly, although IHT is linearly convergent (Blumensath & Davies, 2009) under rather strong assumptions.
Revised: However, their discussions are not applicable to LISTA directly, although IHT is linearly convergent (Blumensath & Davies, 2009) under rather strong assumptions.

Original: Their discussions were also limited to linear sparse coding and resulting fully-connected networks only.
Revised: Their discussions were also limited to linear sparse coding and resulting fully-connected networks only.

Original: (Borgerding et al., 2017; Metzler et al., 2017) studied a similar learning-based model inspired from another LASSO solver, called approximated message passing (AMP).
Revised: (Borgerding et al., 2017; Metzler et al., 2017) studied a similar learning-based model inspired from another LASSO solver, called approximated message passing (AMP).

Original: (Borgerding et al., 2017) showed the MMSE-optimality of an AMP-inspired model, but not accompanied with any convergence rate result.
Revised: (Borgerding et al., 2017) showed the MMSE-optimality of an AMP-inspired model, but not accompanied with any convergence rate result.

Original: Also, the popular assumption in analyzing AMP algorithms (called “state evolution”) does not hold when analyzing ISTA.
Revised: Also, the popular assumption in analyzing AMP algorithms (called “state evolution”) does not hold when analyzing ISTA.

Original: 1. M OTIVATION AND C ONTRIBUTIONS
Revised: 1. M OTIVATION AND C ONTRIBUTIONS

Original: This paper presents multi-fold contributions in advancing the theoretical understanding of LISTA, beyond state-of-the-art results.
Revised: This paper presents multi-fold contributions in advancing the theoretical understanding of LISTA, beyond state-of-the-art results.

Original: Firstly, we show that the layer-wise weights in LISTA need not being learned from data .
Revised: Firstly, we show that the layer-wise weights in LISTA need not being learned from data .

Original: That is based on decoupling LISTA training into a data-free analytic optimization stage followed by a lighter-weight data-driven learning stage without compromising the optimal linear convergence rate proved in (Chen et al., 2018).
Revised: That is based on decoupling LISTA training into a data-free analytic optimization stage followed by a lighter-weight data-driven learning stage without compromising the optimal linear convergence rate proved in (Chen et al., 2018).

Original: We establish a minimum-coherence criterion between the desired LISTA weights and the dictionary D , which leads to an efﬁcient algorithm that can analytically solve the former from the latter, independent of the distribution of x .
Revised: We establish a minimum-coherence criterion between the desired LISTA weights and the dictionary D , which leads to an efﬁcient algorithm that can analytically solve the former from the latter, independent of the distribution of x .

Original: The data-driven training is then reduced to learning layer-wise step sizes and thresholds only, which will ﬁt the distribution of x .
Revised: The data-driven training is then reduced to learning layer-wise step sizes and thresholds only, which will ﬁt the distribution of x .

Original: The new scheme, called Analytic LISTA ( ALISTA ), provides important insights into the working mechanism of LISTA.
Revised: The new scheme, called Analytic LISTA ( ALISTA ), provides important insights into the working mechanism of LISTA.

Original: Experiments shows ALISTA to perform comparably with previous LISTA models (Gregor & LeCun, 2010; Chen et al., 2018) with much lighter-weight training.
Revised: Experiments shows ALISTA to perform comparably with previous LISTA models (Gregor & LeCun, 2010; Chen et al., 2018) with much lighter-weight training.

Original: Then, We extend the above discussions and conclusions to CSC, and introduce an efﬁcient algorithm to solve the convolutional version of coherence minimization.
Revised: Then, We extend the above discussions and conclusions to CSC, and introduce an efﬁcient algorithm to solve the convolutional version of coherence minimization.

Original: Further, we introduce a new robust LISTA learning scheme beneﬁting from the decoupled structure, by adding perturbations to D during training.
Revised: Further, we introduce a new robust LISTA learning scheme beneﬁting from the decoupled structure, by adding perturbations to D during training.

Original: The resulting model is shown to possess much stronger robustness when the input distribution varies, even when D changes to some extent, compared to classical LISTA models that learn to (over-)ﬁt one speciﬁc D .
Revised: The resulting model is shown to possess much stronger robustness when the input distribution varies, even when D changes to some extent, compared to classical LISTA models that learn to (over-)ﬁt one speciﬁc D .

Original:  A NALYTIC LISTA: CALCULATING WEIGHTS WITHOUT TRAINING
Revised:  A NALYTIC LISTA: CALCULATING WEIGHTS WITHOUT TRAINING

Original: We theoretically analyze the LISTA form deﬁned in (Chen et al., 2018):
Revised: We theoretically analyze the LISTA-CPSS model deﬁned in (Chen et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: (6) where W ( k ) =
Revised: (6) where W ( k ) =

Original: [ w ( k ) 1 , · · · , w ( k ) M ] ∈ R N × M is a linear operator with the same dimensionality with
Revised: [ w ( k ) 1 , · · · , w ( k ) M ] ∈ R N × M is a linear operator with the same dimensionality with

Original: D , x ( k ) = (cid:2) x ( k ) 1 , · · · , x ( k ) M (cid:3) is the k th layer node.
Revised: D , x ( k ) = (cid:2) x ( k ) 1 , · · · , x ( k ) M (cid:3) is the k th layer node.

Original: In (6), Θ = { W ( k ) , θ ( k ) } k are parameters to train.
Revised: In (6), Θ = { W ( k ) , θ ( k ) } k are parameters to train.

Original: Model (6) can be derived from (4) with W ( k )
Revised: Model (6) can be derived from (4) with W ( k )

Original: [Equation]
Revised: [Equation]

Original: ) showed that (6) has the same representation capability with (4) on the sparse recovery problem, with a speciﬁcally light weight structure.
Revised: ) showed that (6) has the same representation capability with (4) on the sparse recovery problem, with a speciﬁcally light weight structure.

Original: Our theoretical analysis will further deﬁne and establish properties of “good” parameters Θ in (6), and then discuss how to analytically computer those good parameters rather than relying solely on black-box training.
Revised: Our theoretical analysis will further deﬁne and establish properties of “good” parameters Θ in (6), and then discuss how to analytically computer those good parameters rather than relying solely on black-box training.

Original: In this way, the LISTA model could be further signiﬁcantly simpliﬁed, without little performance loss.
Revised: In this way, the LISTA model could be further signiﬁcantly simpliﬁed, with little performance loss.

Original: The proofs of all the theorems in this paper are provided in the appendix.
Revised: The proofs of all the theorems in this paper are provided in the appendix.

Original: 2.  R ECOVERY E RROR U PPER BOUND
Revised:  R ECOVERY E RROR U PPER BOUND

Original: We start with an assumption on the “ground truth” signal x ∗ and the noise ε .
Revised: We start with an assumption on the “ground truth” signal x ∗ and the noise ε .

Original: Assumption 1 (Basic assumptions) .
Revised: Assumption 1 (Basic assumptions) .

Original: Signal x ∗ is sampled from the following set:
Revised: Signal x ∗ is sampled from the following set:

Original: [Equation]
Revised: [Equation]

Original: In other words, x ∗ is bounded and s -sparse 2 ( s ≥ 2 ).
Revised: In other words, x ∗ is bounded and s -sparse 2 ( s ≥ 2 ).

Original: Furthermore, we assume ε = 0 .
Revised: Furthermore, we assume ε = 0 .

Original: The zero-noise assumption is for simplicity of the proofs.
Revised: The zero-noise assumption is for simplicity of the proofs.

Original: Our experiments will show that our models are robust to noisy cases.
Revised: Our experiments will show that our models are robust to noisy cases.

Revised: The mutual coherence of the dictionary D is a signiﬁcant concept in compressive sensing (Donoho & Elad, 2003; Elad, 2007; Lu et al., 2018).

Revised: A dictionary with small coherence possesses better sparse recovery performance.

Revised: Motivated by this point, we introduce the following deﬁnition.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given D ∈ R N × M with each of its column normalized, a weight matrix W is “good” if it belongs to
Revised: Given D ∈ R N × M with each of its column normalized, we deﬁne the generalized mutual coherence:

Original: [Equation]
Revised: [Equation]

Original: Taking a W ∈ W ( D ) [Equation]
Revised: Additionally, We deﬁne W ( D ) =

Original: we deﬁne the generalized mutual coherence:
Revised: (cid:8) W ∈ R N × M : W attains the inﬁmum given (8) (cid:9) .

Original: [Equation]
Revised: A weight matrix W is “good” if W ∈ W ( D ) . In the above deﬁnition, problem (8) is feasible and attainable, i.e., W ( D ) (cid:54) = ∅ , which was proven in Lemma 1 of (Chen et al., 2018).

Original: Theorem 1 (Recovery error upper bound) .
Revised: Theorem 1 (Recovery error upper bound) .

Original: Take any x ∗ ∈ X ( B, s ) , any W ∈ W ( D ) , and any sequence γ ( k ) ∈ (0 , 2 2˜ µs − ˜ µ +1 ) .
Revised: Take any x ∗ ∈ X ( B, s ) , any W ∈ W ( D ) , and any sequence γ ( k ) ∈ (0 , 2 2˜ µs − ˜ µ +1 ) .

Original: Using them, deﬁne the parameters { W ( k ) , θ ( k ) } :
Revised: Using them, deﬁne the parameters { W ( k ) , θ ( k [Equation]

Original: [Equation]
Revised: [Equation]

Original: while the sequence { x ( k ) ( x ∗ ) } ∞ k =1 is generated by (6) using the above parameters and x (0) = 0 .
Revised: while the sequence { x ( k ) ( x ∗ ) } ∞ k =1 is generated by (6) using the above parameters and x (0) = 0 .

Original: (Each x ( k ) ( x ∗ ) depends only on θ ( k − 1) , θ ( k − 2) , . . .
Revised: (Each x ( k ) ( x ∗ ) depends only on θ ( k − 1) , θ ( k − 2) , . . .

Original: and deﬁnes θ ( k ) .)
Revised: and deﬁnes θ ( k ) .)

Original: Let Assumption 1 hold with any B > 0 and s < (1 + 1 / ˜ µ ) / 2 .
Revised: Let Assumption 1 hold with any B > 0 and s < (1 + 1 / ˜ µ ) / 2 .

Original: Then, we have support( x ( k ) ( x ∗ ))
Revised: Then, we have support( x ( k ) ( x ∗ ))

Original: ⊂ S ,
Revised: ⊂ S ,

Original: [Equation]
Revised: [Equation]

Original: where S is the support of x ∗ and c ( k )
Revised: where S is the support of x ∗ and c ( k )

Original: = − log (cid:0) (2˜ µs − ˜ µ ) γ ( k ) + | 1 − γ ( k ) | (cid:1) is a positive constant.
Revised: = − log (cid:0) (2˜ µs − ˜ [Equation] [Equation] is a positive constant.

Original: In Theorem 1, Eqn.
Revised: In Theorem 1, Eqn.

Original: (10) deﬁnes the properties of “good” parameters:
Revised: (9) deﬁnes the properties of “good” parameters:

Original: • The weights W ( k ) can be separated as the product of a scalar γ ( k ) and a matrix W independent of layer index k . • W has small coherence with D .
Revised: • The weights W ( k ) can be separated as the product of a scalar γ ( k ) and a matrix W independent of layer index k , where W has small coherence with D .

Original: • γ ( k ) is bounded in an interval.
Revised: • γ ( k ) is bounded in an interval.

Original: • θ ( k ) /γ
Revised: • θ ( k ) /γ

Original: ( k ) is proportional to the (cid:96) 1 error of the output of the k th layer.
Revised: ( k ) is proportional to the (cid:96) 1 error of the output of the k th layer.

Original: The factor c ( k ) takes the maximum at γ ( k )
Revised: The factor c ( k ) takes the maximum at γ ( k )

Original: = 1 .
Revised: = 1 .

Original: If γ ( k ) ≡ 1 , the recovery error converges to zero in a linear rate (Chen et al., 2018):
Revised: If γ ( k ) ≡ 1 , the recovery error converges to zero in a linear rate (Chen et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: where c = − log(2˜ µs − ˜ µ ) ≥ c ( k ) .
Revised: where c = − log(2˜ µs − ˜ µ ) ≥ c ( k ) .

Original: Although γ ( k ) ≡ 1 gives the optimal theoretical upper bound if there are inﬁnitely many layers k = 0 , 1 , 2 , · · · , it is not the optimal choice for ﬁnite k .
Revised: Although γ ( k ) ≡ 1 gives the optimal theoretical upper bound if there are inﬁnitely many layers k = 0 , 1 , 2 , · · · , it is not the optimal choice for ﬁnite k .

Original: Practically, there are ﬁnitely many layers and γ ( k ) obtained by learning is bounded in an interval.
Revised: Practically, there are ﬁnitely many layers and γ ( k ) obtained by learning is bounded in an interval.

Original: 2. R ECOVERY E RROR L OWER BOUND
Revised: 2. R ECOVERY E RROR L OWER BOUND

Original: In this subsection, we introduce a lower bound of the recovery error of LISTA, which illustrates that the parameters analytically given by (10) are optimal in the convergence order (linear) too.
Revised: In this subsection, we introduce a lower bound of the recovery error of LISTA, which illustrates that the parameters analytically given by (9) are optimal in the convergence order (linear).

Original: Assumption 2.
Revised: Assumption 2.

Original: The signal x ∗ is a random variable following the distribution P X .
Revised: The signal x ∗ is a random variable following the distribution P X .

Original: Let S = support ( x ∗ ) .
Revised: Let S = support ( x ∗ ) .

Original: P X satisﬁes: 2 ≤ | S | ≤ s ; S uniformly distributes on the whole index set; nonzero part x ∗ S satisﬁes the uniform distribution with bound B : | x ∗ i | ≤ B, ∀ i ∈ S .
Revised: P X satisﬁes: 2 ≤ | S | ≤ s ; S uniformly distributes on the whole index set; nonzero part x ∗ S satisﬁes the uniform distribution with bound B : | x ∗ i | ≤ B, ∀ i ∈ S .

Original: Moreover, the observation noise ε = 0 .
Revised: Moreover, the observation noise ε = 0 .

Revised: Theorem 1 tells that an ideal weight W ∈ W ( D ) satisﬁes I − W T D ≈ 0 .

Revised: But this cannot be met exactly in the overcomplete D case, i.e., N < M .

Revised: Deﬁnition 2 deﬁnes the set of matrices W such that W T D is bounded away from the identity I .

Revised: In Appendix D, we discuss the feasibility of (11).

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Given D ∈ R N × M , s ≥ 2 , ¯ σ min > 0 , we deﬁne a set that W ( k ) are chosen from:
Revised: Given D ∈ R N × M , s ≥ 2 , ¯ σ min > 0 , we deﬁne a set that W ( k ) are chosen from:

Original: [Equation]Theorem 1 tells that an ideal weight W ∈ W ( D ) satisﬁes I − W T D ≈ 0 . But this cannot be met exactly in the overcomplete D case, i.e., N < M .
Revised: [Equation]

Original:  Deﬁnition 2 addresses this point.
Revised: Based on Deﬁnition 2, we deﬁne a set that Θ = { W ( k ) , θ ( k ) } ∞ k =0 are chosen from:

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Based on Deﬁnition 2, we deﬁne a set that Θ = { W ( k ) , θ ( k ) } ∞ k =0 are chosen from:
Revised: Let { x ( k ) ( x ∗ ) } ∞ k =1 be generated by (6) with { W ( k ) , θ ( k ) } ∞ k =0 and x (0) = 0 . Then we deﬁne T as the set of parameters that guarantee there is no false positive in x ( k ) :

Original: [Equation]
Revised: [Equation]

Original: The conclusion (11) demonstrates that T is nonempty because “support ( x ( k ) ( x ∗ )) ⊂ S ”(no false positive) is satisﬁed as long as θ ( k ) large enough.
Revised: The conclusion (10) demonstrates that T is nonempty because “support ( x ( k ) ( x ∗ )) ⊂ S ” is satisﬁed as long as θ ( k − 1) large enough.

Original: Actually, T contains almost all “good” parameters because considerable false positives lead to large recovery errors.
Revised: Actually, T contains almost all “good” parameters because considerable false positives lead to large recovery errors.

Original: With T deﬁned, we have:
Revised: With T deﬁned, we have:

Original: Theorem 2 (Recovery error lower bound) .
Revised: Theorem 2 (Recovery error lower bound) .

Original: Let the sequence { x ( k ) ( x ∗ ) } ∞ k =1 be generated by (6) with { W ( k ) , θ ( k ) }
Revised: Let the sequence { x ( k ) ( x ∗ ) } ∞ k =1 be generated by (6) with { W ( k ) , θ ( k ) }

Original: ∞ k =0 and x (0) = 0 .
Revised: ∞ k =0 and x (0) = 0 .

Original: Under Assumption 2, for all parameters { W ( k ) , θ ( k ) } ∞ k =0 ∈ T and any sufﬁcient small (cid:15) > 0 , we have 
Revised: Under Assumption 2, for all parameters { W ( k ) , θ ( k ) } ∞ k =0 ∈ T and any sufﬁcient small (cid:15) > 0 , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: with probability at least (1 − (cid:15)s 3 / 2 − (cid:15) 2 ) , where ¯ c = s log(3) − log(¯ σ min ) .
Revised: with probability at least (1 − (cid:15)s 3 / 2 − (cid:15) 2 ) , where ¯ c = s log(3) − log(¯ σ min ) .

Original: This theorem illustrates that, with high probability, the convergence rate of LISTA cannot be faster than a linear rate.
Revised: This theorem illustrates that, with high probability, the convergence rate of LISTA cannot be faster than a linear rate.

Original: Thus, the parameters given in (10), that leads to the linear convergence if γ k is bounded within an interval near 1 , are optimal with respect to the order of convergence of LISTA.
Revised: Thus, the parameters given in (9), that leads to the linear convergence if γ k is bounded within an interval near 1 , are optimal with respect to the order of convergence of LISTA.

Original: 2. A NALYTIC LISTA: LESS PARAMETERS TO LEARN
Revised: 2. A NALYTIC LISTA: LESS PARAMETERS TO LEARN

Original: Following Theorems 1 and 2, we set W ( k ) =
Revised: Following Theorems 1 and 2, we set W ( k ) =

Original: γ ( k ) W , where γ ( k ) is a scalar, and propose Tied
Revised: γ ( k ) W , where γ ( k ) is a scalar, and propose Tied

Original: LISTA  :
Revised: LISTA (TiLISTA) :

Original: [Equation]
Revised: [Equation]

Original: (15) where Θ = (cid:8) { γ ( k ) } k , { θ ( k ) } k , W (cid:9) are parameters to train.
Revised: (14) where Θ = (cid:8) { γ ( k ) } k , { θ ( k ) } k , W (cid:9) are parameters to train.

Original: The matrix W is tied over all the layers.
Revised: The matrix W is tied over all the layers.

Original: Further, we notice that the selection of W from W ( D ) depends on D only.
Revised: Further, we notice that the selection of W from W ( D ) depends on D only.

Original: Hence we propose the analytic LISTA (ALISTA) that decomposes tied-LISTA into two stages:
Revised: Hence we propose the analytic LISTA (ALISTA) that decomposes tied-LISTA into two stages:

Original: [Equation]
Revised: [Equation]

Original: where ˜ W is pre-computed by solving the following problem ( Stage 1 ):
Revised: where ˜ W is pre-computed by solving the following problem ( Stage 1 ):

Original: [Equation]
Revised: [Equation]

Original: Then with ˜ W ﬁxed, { γ ( k ) , θ ( k ) } k in (16) are learned from end to end ( Stage 2 ).
Revised: Then with ˜ W ﬁxed, { γ ( k ) , θ ( k ) } k in (15) are learned from end to end ( Stage 2 ).

Original: (17) reformulates (8) to minimizing the Frobenius norm of W T D (a quadratic objective), over linear constraints.
Revised: (16) reformulates (8) to minimizing the Frobenius norm of W T D (a quadratic objective), over linear constraints.

Original: This is a standard convex quadratic program, which is easy to solve.
Revised: This is a standard convex quadratic program, which is easier to solve than to solve (8) directly. [Table] [Equation] [Table]

Revised: [Equation]

Original: C ONVOLUTIONAL A NALYTIC LISTA
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  We extend the analytic LISTA to the convolutional case in this section, starting from discussing the convolutional sparse coding (CSC).
Revised: [Equation] [Table]  C ONVOLUTIONAL A NALYTIC LISTA We extend the analytic LISTA to the convolutional case in this section, starting from discussing the convolutional sparse coding (CSC).

Original: Many works studied CSC and proposed efﬁcient algorithms for that (Bristow et al., 2013; Heide et al., 2015; Wohlberg, 2014; 2016; Papyan et al., 2017; GarciaCardona & Wohlberg, 2018;
Revised: Many works studied CSC and proposed efﬁcient algorithms for that (Bristow et al., 2013; Heide et al., 2015; Wohlberg, 2014; 2016; Papyan et al., 2017; GarciaCardona & Wohlberg, 2018;

Original: Wang et al., 2018; Liu et al., 2017; 2018).
Revised: Wang et al., 2018; Liu et al., 2017; 2018).

Original: In CSC, the general linear transform is replaced by convolutions in order to learn spatially invariant features:
Revised: In CSC, the general linear transform is replaced by convolutions in order to learn spatially invariant features:

Original: [Equation]
Revised: [Equation]

Original: where each d m is a dictionary kernel (or ﬁlter).
Revised: where each d m is a dictionary kernel (or ﬁlter).

Original: { d m } Mm =1 is the dictionary of ﬁlters, M denotes the number of ﬁlters.
Revised: { d m } Mm =1 is the dictionary of ﬁlters, M denotes the number of ﬁlters.

Original: { x ∗ m } Mm =1 is the set of coefﬁcient maps that are assumed to have sparse structure, and ∗ is the convolution operator.
Revised: { x ∗ m } Mm =1 is the set of coefﬁcient maps that are assumed to have sparse structure, and ∗ is the convolution operator.

Original: Now we consider 2D convolution. Let b ∈ R N 2 , d m ∈
Revised: Now we consider 2D convolution and take 3 b ∈ R N 2 , d m ∈

Original: R D 2 , x m ∈ R ( N + D − 1) 2 .
Revised: R D 2 , x m ∈ R ( N + D − 1) 2 .

Original: Equation (18) is pointwisely deﬁned as 3 :
Revised: Equation (17) is pointwisely deﬁned as 4 :

Original: [Equation]
Revised: [Equation]

Original: We vectorize b , d m , x m and let d = [ d 1 , · · · , d M ] T and x = [ x 1 , · · · , x M ] T . Then the above transform can be written as
Revised: We concatenate d m s and x m s: d = [ d 1 , · · · , d M ] T , x = [ x 1 , · · · , x M ] T , and rewrite (18) as:

Original: [Equation]
Revised: [Equation]

Original: where  D N conv ( d ) =
Revised: where the matrix D N conv ( d ) =

Original: [ D N conv , 1 ( d 1 ) , · · · , D N conv ,M ( d M )] ∈ R N 2 × ( N + D − 1) 2 M is a matrix depending on the signal size N and the dictionary d .
Revised: [ D N conv , 1 ( d 1 ) , · · · , D N conv ,M ( d M )] ∈ R N 2 × ( N + D − 1) 2 M , depending on the signal size N and the dictionary d , is deﬁned in detail in (48) in Appendix C.2.

Original: From (18), the convolutional LISTA becomes a natural extension of the fully-connected LISTA (6):
Revised: From (17), the convolutional LISTA becomes a natural extension of the fully-connected LISTA (6):

Original: [Equation]
Revised: [Equation]

Original: [Equation] where { w ( k ) m } Mm =1 share the same sizes with { d m } Mm =1 .
Revised: where { w ( k ) m } Mm =1 share the same sizes with { d m } Mm =1 and ( · ) (cid:48) means a 180 rotation of the ﬁlter (Chalasani et al., 2013).

Original: Let w ( k ) =
Revised: We concatenate the ﬁlters together: w ( k ) =

Original: [ w ( k ) 1 , · · · , w ( k ) M ] T ∈ R D 2 M .
Revised: [ w ( k ) 1 , · · · , w ( k ) M ] T ∈ R D 2 M .

Original: Parameters to train are Θ = { w ( k ) , θ ( k ) } k .
Revised: Parameters to train are Θ = { w ( k ) , θ ( k ) } k .

Original: Let W N conv ( w ( k ) ) be the matrix induced by dictionary w ( k ) with the same dimensionality as
Revised: Let W N conv ( w ( k ) ) be the matrix induced by dictionary w ( k ) with the same dimensionality as

Original: D N conv ( d ) .
Revised: D N conv ( d ) .

Original: Since convolution can be written as a matrix form (20), (21) is equivalent to
Revised: Since convolution can be written as a matrix form (19), (20) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Then by just substituting D , W ( k ) with D N conv ( d ) , W N conv ( w ( k ) ) respectively, Theorems 1 andcan be applied to the convolutional LISTA.
Revised: Then by just substituting D , W ( k ) with D N conv ( d ) , W N conv ( w ( k ) ) respectively, Theorems 1 andcan be applied to the convolutional LISTA.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let D = D N conv ( d ) and W ( k )
Revised: Let D = D N conv ( d ) and W ( k )

Original: =
Revised: =

Original: W N conv ( w ( k ) ) .
Revised: W N conv ( w ( k ) ) .

Original: With Assumption 1 and other settings the same with those in Theorem 1, (11) holds.
Revised: With Assumption 1 and other settings the same with those in Theorem 1, (10) holds.

Original: With Assumption 2 and other settings the same with those in Theorem 2, (14) holds.
Revised: With Assumption 2 and other settings the same with those in Theorem 2, (13) holds.

Original: Similar to the fully connected case (16), based on the results in Proposition 1, we should set w ( k ) m
Revised: Similar to the fully connected case (15), based on the results in Proposition 1, we should set w ( k ) m

Original: = γ ( k ) m ˜ w m , m = 1 , 2 , · · · , M , where ˜ w =
Revised: = γ ( k ) m ˜ w m , m = 1 , 2 , · · · , M , where ˜ w =

Original: [ ˜ w 1 , · · · , ˜ w M ] T is chosen from
Revised: [ ˜ w 1 , · · · , ˜ w M ] T is chosen from

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: However, (23) is not as efﬁcient to solve as (17).
Revised: However, (22) is not as efﬁcient to solve as (16).

Original: To see that, matrices D N conv ( d ) and W N conv ( w ) are both of size N 2 × ( N + D − 1) 2 M , the coherence matrix (cid:0) W N conv ( w ) (cid:1) T D N conv ( d ) is thus of size ( N + D − 1) 2 M ×
Revised: To see that, matrices D N conv ( d ) and W N conv ( w ) are both of size N 2 × ( N + D − 1) 2 M , the coherence matrix (cid:0) W N conv ( w ) (cid:1) T D N conv ( d ) is thus of size ( N + D − 1) 2 M ×

Original: ( N + D − 1) 2 M .
Revised: ( N + D − 1) 2 M .

Original: In the typical application setting of CSC, b is usually an image rather than a small patch.
Revised: In the typical application setting of CSC, b is usually an image rather than a small patch.

Original: For example, if the image size is 100 × 100 , dictionary size is 7 × 7 × 64 ,
Revised: For example, if the image size is 100 × 100 , dictionary size is 7 × 7 × 64 ,

Original: N = 100 , D = 7 , M = 64 , then ( N + D − 1) 2 M × ( N + D − 1) 2 M ≈ 5 × 10 11 .
Revised: N = 100 , D = 7 , M = 64 , then ( N + D − 1) [Equation] M × [Equation]

Original: 3. C ALCULATING CONVOLUTIONAL WEIGHTS ANALYTICALLY AND EFFICIENTLY
Revised: N [Equation] M [Equation]

Original: {+5 × 
 [Equation] 3. C ALCULATING CONVOLUTIONAL WEIGHTS ANALYTICALLY AND EFFICIENTLY+} To overcome the computational challenge of solving (23), we exploit the following circular convolution as an efﬁcient approximated way:
Revised: {+5 × 
 [Equation] 3. C ALCULATING CONVOLUTIONAL WEIGHTS ANALYTICALLY AND EFFICIENTLY+} To overcome the computational challenge of solving (22), we exploit the following circular convolution as an efﬁcient approximation:

Original: [Equation]
Revised: [Equation]

Original: where b ∈ R N 2 , d m ∈ R D 2 , x m ∈ R N 2 .
Revised: where b ∈ R N 2 , d m ∈ R D 2 , x m ∈ R N 2 .

Original: The corresponding matrix form of (24) is:
Revised: Similar to (18), we rewrite (23) in a compact way:

Original: [Equation]
Revised: [Equation]

Original: where D N conv ( d ) : R N 2 M → R N 2 is a matrix depending on the signal size N and the dictionary d .
Revised: where D N cir ( d ) : R N 2 M → R N 2 is a matrix depending on the signal size N and the dictionary d .

Original: Then the coherence minimization with the circular convolution is given by
Revised: Then the coherence minimization with the circular convolution is given by

Original: [Equation]
Revised: [Equation]

Original:  The following theorem motivates us to use the solution to (25) to approximate that of (23).
Revised: [Equation] The following theorem motivates us to use the solution to (24) to approximate that of (22).

Original: Theorem 3.
Revised: Theorem 3.

Original: The solution sets of (23) and (25) satisfy the following properties:
Revised: The solution sets of (22) and (24) satisfy the following properties:

Original: [Equation]
Revised: [Equation]

Original: If at least one of the matrices { D 2 D − 1 cir , 1 , · · · , D 2 D − 1 cir ,M } is non-singular, W 2 D − 1 cir
Revised: If at least one of the matrices { D 2 D − 1 cir , 1 , · · · , D 2 D − 1 cir ,M } is non-singular, W 2 D − 1 cir

Original: involves
Revised: involves

Original: only a unique element.
Revised: only a unique element.

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: The solution set W N cir is not related with the image size N as long as N ≥ 2 D − 1 , thus one can deal with a much smaller-size problem (let N = 2 D − 1 ).
Revised: The solution set W N cir is not related with the image size N as long as N ≥ 2 D − 1 , thus one can deal with a much smaller-size problem (let N = 2 D − 1 ).

Original: Further, (26) indicates that as N gets (much) larger than D , the boundary condition becomes less important.
Revised: Further, (25) indicates that as N gets (much) larger than D , the boundary condition becomes less important.

Original: Thus, one can use W 2 D − 1 cir to approximate W N conv .
Revised: Thus, one can use W 2 D − 1 cir to approximate W N conv .

Original: In Appendix D, we introduce the algorithm details of solving (25).
Revised: In Appendix E, we introduce the algorithm details of solving (24).

Original: Based on Proposition 1 and Theorem 3, we obtain the convolutional ALISTA :
Revised: Based on Proposition 1 and Theorem 3, we obtain the convolutional ALISTA :

Original: [Equation]
Revised: [Equation]

Original: where ˜ w =
Revised: where ˜ w =

Original: [ ˜ w 1 , · · · , ˜ w M ] T ∈ W 2 D − 1 cir and Θ = {{ γ ( k ) m } m,k , { θ ( k ) } k } are the parameters to train.
Revised: [ ˜ w 1 , · · · , ˜ w M ] T ∈ W 2 D − 1 cir and Θ = {{ γ ( k ) m } m,k , { θ ( k ) } k } are the parameters to train.

Original: (27) is a simpliﬁed form, compared to the empirically unfolded CSC model recently proposed in (Sreter & Giryes, 2018)
Revised: (26) is a simpliﬁed form, compared to the empirically unfolded CSC model recently proposed in (Sreter & Giryes, 2018)

Original:  R OBUST ALISTA TO M ODEL P ERTURBATION
Revised:  R OBUST ALISTA TO M ODEL P ERTURBATION

Original: Many applications, such as often found in surveillance video scenarios (Zhao et al., 2011; Han et al., 2013), can be formulated as sparse coding models whose dictionaries are subject to small dynamic perturbations (e.g, slowly varied over time).
Revised: Many applications, such as often found in surveillance video scenarios (Zhao et al., 2011; Han et al., 2013), can be formulated as sparse coding models whose dictionaries are subject to small dynamic perturbations (e.g, slowly varied over time).

Original: Speciﬁcally, the linear system model (1) may have uncertain D : ˜ D
Revised: Speciﬁcally, the linear system model (1) may have uncertain D : ˜ D

Original: = D + ε D , where ε D is some small stochastic perturbation.
Revised: = D + ε D , where ε D is some small stochastic perturbation.

Original: Classical
Revised: Classical

Original: LISTA entangles the learning of all its parameters, and the trained model is tied to one static D .
Revised: LISTA entangles the learning of all its parameters, and the trained model is tied to one static D .

Original: The important contribution of ALISTA is to decompose ﬁtting W w.r.t.
Revised: The important contribution of ALISTA is to decompose ﬁtting W w.r.t.

Original: D , from adapting other parameters { γ ( k ) , θ ( k ) } k to training data.
Revised: D , from adapting other parameters { γ ( k ) , θ ( k ) } k to training data.

Original: In this section, we develop a robust variant of ALISTA that is a fast regressor not only for a given
Revised: In this section, we develop a robust variant of ALISTA that is a fast regressor not only for a given

Original: D , but all its randomly perturbations ˜
Revised: D , but all its randomly perturbations ˜

Original: D to some extent.
Revised: D to some extent.

Original: Up to our best knowledge, this approach is new.
Revised: Up to our best knowledge, this approach is new.

Original: Robust ALISTA can be sketched as the following empirical routine (at each iteration):
Revised: Robust ALISTA can be sketched as the following empirical routine (at each iteration):

Original: • Sample a perturbed dictionary ˜ D .
Revised: • Sample a perturbed dictionary ˜ D .

Original: Sample x and ε to generate b w.r.t.
Revised: Sample x and ε to generate b w.r.t.

Original: ˜ D .
Revised: ˜ D .

Original: • Apply Stage 1 of ALISTA w.r.t.
Revised: • Apply Stage 1 of ALISTA w.r.t.

Original: ˜ D and obtain ˜ W ; however, instead of an iterative minimization algorithm, we use a neural network that unfolds that algorithm to produce ˜ W .
Revised: ˜ D and obtain ˜ W ; however, instead of an iterative minimization algorithm, we use a neural network that unfolds that algorithm to produce ˜ W .

Original: • Apply Stage 2 of ALISTA w.r.t.
Revised: • Apply Stage 2 of ALISTA w.r.t.

Original: ˜ W , D , x , and b to obtain { γ ( k ) , θ ( k ) } k .
Revised: ˜ W , D , x , and b to obtain { γ ( k ) , θ ( k ) } k .

Original: In Robust ALISTA above, ˜ D becomes a part of the data for training the neural network that generates ˜
Revised: In Robust ALISTA above, ˜ D becomes a part of the data for training the neural network that generates ˜

Original: W .
Revised: W .

Original: This neural network is faster to apply than the minimization algorithm.
Revised: This neural network is faster to apply than the minimization algorithm.

Original: One might attempt to use ˜ D in the last step, rather than D , but ˜ D makes training less stable, potentially because of larger weight variations between training iterations due to the random perturbations in ˜ D .
Revised: One might attempt to use ˜ D in the last step, rather than D , but ˜ D makes training less stable, potentially because of larger weight variations between training iterations due to the random perturbations in ˜ D .

Original: We observe that using D stabilizes training better and empirically achieves a good prediction.
Revised: We observe that using D stabilizes training better and empirically achieves a good prediction.

Original: More details of training Robust ALISTA are given in Appendix E.
Revised: More details of training Robust ALISTA are given in Appendix F.

Original:  N UMERICAL R ESULTS
Revised:  N UMERICAL R ESULTS

Original: In this section, we conduct extensive experiments on both synthesized and real data to demonstrate:
Revised: In this section, we conduct extensive experiments on both synthesized and real data to demonstrate:

Original: • We experimentally validate Theorems 1 and 2, and show that ALISTA is as effective as classical LISTA (Gregor & LeCun, 2010; Chen et al.
Revised: • We experimentally validate Theorems 1 and 2, and show that ALISTA is as effective as classical LISTA (Gregor & LeCun, 2010; Chen et al.

Original: , 2018)but is much easier to train.
Revised: , 2018)but is much easier to train.

Original: • Similar conclusions can be drawn for convolutional analytic LISTA.
Revised: • Similar conclusions can be drawn for convolutional analytic LISTA.

Original: • The robust analytic LISTA further shows remarkable robustness in sparse code prediction, given that D is randomly perturbed within some extent.
Revised: • The robust analytic LISTA further shows remarkable robustness in sparse code prediction, given that D is randomly perturbed within some extent.

Original: Notation For brevity, we let LISTA denote the vanilla LISTA model (4) in (Gregor & LeCun, 2010); LISTA-CPSS refers to the lately-proposed fast LISTA variant (Chen et al., 2018) with weight coupling and support selection; TiLISTA is the tied LISTA (15); and ALISTA is our proposed Analytic
Revised: Notation For brevity, we let LISTA denote the vanilla LISTA model (4) in (Gregor & LeCun, 2010); LISTA-CPSS refers to the lately-proposed fast LISTA variant (Chen et al., 2018) with weight coupling and support selection; TiLISTA is the tied LISTA (14); and ALISTA is our proposed Analytic

Original: LISTA (16).
Revised: LISTA (15).

Original: If the model is for convolutional case, then we add “ Conv ” as the preﬁx for model name, such as “ Conv ALISTA ” that represents the convolutional analytic LISTA.
Revised: If the model is for convolutional case, then we add “ Conv ” as the preﬁx for model name, such as “ Conv ALISTA ” that represents the convolutional analytic LISTA.

Original: 5. V ALIDATION OF T HEOREMS 1 AND 2 (A NALYTIC LISTA)
Revised: 5. V ALIDATION OF T HEOREMS 1 AND 2 (A NALYTIC LISTA)

Original: We follow the same N = 250 , M = 500 setting as (Chen et al., 2018) by default.
Revised: We follow the same N = 250 , M = 500 setting as (Chen et al., 2018) by default.

Original: We sample the entries of D i.i.d.
Revised: We sample the entries of D i.i.d.

Original: from the standard Gaussian distribution, D ij ∼ N (0 , 1 /N )
Revised: from the standard Gaussian distribution, D ij ∼ N (0 , 1 /N )

Original: and then normalize its columns to have the unit (cid:96) 2 norm.
Revised: and then normalize its columns to have the unit (cid:96) 2 norm.

Original: We ﬁx a dictionary D in this section.
Revised: We ﬁx a dictionary D in this section.

Original: To generate sparse vectors x ∗ , we decide each of its entry to be non-zero following the Bernoulli distribution with p b = 0 .
Revised: To generate sparse vectors x ∗ , we decide each of its entry to be non-zero following the Bernoulli distribution with p b = 0 .

Original: The values of the non-zero entries are sampled from the standard Gaussian distribution.
Revised: The values of the non-zero entries are sampled from the standard Gaussian distribution.

Original: A test set of 1000 samples generated in the above manner is ﬁxed for all tests in our simulations.
Revised: A test set of 1000 samples generated in the above manner is ﬁxed for all tests in our simulations.

Original: The analytic weight W that we use in the ALISTA is obtained by solving (17).
Revised: The analytic weight W that we use in the ALISTA is obtained by solving (16).

Original: All networks used (vanilla LISTA, LISTA-CPSS, TiLISTA and ALISTA) have the same number of 16 layers.
Revised: All networks used (vanilla LISTA, LISTA-CPSS, TiLISTA and ALISTA) have the same number of 16 layers.

Original: We also include two classical iterative solvers: ISTA and FISTA.
Revised: We also include two classical iterative solvers: ISTA and FISTA.

Original: We train the networks with four different levels of noises: SNR (Signal-to-Noise Ratio) = 20 , 30 , 40 , and ∞ .
Revised: We train the networks with four different levels of noises: SNR (Signal-to-Noise Ratio) = 20 , 30 , 40 , and ∞ .

Original: While our theory mainly discussed the noise-free case (SNR = ∞ ), we hope to empirically study the algorithm performance under noise too.
Revised: While our theory mainly discussed the noise-free case (SNR = ∞ ), we hope to empirically study the algorithm performance under noise too.

Original: As shown in Figure 1, the x-axes denotesis the indices of layers for the networks, or the number of iterations for the iterative algorithms.
Revised: As shown in Figure 1, the x-axes denotes the indices of layers for the networks, or the number of iterations for the iterative algorithms.

Original: The y-axes represent the NMSE (Normalized Mean Squared Error) in the decibel (dB) unit:
Revised: The y-axes represent the NMSE (Normalized Mean Squared Error) in the decibel (dB) unit:

Original: [Equation]
Revised: [Equation]

Original: where x ∗ is the ground truth and ˆ x is the estimated one
Revised: where x ∗ is the ground truth and ˆ x is the estimated one

Original: [Figure]
Revised: [Figure]

Original: In Figure 1 (a) noise-less case, all four learned models apparently converge much faster than two iterative solvers (ISTA/FISTA curves almost overlap in this y-scale, at the small number of iterations).
Revised: In Figure 1 (a) noise-less case, all four learned models apparently converge much faster than two iterative solvers (ISTA/FISTA curves almost overlap in this y-scale, at the small number of iterations).

Original: Among the four networks, classical-LISTA is inferior to the other three by an obvious margin.
Revised: Among the four networks, classical-LISTA is inferior to the other three by an obvious margin.

Original: LISTA-CPSS, TiLISTA and ALISTA perform comparably: ALISTA is observed to eventually achieve the lowest NMSE.
Revised: LISTA-CPSS, TiLISTA and ALISTA perform comparably: ALISTA is observed to eventually achieve the lowest NMSE.

Original: Figure 1(a)(a) also supports Theorem 2, that all networks have at most linear convergence, regardless of how freely their parameters can be end-to-end learned.
Revised: Figure 1(a) also supports Theorem 2, that all networks have at most linear convergence, regardless of how freely their parameters can be end-to-end learned.

Original: Figure 1 (b) - (d) further show that even in the presence of noise, ALISTA can empirically perform comparably with LISTA-CPSS and TiLISTA, and stay clearly better than LISTA and ISTA/FISTA.
Revised: Figure 1 (b) - (d) further show that even in the presence of noise, ALISTA can empirically perform comparably with LISTA-CPSS and TiLISTA, and stay clearly better than LISTA and ISTA/FISTA.

Original: Always note that ALISTA the smallest amount of parameters to learn from the end-to-end training
Revised: Always note that ALISTA the smallest amount of parameters to learn from the end-to-end training

Original: (Stage 2).
Revised: (Stage 2).

Original: The above results endorse that: i) the optimal LISTA layer-wise weights could be structured as W ( k ) =
Revised: The above results endorse that: i) the optimal LISTA layer-wise weights could be structured as W ( k ) =

Original: γ ( k ) W ; and ii) W could be analytically solved rather than learned from data, without incurring performance loss.
Revised: γ ( k ) W ; and ii) W could be analytically solved rather than learned from data, without incurring performance loss.

Original: We also observe the signiﬁcant reduction of training time for
Revised: We also observe the signiﬁcant reduction of training time for

Original: ALISTA: while LISTA-CPSS of the same depth took 6.5 hours to train, ALISTA was trained with 1.5 hours, on the same hardware (one 1080 Ti on server).
Revised: ALISTA: while LISTA-CPSS of the same depth took 6.5 hours to train, ALISTA was trained with 1.5 hours, on the same hardware (one 1080 Ti on server).

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We further supply Figures 2 andto support Theorem 1 from different perspectives.
Revised: We further supply Figures 2 and 3 to justify Theorem 1 from different perspectives.

Original: Figure 2 plots the learned parameters { γ ( k ) , θ ( k ) } in ALISTA (Stage 2), showing that they satisfy the properties proposed in Theorem 1: γ ( k ) bounded; θ ( k ) and γ ( k ) is proportional to sup x ∗ (cid:107) x ( k ) ( x ∗ ) − x ∗ (cid:107) 1 (“ sup x ∗ ” is taken over the test set).
Revised: Figure 2 plots the learned parameters { γ ( k ) , θ ( k ) } in ALISTA (Stage 2), showing that they satisfy the properties proposed in Theorem 1: γ ( k ) bounded; θ ( k ) and γ ( k ) is proportional to sup x ∗ (cid:107) x ( k ) ( x ∗ ) − x ∗ (cid:107) 1 (“ sup x ∗ ” is taken over the test set).

Original: Figure 3 reports the average magnitude  of the false positives and the true positives in x k ( x ∗ ) of ALISTA: the “true positives” curve draws the values of E {(cid:107) x k S ( x ∗ ) (cid:107) 22 / (cid:107) x k ( x ∗ ) (cid:107) 22 } w.r.t.
Revised: Figure 3 reports the average magnitude 5 of the false positives and the true positives in x k ( x ∗ ) of ALISTA: the “true positives” curve draws the values of E {(cid:107) x k S ( x ∗ ) (cid:107) 22 / (cid:107) x k ( x ∗ ) (cid:107) 22 } w.r.t.

Original: k (the expectation is taken over the test set), while “false positives” for E {(cid:107) x k S c ( x ∗ ) (cid:107) 22 / (cid:107) x k ( x ∗ ) (cid:107) 22 } .
Revised: k (the expectation is taken over the test set), while “false positives” for E {(cid:107) x k S c ( x ∗ ) (cid:107) 22 / (cid:107) x k ( x ∗ ) (cid:107) 22 } .

Original: The results support the Theorem 1 conclusion that support( x k ( x ∗ )) ⊂ S .
Revised: False positives take up small proportion over the positives, which supports the Theorem 1 conclusion that support( x k ( x ∗ )) ⊂ S .

Original: 5. V ALIDATION OF T HEOREM 3 (C ONVOLUTIONAL A NALYTIC LISTA)
Revised: 5. V ALIDATION OF T HEOREM 3 (C ONVOLUTIONAL A NALYTIC LISTA)

Original: For convolutional cases, we use real image data to verify Theorem 3.
Revised: For convolutional cases, we use real image data to verify Theorem 3.

Original: We train a convolutional dictionary d with D = 7 , M = 64 on the BSD500 training set (400 images), using the Algorithmin (Liu et al., 2018).
Revised: We train a convolutional dictionary d with D = 7 , M = 64 on the BSD500 training set (400 images), using the Algorithmin (Liu et al., 2018).

Original: We then use it for problems (23) and (25) and solve them with different N s.
Revised: We then use it for problems (22) and (24) and solve them with different N s.

Original: In Table 1, we take w N cir ∈ W N cir ,
Revised: In Table 2, we take w N cir ∈ W N cir ,

Original: w ∗ ∈
Revised: w ∗ ∈

Original: W 50cir (consider 50 as large enough)
Revised: W 50cir (consider 50 as large enough)

Original: For this example, W N cir has only one element.
Revised: For this example, W N cir has only one element.

Original: Table 1 shows that w N cir
Revised: Table 2 shows that w N cir

Original: = w ∗ for N ≥ 13 , i.e., the solution of the problem (25) is independent of N if N ≥ 2 D − 1 , justifying the ﬁrst conclusion in Theorem 3.
Revised: = w ∗ for N ≥ 13 , i.e., the solution of the problem (24) is independent of N if N ≥ 2 D − 1 , justifying the ﬁrst conclusion in Theorem 3.

Original: In Table 2, we take w N conv ∈ W N conv and w ∗ ∈
Revised: In Table 3, we take w N conv ∈ W N conv and w ∗ ∈

Original: w 13cir , where W N conv also has only one element.
Revised: w 13 cir , where W N conv also has only one element.

Original: Tableshows w N conv → w ∗ , i.e., the solution of the problem (23) converges to that of (25) as N increases, validating the second conclusion of Theorem 3.
Revised: Tableshows w N conv → w ∗ , i.e., the solution of the problem (22) converges to that of (24) as N increases, validating the second conclusion of Theorem 3.

Original: Visualized w ∗ ∈ w 13cir is displayed in Appendix F.
Revised: Visualized w ∗ ∈ w 13cir is displayed in Appendix G.

Original: Besides validating Theorem 3, we also present a real image denoising experiment to verify the effectiveness of Conv ALISTA.
Revised: Besides validating Theorem 3, we also present a real image denoising experiment to verify the effectiveness of Conv ALISTA.

Original: The detailed settings and results are presented in Appendix G.
Revised: The detailed settings and results are presented in Appendix H.

Original: [Table]
Revised: [Table]

Original:  w ∗ (cid:107)
Revised: / (cid:107) w ∗ (cid:107)

Original: N =
Revised: N =

Original: N = N =
Revised: N = N =

Original: N = N =
Revised: N = N =

Original: N =2 . 

Revised: N =2 .

Original: × 10 −9 .
Revised: × 10 −9 .

Original: 3 × 10 −3 .
Revised: 3 × 10 −3 .

Original: 9 × 10 −1 .
Revised: 9 × 10 −1 .

Original: 4 × 10 −8 .
Revised: 4 × 10 −8 .

Original: 8 × 10 −5 .
Revised: 8 × 10 −5 .

Original: 9 × 10 −
Revised: 9 × 10 −

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5. V ALIDATION OF R OBUST ALISTA
Revised: 5. V ALIDATION OF R OBUST ALISTA

Original: We empirically verify the effectiveness of Robust ALISTA, by sampling the dictionary perturbation ε D entry-wise i.i.d.
Revised: We empirically verify the effectiveness of Robust ALISTA, by sampling the dictionary perturbation ε D entry-wise i.i.d.

Original: from another Gaussian distribution N (0 , σ 2 max ) .
Revised: from another Gaussian distribution N (0 , σ 2 max ) .

Original: We choose σ max = 0.02 and 0.03.
Revised: We choose σ max = 0.02 and 0.03.

Original: Other simulation settings are by default the same as in Section 5.1.
Revised: Other simulation settings are by default the same as in Section 5.1.

Original: We then build the Robust
Revised: We then build the Robust

Original: ALISTA model, following the strategy in Section 4 and using a 4-layer encoder for approximating its second step (see Appendix E for details).
Revised: ALISTA model, following the strategy in Section 4 and using a 4-layer encoder for approximating its second step (see Appendix F for details).

Original: Correspondingly, we compare Robust ALISTA with
Revised: Correspondingly, we compare Robust ALISTA with

Original: TiLISTA and ALISTA with speciﬁc data augmentation : we straightforwardly augment their training sets, by including all data generated with randomly perturbed ˜ D s when training Robust ALISTA.
Revised: TiLISTA and ALISTA with speciﬁc data augmentation : we straightforwardly augment their training sets, by including all data generated with randomly perturbed ˜ D s when training Robust ALISTA.

Original: We also include the data-free FISTA algorithm into the comparison.
Revised: We also include the data-free FISTA algorithm into the comparison.

Original: Figure 4 plots the results when the trained models are applied on the testing data, generated with the same dictionary and perturbed by N (0 , σ t ) .
Revised: Figure 4 plots the results when the trained models are applied on the testing data, generated with the same dictionary and perturbed by N (0 , σ t ) .

Original: We vary σ t from zero to slightly above σ max .
Revised: We vary σ t from zero to slightly above σ max .

Original: Not surprisingly, FISTA is unaffected, while the other three data-driven models all slight degrade as σ t increases.
Revised: Not surprisingly, FISTA is unaffected, while the other three data-driven models all slight degrade as σ t increases.

Original: Compared to the augmented TiLISTA and ALISTA whose performance are both inferior to FISTA, the proposed Robust ALISTA appears to be much more favorable in improving robustness to model perturbations.
Revised: Compared to the augmented TiLISTA and ALISTA whose performance are both inferior to FISTA, the proposed Robust ALISTA appears to be much more favorable in improving robustness to model perturbations.

Original: In both σ max cases, it consistently achieves much lower NMSE than FISTA, even when σ t has slightly surpassed σ max .
Revised: In both σ max cases, it consistently achieves much lower NMSE than FISTA, even when σ t has slightly surpassed σ max .

Original: Although the NMSE of ALISTA may decrease faster if σ t continues growing larger, such decrease could be alleviated by improving σ max in training, e.g., by comparing σ max = 0.02 and 0.03.
Revised: Although the NMSE of ALISTA may decrease faster if σ t continues growing larger, such decrease could be alleviated by improving σ max in training, e.g., by comparing σ max = 0.02 and 0.03.

Original: Robust ALISTA demonstrates remarkable robustness and maintains the best NMSE performance, within at least the [0, σ max ] range.
Revised: Robust ALISTA demonstrates remarkable robustness and maintains the best NMSE performance, within at least the [0, σ max ] range.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: Based on the recent theoretical advances of LISTA, we have made further steps to reduce the training complexity and improve the robustness of LISTA.
Revised: Based on the recent theoretical advances of LISTA, we have made further steps to reduce the training complexity and improve the robustness of LISTA.

Original: Speciﬁcally, we no longer train any matrix for
Revised: Speciﬁcally, we no longer train any matrix for

Original: LISTA but directly use the solution to an analytic minimization problem to solve for its layer-wise weights.
Revised: LISTA but directly use the solution to an analytic minimization problem to solve for its layer-wise weights.

Original: Therefore, only two scalar sequences (stepsizes and thresholds) still need to be trained.
Revised: Therefore, only two scalar sequences (stepsizes and thresholds) still need to be trained.

Original: Excluding the matrix from training is backed by our theoretical upper and lower bounds.
Revised: Excluding the matrix from training is backed by our theoretical upper and lower bounds.

Original: The resulting method, Analytic LISTA or ALISTA, is not only faster to train but performs as well as the state-of-the-art variant of LISTA by (Chen et al., 2018).
Revised: The resulting method, Analytic LISTA or ALISTA, is not only faster to train but performs as well as the state-of-the-art variant of LISTA by (Chen et al., 2018).

Original: This discovery motivates us to further replace the minimization algorithm by its unfolding neural network, and train this neural network to more quickly produce the weight matrix.
Revised: This discovery motivates us to further replace the minimization algorithm by its unfolding neural network, and train this neural network to more quickly produce the weight matrix.

Original: The resulting algorithm is used to handle perturbations in the model dictionary — we only train once for a dictionary with all its small perturbations.
Revised: The resulting algorithm is used to handle perturbations in the model dictionary — we only train once for a dictionary with all its small perturbations.

Original: Our future work will investigate the theoretical sensitivity of ALISTA (and its convolutional version) to noisy measurements.
Revised: Our future work will investigate the theoretical sensitivity of ALISTA (and its convolutional version) to noisy measurements.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: In this proof, we use the notion x ( k ) to replace x ( k ) ( x ∗ ) for simplicity.
Revised: In this proof, we use the notion x ( k ) to replace x ( k ) ( x ∗ ) for simplicity.

Original: We ﬁx D in the proof, ˜ µ ( D ) can be simply written as ˜ µ .
Revised: We ﬁx D in the proof, ˜ µ ( D ) can be simply written as ˜ µ .

Original: Before proving Theorem 1, we present and prove a lemma.
Revised: Before proving Theorem 1, we present and prove a lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: With all the settings the same with those in Theorem 1, we have support( x ( k ) )
Revised: With all the settings the same with those in Theorem 1, we have support( x ( k ) )

Original: ⊂ S , ∀ k.
Revised: ⊂ S , ∀ k.

Original: In another word, there are no false positives in x ( k ) :
Revised: In another word, there are no false positives in x ( k ) :

Original: x ( k ) i = 0 , ∀ i / ∈ S , ∀ k .
Revised: x ( k ) i = 0 , ∀ i / ∈ S , ∀ k .

Original: Proof.
Revised: Proof.

Original: Take arbitrary x ∗ ∈ X ( B, s ) .
Revised: Take arbitrary x ∗ ∈ X ( B, s ) .

Original: We prove Lemma 1 by induction.
Revised: We prove Lemma 1 by induction.

Original: As k = 0 , (28) is satisﬁed since x (0) = 0 .
Revised: As k = 0 , (27) is satisﬁed since x (0) = 0 .

Original: Fixing k , and assuming support( x ( k ) )
Revised: Fixing k , and assuming support( x ( k ) )

Original: ⊂ S , we have
Revised: ⊂ S , we have

Original: [Equation]
Revised: [Equation]

Original: By (10), the thresholds are taken as θ ( k ) =
Revised: By (9), the thresholds are taken as θ ( k ) =

Original: ˜ µγ ( k ) sup x ∗ {
Revised: ˜ [Equation] ( k ) sup x ∗ {

Original: [Equation]
Revised: [Equation]

Original: x ( k ) − x ∗ (cid:107) 1 } .
Revised: x ( k ) − x ∗ (cid:107) 1 } .

Original: Also, since W ∈ W ( D ) , we have | ( W : ,i ) T D : ,j | ≤ ˜ µ for all j (cid:54) =
Revised: Also, since W ∈ W ( D ) , we have | ( W : ,i ) T D : ,j | ≤ ˜ µ for all j (cid:54) =

Original: i .
Revised: i .

Original: Thus, for all i / ∈ S ,
Revised: Thus, for all i / ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: − x ∗ j (cid:12)(cid:12) = (cid:88) j ∈ S ˜ µγ ( k ) (cid:12)(cid:12) x
Revised: − x ∗ j (cid:12)(cid:12) = (cid:88) j ∈ S ˜ µγ ( k ) (cid:12)(cid:12) x

Original: ( k )
Revised: ( k )

Original: j − x ∗ j (cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) − γ ( k ) (cid:88)
Revised: j − x ∗ j (cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) − γ ( k ) (cid:88)

Original: j ∈ S
Revised: j ∈ S

Original: ( W : ,i ) T D : ,j ( x ( k ) j − x ∗ j ) (cid:12)(cid:12)(cid:12) , which implies x ( k +1)
Revised: ( W : ,i ) T D : ,j ( x ( k ) j − x ∗ j ) (cid:12)(cid:12)(cid:12) , which implies x ( k +1)

Original: i = 0 , ∀ i / ∈ S by the deﬁnition of η θ ( k ) , i.e., support( x ( k +1) )
Revised: i = 0 , ∀ i / ∈ S by the deﬁnition of η θ ( k ) , i.e., support( x ( k +1) )

Original: ⊂ S
Revised: ⊂ S

Original: By induction, (28) is proved.
Revised: By induction, (27) is proved.

Original: With Lemma 1, we are able to prove Theorem 1 now.
Revised: With Lemma 1, we are able to prove Theorem 1 now.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Take arbitrary x ∗ ∈ X ( B, s ) .
Revised: Take arbitrary x ∗ ∈ X ( B, s ) .

Original: For all i ∈ S , by (28), we obtain x
Revised: For all i ∈ S , by (27), we obtain x

Original: ( k +
Revised: ( k +

Original: [Equation]
Revised: [Equation]

Original:  where ∂(cid:96) 1 ( x ) is the sub-gradient of | x | , x ∈ R : ∂(cid:96) 1 ( x ) = (cid:26) { sign ( x ) } if x (cid:54) = 0 , [ − 1 , 1] if x = 0 .
Revised: ) , where ∂(cid:96) 1 ( x ) is the sub-gradient of | x | , x ∈ R : ∂(cid:96) 1 ( x ) = (cid:26) { sign ( x ) } if x (cid:54) = 0 , [ − 1 , 1] if x = 0 .

Original: The choice of W (8) gives ( W : ,i ) T D : ,i = 1 .
Revised: The choice of W ∈ W ( D ) gives ( W : ,i ) T D : ,i = 1 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: j
Revised: Then the following inclusion formula holds for all i ∈ S , x

Original: [Equation]
Revised: [Equation]

Original: Then the following inclusion formula holds for all i ∈ S , x ( k +1)
Revised: k +1)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of ∂(cid:96) 1 , every element in ∂(cid:96) 1 ( x ) , ∀ x ∈ R has a magnitude less than or equal to 1 .
Revised: By the deﬁnition of ∂(cid:96) 1 , every element in ∂(cid:96) 1 ( x ) , ∀ x ∈ R has a magnitude less than or equal to 1 .

Original: Thus, for all i ∈ S ,
Revised: Thus, for all i ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: Equation (28) implies (cid:107) x ( k ) − x ∗ (cid:107) 

Revised: Equation (27) implies (cid:107) x ( k ) − x ∗ (cid:107)

Original: = (cid:107) x ( k ) S − x ∗ S (cid:107) 1 for all k .
Revised: = (cid:107) x ( k ) S − x ∗ S (cid:107) 1 for all k .

Original: Then (cid:107) x
Revised: Then (cid:107) x

Original: ( k +1)
Revised: ( k +1)

Original: − x ∗ (cid:107) 1 = (cid:88)
Revised: − x ∗ (cid:107) 1 = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Taking supremum of the above inequality over x ∗ ∈ X ( B, s ) , by | S | ≤ s ,
Revised: Taking supremum of the above inequality over x ∗ ∈ X ( B, s ) , by | S | ≤ s ,

Original: [Equation]
Revised: [Equation]

Original: By the value of θ ( k ) given in (10), we have
Revised: By the value of θ ( k ) given in (9), we have

Original: [Equation]
Revised: [Equation]

Original: Let c ( τ ) =
Revised: Let c ( τ ) =

Original: − log (cid:0) (2˜ µs − ˜ µ ) γ ( τ ) + | 1 − γ ( τ ) | (cid:1) .
Revised: − log (cid:0) (2˜ µs − ˜ µ ) γ ( τ ) + | 1 − γ ( τ ) | (cid:1) .

Original: Then, by induction,
Revised: Then, by induction,

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) x (cid:107) 2 ≤ (cid:107) x
Revised: Since (cid:107) x (cid:107) 2 ≤ (cid:107) x

Original: (cid:107) 1 for any x ∈ R n , we can get the upper bound for (cid:96) 2 norm:
Revised: (cid:107) 1 for any x ∈ R n , we can get the upper bound for (cid:96) 2 norm:

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: The assumption s <
Revised: The assumption s <

Original: (1 + 1 / ˜ µ ) / 2 gives 2˜ µs − ˜ µ < 1 .
Revised: (1 + 1 / ˜ µ ) / 2 gives 2˜ µs − ˜ µ < 1 .

Original: If 0 < γ ( k ) ≤ 1 , we have c ( k ) > 0 .
Revised: If 0 < γ ( k ) ≤ 1 , we have c ( k ) > 0 .

Original: If 1 < γ ( k ) < 2 / (1 + 2˜ µs − ˜ µ ) , we have (2˜ µs − ˜ µ ) γ ( k ) + | 1 − γ ( k ) | =
Revised: If 1 < γ ( k ) < 2 / (1 + 2˜ µs − ˜ µ ) , we have (2˜ µs − ˜ µ ) γ ( k ) + | 1 − γ ( k ) | =

Original: (2˜ µs − ˜ µ ) γ ( k ) + γ ( k ) − 1 < 1 , which implies c ( k ) > 0 .
Revised: (2˜ µs − ˜ µ ) γ ( k ) + γ ( k ) − 1 < 1 , which implies c ( k ) > 0 .

Original: Theorem 1 is proved.
Revised: Theorem 1 is proved.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: We ﬁx D and sample a x ∗ ∼ P X .
Revised: We ﬁx D and sample a x ∗ ∼ P X .

Original: If we can prove
Revised: If we can prove

Original: P (cid:16) (14) does not hold (cid:12)(cid:12)(cid:12) support ( x ∗ ) =
Revised: P (cid:16) (13) does not hold (cid:12)(cid:12)(cid:12) support ( x ∗ ) =

Original: S (cid:17) ≤ (cid:15) | S | + (cid:15) | S | , (29) then the lower bound (14) in Theorem 2 is proved by
Revised: S (cid:17) ≤ (cid:15) | S | + (cid:15) | S | , (28) then the lower bound (13) in Theorem 2 is proved by

Original: P (cid:16) (14) holds (cid:17) =
Revised: P (cid:16) (13) holds (cid:17) =

Original: (cid:88)
Revised: (cid:88)

Original: S , 2 ≤| S |≤ s
Revised: S , 2 ≤| S |≤ s

Original: P (cid:16) (14) holds (cid:12) (cid:12) (cid:12) support ( x ∗ ) =
Revised: P (cid:16) (13) holds (cid:12) (cid:12) (cid:12) support ( x ∗ ) =

Original: S (cid:17) P (cid:16) support ( x ∗ ) = S (cid:17) ≥ (1 − (cid:15)s 3 / 2 − (cid:15) 2 ) (cid:88) 2 ≤| S |≤ s
Revised: S (cid:17) P (cid:16) support ( x ∗ ) = S (cid:17) ≥ (1 − (cid:15)s 3 / 2 − (cid:15) 2 ) (cid:88) 2 ≤| S |≤ s

Original: P (cid:16) support ( x ∗ ) = S (cid:17) =1 − (cid:15)s 3 / 2 − (cid:15) 2 .
Revised: P (cid:16) support ( x ∗ ) = S (cid:17) =1 − (cid:15)s 3 / 2 − (cid:15) 2 .

Original: Now we ﬁx k and prove inequality (29) by three steps:
Revised: Now we ﬁx k and prove inequality (28) by three steps:

Original: Step 1: If (14) does not hold, then what condition x ∗ should satisfy?
Revised: Step 1: If (13) does not hold, then what condition x ∗ should satisfy?

Original: Fixing k , we deﬁne a set X ( k ) ( (cid:15) ) , which involves all the x ∗ that does not satisfy (14):
Revised: Fixing k , we deﬁne a set X ( k ) ( (cid:15) ) , which involves all the x ∗ that does not satisfy (13):

Original: [Equation]
Revised: [Equation]

Original: Let S = support ( x ∗ ) .
Revised: Let S = support ( x ∗ ) .

Original: For x ∗ ∈ X ( k ) ( (cid:15) ) , we consider two cases:
Revised: For x ∗ ∈ X ( k ) ( (cid:15) ) , we consider two cases:

Original: | x ∗ i | > (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min / 3 s ) k , ∀ i ∈ S . 2.
Revised: | x ∗ i | > (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min / 3 s ) k , ∀ i ∈ S . 2.

Original: | x ∗ i | ≤ (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min / 3 s ) k , for some i ∈ S .
Revised: | x ∗ i | ≤ (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min / 3 s ) k , for some i ∈ S .

Original: If case 1 holds, we obtain that the support of x ( k ) is exactly the same with that of x ∗ : support ( x ( k ) ( x ∗ ))
Revised: If case 1 holds, we obtain that the support of x ( k ) is exactly the same with that of x ∗ : support ( x ( k ) ( x ∗ ))

Original: =
Revised: =

Original: S .
Revised: S .

Original: Then the relationship between x ( k ) and x ( k − 1) can be reduced to an afﬁne transform:
Revised: Then the relationship between x ( k ) and x ( k − 1) can be reduced to an afﬁne transform:

Original: [Equation]
Revised: [Equation]

Original: Subtracting x ∗ from the two sides of (30), we obtain (cid:13)(cid:13)(cid:13)(cid:0)
Revised: Subtracting x ∗ from the two sides of (29), we obtain (cid:13)(cid:13)(cid:13)(cid:0)

Original: [Equation]
Revised: [Equation]

Original: , where the last equality is due to Deﬁnition 3.
Revised: , where the last equality is due to Deﬁnition 3.

Original: Thus, for all x ∗ ∈ X ( k ) ( (cid:15) )
Revised: Thus, for all x ∗ ∈ X ( k ) ( (cid:15) )

Original: [Equation]
Revised: [Equation]

Original: if case 1 holds
Revised: if case 1 holds

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Multiplying both sides of (31) by ( I − ( W ( k − 1) : , S ) T D : , S ) − 1 , we have
Revised: Multiplying both sides of (30) by ( I − ( W ( k − 1) : , S ) T D : , S ) − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to (12).
Revised: where the last inequality is due to (11).

Original: Let ˜ x
Revised: Let ˜ x

Original: ( k − 1) denote the bias of x
Revised: ( k − 1) denote the bias of x

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( x (
Revised: ( x (

Original: [Equation]
Revised: [Equation]

Original: then we get a condition that x ∗ satisﬁes if case 1 holds:
Revised: then we get a condition that x ∗ satisﬁes if case 1 holds:

Original: [Equation]
Revised: [Equation]

Original: If case 2 holds, x ∗ belongs to the following set:
Revised: If case 2 holds, x ∗ belongs to the following set:

Original: [Equation]
Revised: [Equation]

Original: for some i ∈ S
Revised: for some i ∈ S

Original: [Equation]
Revised: [Equation]

Original: Then for any x ∗ ∈ X ( k ) ( (cid:15) ) , either x ∗ ∈ X ( k − 1) ( (cid:15) ) or x ∗ ∈ ˜
Revised: Then for any x ∗ ∈ X ( k ) ( (cid:15) ) , either x ∗ ∈ X ( k − 1) ( (cid:15) ) or x ∗ ∈ ˜

Original: X
Revised: X

Original: ( k ) ( (cid:15) ) holds.
Revised: ( k ) ( (cid:15) ) holds.

Original: In another word,
Revised: In another word,

Original: [Equation]
Revised: [Equation]

Original: Step 2:
Revised: Step 2:

Original: By imitating the construction of X ( k ) ( (cid:15) ) , we construct
Revised: By imitating the construction of X ( k ) ( (cid:15) ) , we construct

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: Similar to Step 1, we divide X ( k − 1) ( (cid:15) ) into two sets: ˜ X ( k − 1) ( (cid:15) ) and X ( k − 2) ( (cid:15) ) , then we divide
Revised: Similar to Step 1, we divide X ( k − 1) ( (cid:15) ) into two sets: ˜ X ( k − 1) ( (cid:15) ) and X ( k − 2) ( (cid:15) ) , then we divide

Original: X ( k − 2) ( (cid:15) ) into ˜ X
Revised: X ( k − 2) ( (cid:15) ) into ˜ X

Original: k − 2) ( (cid:15) ) and X ( k − 3) ( (cid:15) ) .
Revised: k − 2) ( (cid:15) ) and X ( k − 3) ( (cid:15) ) .

Original: Repeating the process, until dividing X (1) ( (cid:15) ) into ˜ X (1) ( (cid:15) ) and X (0) ( (cid:15) ) .
Revised: Repeating the process, until dividing X (1) ( (cid:15) ) into ˜ X (1) ( (cid:15) ) and X (0) ( (cid:15) ) .

Original: By induction, we have
Revised: By induction, we have

Original: [Equation]
Revised: [Equation]

Original: where the sets are deﬁned as follows for all j = 0 , 1 ,
Revised: where the sets are deﬁned as follows for all j = 0 , 1 ,

Original: [Equation]
Revised: [Equation]

Original: · · · , k :
Revised: · · · , k :

Original: [Equation]
Revised: [Equation]

Original: and the bias is deﬁned as following for all j = 0 , 1 , 2 , · · · , k :
Revised: and the bias is deﬁned as following for all j = 0 , 1 , 2 , · · · , k :

Original: [Equation]
Revised: [Equation]

Original: Step 3: Estimating the probabilities of all the sets in (32).
Revised: Step 3: Estimating the probabilities of all the sets in (31).

Original: By (32), we have
Revised: By (31), we have

Original: [Equation]
Revised: [Equation]

Original: Now we have to prove that each of the above terms is small, then P ( x ∗ ∈ X ( k ) ( (cid:15) )
Revised: Now we have to prove that each of the above terms is small, then P ( x ∗ ∈ X ( k ) ( (cid:15) )

Original: | support ( x ∗ ) = S ) is small and (29) will be proved.
Revised: | support ( x ∗ ) = S ) is small and (28) will be proved.

Original: Deﬁne a set of n -dimensional sign numbers
Revised: Deﬁne a set of n -dimensional sign numbers

Original: [Equation]
Revised: [Equation]

Original: Since sign (cid:0) x ( k − j + t ) S (cid:1) ∈
Revised: Since sign (cid:0) x ( k − j + t ) S (cid:1) ∈

Original: Si ( | S | ) for all t
Revised: Si ( | S | ) for all t

Original: = 1 , 2 , · · · , j , { sign ( x ( k − j + t ) S ) }
Revised: = 1 , 2 , · · · , j , { sign ( x ( k − j + t ) S ) }

Original: jt =1 has ﬁnitely possible values.
Revised: jt =1 has ﬁnitely possible values.

Original: Let sign ( x ( k − j + t ) S )
Revised: Let sign ( x ( k − j + t ) S )

Original: =
Revised: =

Original: s ( t ) for t = 1 , 2 , · · · , j .
Revised: s ( t ) for t = 1 , 2 , · · · , j .

Original: Then ˜ x ( k − j ) i ( x ∗ ) is independent of x ∗ and can be written as ˜ x
Revised: Then ˜ x ( k − j ) i ( x ∗ ) is independent of x ∗ and can be written as ˜ x

Original: ( k − j )
Revised: ( k − j )

Original: i ( s (1) , s (2) , · · · , s ( j ) ) .
Revised: i ( s (1) , s (2) , · · · , s ( j ) ) .

Original: Thus, we have
Revised: Thus, we have

Original: P ( x ∗ ∈ ˜
Revised: P ( x ∗ ∈ ˜

Original: X
Revised: X

Original: ( k − j ) ( (cid:15) )
Revised: ( k − j ) ( (cid:15) )

Original: | support ( x ∗ ) = S ) = (cid:88) i ∈ S (cid:88) s (1) ∈
Revised: | support ( x ∗ ) = S ) = (cid:88) i ∈ S (cid:88) s (1) ∈

Original: Si ( | S | ) (cid:88) s (2) ∈
Revised: Si ( | S | ) (cid:88) s (2) ∈

Original: Si ( | S | ) · · · (cid:88) s (
Revised: Si ( | S | ) · · · (cid:88) s (

Original: j ) ∈ Si ( | S | )
Revised: j ) ∈ Si ( | S | )

Original: P (cid:16) | x ∗ i + ˜ x ( k − j ) i ( x ∗ )
Revised: P (cid:16) | x ∗ i + ˜ x ( k − j ) i ( x ∗ )

Original: | < (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min ) k − j 3 − ks , sign ( x ( k ) S )
Revised: | < (cid:15) (cid:107) x ∗ (cid:107) 2 (¯ σ min ) k − j 3 − ks , sign ( x ( k ) S )

Original: = s (1) , · · · , sign ( x ( k − j +1) S ) =
Revised: = s (1) , · · · , sign ( x ( k − j +1) S ) =

Original: s ( j ) (cid:12)(cid:12)(cid:12) support ( x ∗ ) =
Revised: s ( j ) (cid:12)(cid:12)(cid:12) support ( x ∗ ) =

Original: S (cid:17) ≤ (cid:88) i ∈ S (cid:88) s (1) ∈
Revised: S (cid:17) ≤ (cid:88) i ∈ S (cid:88) s (1) ∈

Original: Si ( | S | ) (cid:88) s (2) ∈
Revised: Si ( | S | ) (cid:88) s (2) ∈

Original: Si ( | S | ) · · · (cid:88) s (
Revised: Si ( | S | ) · · · (cid:88) s (

Original: j ) ∈ Si ( | S | )
Revised: j ) ∈ Si ( | S | )

Original: P (cid:16) | x ∗ i + ˜ x ( k − j ) i ( s (1) , s (2) , · · · , s ( j ) ) | < (cid:15) (cid:112) | S | B (¯ σ min )
Revised: P (cid:16) | x ∗ i + ˜ x ( k − j ) i ( s (1) , s (2) , · · · , s ( j ) ) | < (cid:15) (cid:112) | S | B (¯ σ min )

Original: k − j 3 − ks (cid:12)(cid:12)(cid:12) support ( x ∗ ) =
Revised: k − j 3 − ks (cid:12)(cid:12)(cid:12) support ( x ∗ ) =

Original: S (cid:17) ≤ (cid:88) i ∈ S (cid:88) s (1) ∈
Revised: S (cid:17) ≤ (cid:88) i ∈ S (cid:88) s (1) ∈

Original: Si ( | S | ) (cid:88) s (2) ∈
Revised: Si ( | S | ) (cid:88) s (2) ∈

Original: Si ( | S | ) · · · (cid:88) s (
Revised: Si ( | S | ) · · · (cid:88) s (

Original: j ) ∈ Si ( | S | ) (cid:15) (cid:112) | S | B (¯ σ min )
Revised: j ) ∈ Si ( | S | ) (cid:15) (cid:112) | S | B (¯ σ min )

Original: k − j 3 − ks
Revised: k − j 3 − ks

Original: B = | S | 3 j | S | ( (cid:15) (cid:112) | S | (cid:16) (¯ σ min ) k − j 3 − ks (cid:17) ≤ (cid:15) | S | 3 / 2 (¯ σ min ) k − j
Revised: B = | S | 3 j | S | ( (cid:15) (cid:112) | S | (cid:16) (¯ σ min ) k − j 3 − ks (cid:17) ≤ (cid:15) | S | 3 / 2 (¯ σ min ) k − j

Original: [Equation]
Revised: [Equation]

Original: where the second inequality comes from the uniform distribution of x ∗ S (Assumption 2), the last inequality comes from | S | ≤ s .
Revised: where the second inequality comes from the uniform distribution of x ∗ S (Assumption 2), the last inequality comes from | S | ≤ s .

Original: The last term, due to the uniform distribution of x ∗ S and x (0) = 0 , can be bounded by
Revised: The last term, due to the uniform distribution of x ∗ S and x (0) = 0 , can be bounded by

Original: [Equation]
Revised: [Equation]

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: Then (29) is proved.
Revised: Then (28) is proved.

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: There are two conclusions in Theorem 3.
Revised: There are two conclusions in Theorem 3.

Original: We prove the two conclusions in the following two subsections respectively.
Revised: We prove the two conclusions in the following two subsections respectively.

Original: C. P ROOF OF C ONCLUSION 1.
Revised: C. P ROOF OF C ONCLUSION 1.

Original: Before proving Conclusion 1, we analyze the operator D N cir in detail.
Revised: Before proving Conclusion 1, we analyze the operator D N cir in detail.

Original: The circular convolution (24) is equivalent with:
Revised: The circular convolution (23) is equivalent with:

Original: [Equation]
Revised: [Equation]

Original: where the circulant matrix is element-wise deﬁned as:
Revised: where the circulant matrix is element-wise deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: others (36)
Revised: others (35)

Original: Similarly, the corresponding circulant matrix W N cir ( i, j ; k, l, m ) of dictionary w is:
Revised: Similarly, the corresponding circulant matrix W N cir ( i, j ; k, l, m ) of dictionary w is:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j

Original: [Equation]
Revised: [Equation]

Original: 0 , others (37)
Revised: others (36)

Original: If we vectorize b and x = [ x 1 , · · · , x M ] T ,  operator D N cir is a matrix, where ( i, j ) is its row index and ( k, l, m ) is its column index.
Revised: As we deﬁned in Section 3, b is a vector. With x = [ x 1 , · · · , x M ] T , x is a vector. Then the operator D N cir is a matrix, where ( i, j ) is its row index and ( k, l, m ) is its column index.

Original: Deﬁne a function measuring the difference between i and k :
Revised: Deﬁne a function measuring the difference between i and k :

Original: [Equation]
Revised: [Equation]

Original: The coherence between D N cir ( i, j ; k, l, m ) and W N cir ( i, j ; k, l, m ) :
Revised: The coherence between D N cir ( i, j ; k, l, m ) and W N cir ( i, j ; k, l, m ) :

Original: B coh =
Revised: B coh =

Original: [Equation]

Original:  D N cir ) T W N cir is elementwise deﬁned by:
Revised: ( D N cir ) T W N cir is elementwise deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Given N ≥ 2 D − 1 , it holds that:
Revised: Given N ≥ 2 D − 1 , it holds that:

Original: (a) I ( k 1 , k 2 ) (cid:54) = ∅ if and only if “ 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 ” or “ 0 < ( k 2 − k 1 ) mod N ≤ D − 1 ” holds.
Revised: (a) I ( k 1 , k 2 ) (cid:54) = ∅ if and only if “ 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 ” or “ 0 < ( k 2 − k 1 ) mod N ≤ D − 1 ” holds.

Original: (b) J ( l 1 , l 2 ) (cid:54) = ∅
Revised: (b) J ( l 1 , l 2 ) (cid:54) = ∅

Original: if and only if “ 0 ≤ ( l 1 − l 2 ) mod N ≤ D − 1 ” or “ 0 < ( l 2 − l 1 ) mod N ≤ D − 1 ” holds.
Revised: if and only if “ 0 ≤ ( l 1 − l 2 ) mod N ≤ D − 1 ” or “ 0 < ( l 2 − l 1 ) mod N ≤ D − 1 ” holds.

Original: Proof.
Revised: Proof.

Original: Now we prove Conclusion (a).
Revised: Now we prove Conclusion (a).

Original: Firstly, we prove “if.”
Revised: Firstly, we prove “if.”

Original: If 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 and
Revised: If 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 and

Original: N ≥ 2 D − 1 , we have
Revised: N ≥ 2 D − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: If 0 < ( k 2 − k 1 ) mod N ≤ D − 1 and N ≥ 2 D − 1 , we have
Revised: If 0 < ( k 2 − k 1 ) mod N ≤ D − 1 and N ≥ 2 D − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Secondly, we prove “only if.”
Revised: Secondly, we prove “only if.”

Original: If I ( k 1 , k 2 ) (cid:54) = ∅ , we can select an i ∈ I ( k 1 , k 2 ) .
Revised: If I ( k 1 , k 2 ) (cid:54) = ∅ , we can select an i ∈ I ( k 1 , k 2 ) .

Original: Let r 

Revised: Let r

Original: =
Revised: =

Original: ( k 1 − i ) mod N and r 2 =
Revised: ( k 1 − i ) mod N and r 2 =

Original: ( k 2 − i ) mod N .
Revised: ( k 2 − i ) mod N .

Original: By the deﬁnition of I ( k 1 , k 2 ) , we have 0 ≤ r 1 , r 2 ≤ D − 1 .
Revised: By the deﬁnition of I ( k 1 , k 2 ) , we have 0 ≤ r 1 , r 2 ≤ D − 1 .

Original: Two cases should be considered here.
Revised: Two cases should be considered here.

Original: Case 1: r 1 ≥ r 2 .
Revised: Case 1: r 1 ≥ r 2 .

Original: Since 0 ≤ r 1 − r 2 ≤ D − 1 ≤ N − 1 , it holds that r 1 − r 

Revised: Since 0 ≤ r 1 − r 2 ≤ D − 1 ≤ N − 1 , it holds that r 1 − r

Original: =
Revised: =

Original: ( r 1 − r 2 ) mod N .
Revised: ( r 1 − r 2 ) mod N .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: The equality “ 0 ≤ r 1 − r 2 ≤ D − 1 ” leads to the conclusion “ 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 ”.
Revised: The equality “ 0 ≤ r 1 − r 2 ≤ D − 1 ” leads to the conclusion “ 0 ≤ ( k 1 − k 2 ) mod N ≤ D − 1 ”.

Original: In case 2 where r 1 < r 2 , we can obtain 0 < ( k 2 − k 1 ) mod N ≤ D − 1 with the similar arguments.
Revised: In case 2 where r 1 < r 2 , we can obtain 0 < ( k 2 − k 1 ) mod N ≤ D − 1 with the similar arguments.

Original: Conclusion (b) can be proved by the same argument with the proof of (a).
Revised: Conclusion (b) can be proved by the same argument with the proof of (a).

Original: Lemma 2 is proved.
Revised: Lemma 2 is proved.

Original: Now we ﬁx k 1 , l 1 and consider what values of k 2 , l 2 give I ( k 1 , k 2 ) (cid:54) = ∅ and J ( l 1 , l 2 ) (cid:54) = ∅ .
Revised: Now we ﬁx k 1 , l 1 and consider what values of k 2 , l 2 give I ( k 1 , k 2 ) (cid:54) = ∅ and J ( l 1 , l 2 ) (cid:54) = ∅ .

Original: Deﬁne four index sets given 0 ≤ k 1 , l 1 ≤ N − 1 :
Revised: Deﬁne four index sets given 0 ≤ k 1 , l 1 ≤ N − 1 :

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: If N ≥ 2 D − 1 , we have:
Revised: If N ≥ 2 D − 1 , we have:

Original: (a) The cardinality of K ( k 1 ) , ¯ K ( k 1 ) :
Revised: (a) The cardinality of K ( k 1 ) , ¯ K ( k 1 ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , | ¯ K ( k 1 ) | =
Revised: , | ¯ K ( k 1 ) | =

Original: D − 1 .
Revised: D − 1 .

Original: (b) K ( k 1 )
Revised: (b) K ( k 1 )

Original: ∩ ¯ K ( k 1 )
Revised: ∩ ¯ K ( k 1 )

Original: = ∅ .
Revised: = ∅ .

Original: (c) The cardinality of L ( l 1 ) , ¯ L ( l 1 ) :
Revised: (c) The cardinality of L ( l 1 ) , ¯ L ( l 1 ) :

Original: | L ( l 1 ) | =
Revised: | L ( l 1 ) | =

Original: D, | ¯ L ( l 1 ) | =
Revised: D, | ¯ L ( l 1 ) | =

Original: D − 1 .
Revised: D − 1 .

Original: (d) L ( l 1 ) ∩ ¯ L ( l 1 )
Revised: (d) L ( l 1 ) ∩ ¯ L ( l 1 )

Original: = ∅ .
Revised: = ∅ .

Original: Proof.
Revised: Proof.

Original: Now we prove Conclusion (a).
Revised: Now we prove Conclusion (a).

Original: The set K ( k 1 ) can be equivalently written as
Revised: The set K ( k 1 ) can be equivalently written as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let k ( r k ) =
Revised: Let k ( r k ) =

Original: ( k 1 − r k ) mod N .
Revised: ( k 1 − r k ) mod N .

Original: We want to show that k ( r 1 k ) (cid:54) =
Revised: We want to show that k ( r 1 k ) (cid:54) =

Original: k ( r 2 k ) as long as r 1 k (cid:54) =
Revised: k ( r 2 k ) as long as r 1 k (cid:54) =

Original: r 2 k .
Revised: r 2 k .

Original: Without loss of generality, we assume 0 ≤ r 1 k < r 2 k ≤ D − 1 .
Revised: Without loss of generality, we assume 0 ≤ r 1 k < r 2 k ≤ D − 1 .

Original: By the deﬁnition of modulo operation, There exist two integers q, q (cid:48) such that
Revised: By the deﬁnition of modulo operation, There exist two integers q, q (cid:48) such that

Original: [Equation]
Revised: [Equation]

Original: Suppose k ( r 1 k ) =
Revised: Suppose k ( r 1 k ) =

Original: k ( r 2 k ) .
Revised: k ( r 2 k ) .

Original: Taking the difference between the above two equations, we obtain r 2 k − r 1 k
Revised: Taking the difference between the above two equations, we obtain r 2 k − r 1 k

Original: =
Revised: =

Original: ( q (cid:48) − q ) N , i.e, N divides r 2 k − r 1 k .
Revised: ( q (cid:48) − q ) N , i.e, N divides r 2 k − r 1 k .

Original: However, 0 ≤ r 1 k < r 2 k ≤ D − 1 implies 1 ≤ r 2 k − r 1 k ≤
Revised: However, 0 ≤ r 1 k < r 2 k ≤ D − 1 implies 1 ≤ r 2 k − r 1 k ≤

Original: D − 1 ≤ N − 1 , which contradicts with “ N dividing r 2 k − r 1 k .”
Revised: D − 1 ≤ N − 1 , which contradicts with “ N dividing r 2 k − r 1 k .”

Original: Thus, it holds that k ( r 1 k ) (cid:54) =
Revised: Thus, it holds that k ( r 1 k ) (cid:54) =

Original: k ( r 2 k ) .
Revised: k ( r 2 k ) .

Original: Then we have | K ( k 1 )
Revised: Then we have | K ( k 1 )

Original: |
Revised: |

Original: =
Revised: =

Original: D .
Revised: D .

Original: In the same way, we have
Revised: In the same way, we have

Original: [Equation]
Revised: [Equation]

Original: and | ¯ K ( k 1 ) | =
Revised: and | ¯ K ( k 1 ) | =

Original: D − 1 .
Revised: D − 1 .

Original: Conclusion (a) is proved.
Revised: Conclusion (a) is proved.

Original: Now we prove Conclusion (b).
Revised: Now we prove Conclusion (b).

Original: Suppose K ( k 1 ) ∩ ¯ K ( k 1 ) (cid:54) = ∅ .
Revised: Suppose K ( k 1 ) ∩ ¯ K ( k 1 ) (cid:54) = ∅ .

Original: Pick a k 2 ∈ K ( k 1 )
Revised: Pick a k 2 ∈ K ( k 1 )

Original: ∩ ¯ K ( k 1 ) .
Revised: ∩ ¯ K ( k 1 ) .

Original: Let r 

Revised: Let r

Original: =
Revised: =

Original: ( k 1 − k 2 ) mod N and r 4 =
Revised: ( k 1 − k 2 ) mod N and r 4 =

Original: ( k 2 − k 1 ) mod N .
Revised: ( k 2 − k 1 ) mod N .

Original: Then we have 0 ≤ r 3 ≤ D − 1 and 0 < r 4 ≤ D − 1 .
Revised: Then we have 0 ≤ r 3 ≤ D − 1 and 0 < r 4 ≤ D − 1 .

Original: By the deﬁnition of modulo operation, There exist two integers q, q (cid:48) such that
Revised: By the deﬁnition of modulo operation, There exist two integers q, q (cid:48) such that

Original: [Equation]
Revised: [Equation]

Original: which imply
Revised: which imply

Original: [Equation]
Revised: [Equation]

Original: However, 0 < r 3 + r 4 ≤ 2 D − 2 contradicts with “ q ∈ Z , q
Revised: However, 0 < r 3 + r 4 ≤ 2 D − 2 contradicts with “ q ∈ Z , q

Original: [Equation]
Revised: [Equation]

Original: ∈ Z , N ∈ Z , N ≥ 2 D − 1 .”
Revised: ∈ Z , N ∈ Z , N ≥ 2 D − 1 .”

Original: Conclusion (b) is proved.
Revised: Conclusion (b) is proved.

Original: Conclusions (c) and (d) are actually the same with Conclusions (a) and (b) respectively.
Revised: Conclusions (c) and (d) are actually the same with Conclusions (a) and (b) respectively.

Original: Thus, it holds that
Revised: Thus, it holds that

Original: [Equation]
Revised: [Equation]

Original: and | L ( l 1 ) | =
Revised: and | L ( l 1 ) | =

Original: D, | ¯ L ( l 1 ) | =
Revised: D, | ¯ L ( l 1 ) | =

Original: D − 1 .
Revised: D − 1 .

Original: Lemma 3 is proved.
Revised: Lemma 3 is proved.

Original: With the preparations, we can prove Conclusion 1 of Theorem 3 now.
Revised: With the preparations, we can prove Conclusion 1 of Theorem 3 now.

Original: Proof of Theorem 3, Conclusion 1.
Revised: Proof of Theorem 3, Conclusion 1.

Original: Firstly we ﬁx k 1 ∈ { 0 , 1 , · · · , N − 1 } and consider k 2 ∈ K ( k 1 ) .
Revised: Firstly we ﬁx k 1 ∈ { 0 , 1 , · · · , N − 1 } and consider k 2 ∈ K ( k 1 ) .

Original: Let r k
Revised: Let r k

Original: =
Revised: =

Original: ( k 1 − k 2 ) mod N .
Revised: ( k 1 − k 2 ) mod N .

Original: Then equation (38) implies that, for any i ∈ I ( k 1 , k 2 ) , there exists a δ ( r k ≤ δ ≤ D − 1 ) such that
Revised: Then equation (37) implies that, for any i ∈ I ( k 1 , k 2 ) , there exists a δ ( r k ≤ δ ≤ D − 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: Now we consider another case for k 2 : k 2 ∈ ¯ K
Revised: Now we consider another case for k 2 : k 2 ∈ ¯ K

Original: ( k 1 ) , r k = ( k 2 − k 1 ) mod N .
Revised: ( k 1 ) , r k = ( k 2 − k 1 ) mod N .

Original: Equation (39) implies
Revised: Equation (38) implies

Original: that, for any i ∈ I ( k 1 , k 2 )
Revised: that, for any i ∈ I ( k 1 , k 2 )

Original: , there exists a δ ( r k ≤ δ ≤ D − 1 ) such that
Revised: , there exists a δ ( r k ≤ δ ≤ D − 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: [Equation] Similarly, for any l 1 ∈ { 0 , 1 , · · · , N − 1 } and l 2 ∈ L ( l 1 ) , we denote r l = ( l 1 − l 2 ) mod N .
Revised: Similarly, for any l 1 ∈ { 0 , 1 , · · · , N − 1 } and l 2 ∈ L ( l 1 ) , we denote r l = ( l 1 − l 2 ) mod N .

Original: For any j ∈ J ( l 1 , l 2 ) , there exists a δ ( r l ≤ δ ≤ D − 1 ) such that
Revised: For any j ∈ J ( l 1 , l 2 ) , there exists a δ ( r l ≤ δ ≤ D − 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: Another case for l 2 : l 2 ∈ ¯ L ( l 1 ) , r l = ( l 2 − l 1 ) mod N .
Revised: Another case for l 2 : l 2 ∈ ¯ L ( l 1 ) , r l = ( l 2 − l 1 ) mod N .

Original: For any j ∈ J ( l 1 , l 2 ) , there exists a δ ( r l ≤ δ ≤ D − 1 ) such that
Revised: For any j ∈ J ( l 1 , l 2 ) , there exists a δ ( r l ≤ δ ≤ D − 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: Now let us consider the following function.
Revised: Now let us consider the following function.

Original: By results in Lemmas 2 and 3, we have
Revised: By results in Lemmas 2 and 3, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Combining equations (40)
Revised: Combining equations (39)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (44) and (46)
Revised: (43) and (45)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining  (41), (42), (45) and (46), we obtain
Revised: Combining (40), (41), (44) and (45), we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining (40), (43), (44) and (47), we obtain
Revised: Combining (39), (42), (43) and (46), we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining (41), (43), (45) and (47), we obtain
Revised: Combining (40), (42), (44) and (46), we obtain

Original: [Equation]
Revised: [Equation]

Original: By the above explicit formulas of f i
Revised: By the above explicit formulas of f i

Original: [Equation]
Revised: [Equation]

Original: 1 ≤ i ≤
Revised: 1 ≤ i ≤

Original: [Equation]
Revised: [Equation]

Original: we have f
Revised: we have f

Original: [Equation]
Revised: [Equation]

Original: f 2 , f 3 , f 4 are all independent of k 1 , l 1 and N .
Revised: f 2 , f 3 , f 4 are all independent of k 1 , l 1 and N .

Original: They are only related with m 1 , m 2 for ﬁxed d and m .
Revised: They are only related with m 1 , m 2 for ﬁxed d and m .

Original: Thus, we are able to denote f ( k 1 , l 1 , m 1 , m 2 ) as f ( m 1 , m 2 ) for simplicity.
Revised: Thus, we are able to denote f ( k 1 , l 1 , m 1 , m 2 ) as f ( m 1 , m 2 ) for simplicity.

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: N − 1 (cid:88) k 1 =
Revised: N − 1 (cid:88) k 1 =

Original: N − 1 (cid:88) l 1 =
Revised: N − 1 (cid:88) l 1 =

Original: M (cid:88) m 1 =
Revised: M (cid:88) m 1 =

Original: M (cid:88) m 

Revised: M (cid:88) m

Original: = f
Revised: = f

Original: ( k 1 , l 1 , m 1 , m 2 ) =
Revised: ( k 1 , l 1 , m 1 , m 2 ) =

Original: N
Revised: N

Original: N − 1 (cid:88) k 1 =
Revised: N − 1 (cid:88) k 1 =

Original: N − 1 (cid:88) l 1 =
Revised: N − 1 (cid:88) l 1 =

Original: M (cid:88) m 1 =
Revised: M (cid:88) m 1 =

Original: M (cid:88) m 

Revised: M (cid:88) m

Original: = f
Revised: = f

Original: ( m 1 , m 2 ) =
Revised: ( m 1 , m 2 ) =

Original: N 2 · N 2 ·
Revised: N 2 · N 2 ·

Original: M (cid:88) m 1 =
Revised: M (cid:88) m 1 =

Original: M (cid:88) m 

Revised: M (cid:88) m

Original: = f
Revised: = f

Original: ( m 1 , m 2 ) =
Revised: ( m 1 , m 2 ) =

Original: M (cid:88) m 1 =
Revised: M (cid:88) m 1 =

Original: M (cid:88) m 

Revised: M (cid:88) m

Original: = f
Revised: = f

Original: ( m 1 , m 2 )
Revised: ( m 1 , m 2 )

Original: Thus, 1 N 2 (cid:107) ( D N cir ) T W N cir (cid:107) 2 F is dependent of N :
Revised: Thus, 1 N 2 (cid:107) ( D N cir ) T W N cir (cid:107) 2 F is dependent of N :

Original: [Equation]
Revised: [Equation]

Original: 48) which implies W N cir=
Revised: 47) which implies W N cir

Original:  W 2 D − 1 cir , ∀ N ≥ 2 D − 1 .
Revised: = W 2 D − 1 cir , ∀ N ≥ 2 D − 1 .

Original: C.P ROOF OF C ONCLUSION 2.
Revised: C.

Revised: [Equation] P ROOF OF C ONCLUSION

Original:  Before proving Conclusion 2, let us analyze the relationship between D N conv and D N + D − 1 cir .
Revised: [Equation] Before proving Conclusion 2, let us analyze the relationship between D N conv and D N + D − 1 cir .

Original: Similar to D cir , we use ( i, j ) as the row index and ( k, l, m ) as the column index of D conv .
Revised: Similar to D cir , we use ( i, j ) as the row index and ( k, l, m ) as the column index of D conv .

Original: For 0 ≤ i, j ≤ N − 1 , 1 ≤ m ≤ M ,
Revised: For 0 ≤ i, j ≤ N − 1 , 1 ≤ m ≤ M ,

Original: D N + D − 1 cir
Revised: [Equation] D − 1 cir

Original: [Equation]
Revised: [Equation]

Original: , k, l taken as others
Revised: l taken as others

Original: Matrix D N + D − 1 cir is of dimension ( N + D − 1) 2 × ( N + D − 1) 2 M , where 0 ≤ i, j ≤ N + D − 2 ; matrix D N conv is of dimension ( N ) 2 × ( N + D − 1) 2 M ,
Revised: [Equation] [Equation]

Original: where 0 ≤ i, j ≤ N − 1 . Thus, D N conv is a block in D N+ D − 1 cir , i.e.,
Revised: Matrix D N

Original: [Equation]
Revised: [Equation]

Original: The matrix ∆ N D  is of dimension (( N + D − 1) 2 − N  2 ) ×
Revised: D [Equation] is of dimension ( N + D − 1) 2 × ( N + D − 1) 2 M ,

Original:  ( N + D − 1) 2 M :
Revised: where 0 ≤ i, j ≤ N + D − 2 ; matrix D N conv is of dimension ( N ) 2 × ( N + D − 1) 2 M , where 0 ≤ i, j ≤ N − 1 . Thus, D N conv is a block in D N + D − 1 cir , i.e.,

Original: [Equation]
Revised: [Equation]

Original: 1 cir ( [Equation] [Equation] [Equation]
Revised: The matrix ∆ N D is of dimension (( N + D − 1) 2 − N 2 ) ×

Original: , :, :) (cid:3) , [Equation] [Equation]
Revised: ( N + D − 1) 2 M :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Revised: [Equation]

Original: W
Revised: ∈

Original: [Equation]
Revised: [Equation]

Revised: ∆ .

Original: Then,
Revised: Then,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: For any ( i, j ) ∈ I ∆ , one has
Revised: For any ( i, j ) ∈ I ∆ , one has

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Equation (36) implies that, for ( i, j ) ∈
Revised: Equation (35) implies that, for ( i, j ) ∈

Original: I 1 , 1 ≤ m ≤ M ,
Revised: I 1 , 1 ≤ m ≤ M ,

Original: D N + D − 1 cir ( i, j ; k, l, m )=
Revised: D N + D − 1 cir ( i, j ; k, l, m )

Original:  

Original: d m ( k − i, l − j ) , i

Original: [Equation]
Revised: [Equation]

Original: k [Equation] N
Revised: 0 , k, l taken as others

Original: [Equation] D [Equation] j 
Revised: Thus, for any ( i, j ) ∈ I 1 ,

Original: [Equation]
Revised: [Equation]

Original: d
Revised: cir

Original: [Equation]
Revised: [Equation]

Original: ( k [Equation] i+ N + D − 1 , l − j ) , 0 ≤ k ≤ i − N, j ≤ l ≤ j + D −0 , k, l taken as others Thus, for any ( i, j ) ∈ I 1 , (cid:107) D N
Revised: i

Original: [Equation]
Revised: [Equation]

Original: D − 1 cir ( i, j ; : , : , :) (cid:107) 22 = N + D − 2 (cid:88) k = N + D −
Revised: m

Original: [Equation]
Revised: [Equation]

Original: (cid:88) l [Equation] M (cid:88) m = (cid:12)(cid:12)D N + D − 1 cir ( i, j ; k, l, m ) (cid:12)(cid:12) 2 = (cid:107)
Revised: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Equation (50) is proved.
Revised: Equation (50) is proved.

Original: With the same argument, equation (51) is also proved.
Revised: With the same argument, equation (51) is also proved.

Original: Lemma 5.
Revised: Lemma 5.

Original: If N ≥ 2 D − 1 , we have
Revised: If N ≥ 2 D − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity
Revised: For simplicity

Original: [Equation]
Revised: [Equation]

Original: we denote two row vectors:
Revised: we denote two row vectors:

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: with the same argument in Lemma
Revised: with the same argument in Lemma

Original: [Equation]
Revised: [Equation]

Original: we have: d i 1 ,
Revised: we have: d i 1 ,

Original: [Equation]
Revised: [Equation]

Original: = 0 implies i
Revised: = 0 implies i

Original: 2 ∈ I (cid:48) ∆ (cid:44) { i | 0 ≤ ( i 1 − i ) mod( N + D − 1) ≤ D − 1 or 0 ≤ ( i − i 1 ) mod( N + D − 1)
Revised: 2 ∈ I (cid:48) ∆ (cid:44) { i | 0 ≤ ( i 1 − i ) mod( N + D − 1) ≤ D − 1 or 0 ≤ ( i − i 1 ) mod( N + D − 1)

Original: ≤ D − 1 } j 2 ∈ J (cid:48) ∆ (cid:44) { j | 0 ≤ ( j 1 − j ) mod( N + D − 1) ≤ D − 1 or 0 ≤ ( j − j 1 ) mod( N + D − 1) ≤ D − 1 }
Revised: ≤ D − 1 } j 2 ∈ J (cid:48) ∆ (cid:44) { j | 0 ≤ ( j 1 − j ) mod( N + D − 1) ≤ D − 1 or 0 ≤ ( j − j 1 ) mod( N + D − 1) ≤ D − 1 }

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where the inequality in the second line follows from (50) and (51).
Revised: where the inequality in the second line follows from (50) and (51).

Original: Inequality (52) is proved.
Revised: Inequality (52) is proved.

Original: With these preparations, we can prove Theorem 3, Conclusion 2 now.
Revised: With these preparations, we can prove Theorem 3, Conclusion 2 now.

Original: Proof of Theorem 3, Conclusion 2.
Revised: Proof of Theorem 3, Conclusion 2.

Original: Deﬁne set
Revised: Deﬁne set

Original: [Equation]
Revised: [Equation]

Original: Since d ∈ W normal , the set is nonempty:
Revised: Since d ∈ W normal , the set is nonempty:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne functions F N conv : R D 2 M → R , F N cir : R D 2 M → R .
Revised: Deﬁne functions F N conv : R D 2 M → R , F N cir : R D 2 M → R .

Original: F N conv (
Revised: F N conv (

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnitions of W N conv , W N cir , we have
Revised: By the deﬁnitions of W N conv , W N cir , we have

Original: [Equation]
Revised: [Equation]

Original: Step
Revised: Step

Original: 1: Proving F N conv ( w ) uniformly converges
Revised: 1: Proving F N conv ( w ) uniformly converges

Original: to
Revised: to

Original: F 2 D − 1 cir
Revised: F 2 D − 1 cir

Original: ( w ) on X ∩ W normal for any compact set
Revised: ( w ) on X ∩ W normal for any compact set

Original: X ⊂ R D 2 M .
Revised: X ⊂ R D 2 M .

Original: We arbitrarily choose such a compact set X .
Revised: We arbitrarily choose such a compact set X .

Original: Based on (48), (49) and (52), one has, for all w ∈
Revised: Based on (47), (49) and (52), one has, for all w ∈

Original: X ∩ W normal ,
Revised: X ∩ W normal ,

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: there exists a constant B >
Revised: there exists a constant B >

Original: [Equation]
Revised: [Equation]

Original: which is independent of N
Revised: which is independent of N

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Step 2: Proving F N conv ( w ) epigraphically converges 4 to F 2 D − 1 cir
Revised: Step 2: Proving F N conv ( w ) epigraphically converges 6 to F 2 D − 1 cir

Original: [Equation]
Revised: [Equation]

Original: We want to show, at each point w it holds that
Revised: We want to show, at each point w it holds that

Original: [Equation]
Revised: [Equation]

Original: sequence
Revised: sequence

Original: [Equation]
Revised: [Equation]

Original: →
Revised: →

Original: [Equation]
Revised: [Equation]

Original: Firstly, we prove (56).
Revised: Firstly, we prove (56).

Original: We arbitrarily pick a sequence { w N } ∞ N =0 such that w N → w .
Revised: We arbitrarily pick a sequence { w N } ∞ N =0 such that w N → w .

Original: If w / ∈ W normal , F 2 D − 1 cir ( w ) =
Revised: If w / ∈ W normal , F 2 D − 1 cir ( w ) =

Original: + ∞ .
Revised: + ∞ .

Original: Since W normal is a closed set, there exists a N + such that w N / ∈ W normal for all N ≥ N + .
Revised: Since W normal is a closed set, there exists a N + such that w N / ∈ W normal for all N ≥ N + .

Original: Thus, one has F N conv ( w N ) =
Revised: Thus, one has F N conv ( w N ) =

Original: + ∞ for all N ≥ N + , i.e.,
Revised: + ∞ for all N ≥ N + , i.e.,

Original: [Equation]
Revised: [Equation]

Original: If w ∈ W normal , two cases should be considered.
Revised: If w ∈ W normal , two cases should be considered.

Original: The ﬁrst case is that any subsequences of { w N } ∞ N =0 are not kept within W normal , i.e., there exists a N + such that w N / ∈ W normal for all N ≥ N + .
Revised: The ﬁrst case is that any subsequences of { w N } ∞ N =0 are not kept within W normal , i.e., there exists a N + such that w N / ∈ W normal for all N ≥ N + .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: The second case is that there exists a subsequence { w N k } ∞ k =

Revised: The second case is that there exists a subsequence { w N k } ∞ k =

Original: ⊂ { w N } ∞ N =0 such that
Revised: ⊂ { w N } ∞ N =0 such that

Original: [Equation]
Revised: [Equation]

Original: Since w N converges to w , any subsequences should be Cauchy.
Revised: Since w N converges to w , any subsequences should be Cauchy.

Original: Given any Cauchy sequence { w N k } ∞ k =0 in ﬁnite dimensional Euclidean space, there exists a compact set
Revised: Given any Cauchy sequence { w N k } ∞ k =0 in ﬁnite dimensional Euclidean space, there exists a compact set

Original: X such that
Revised: X such that

Original: [Equation]
Revised: [Equation]

Original: Let B (cid:48)
Revised: Let B (cid:48)

Original: =
Revised: =

Original: sup w ∈ X ∩W normal (cid:107) w (cid:107) .
Revised: sup w ∈ X ∩W normal (cid:107) w (cid:107) .

Original: By (55), we obtain | F N k
Revised: By (55), we obtain | F N k

Original: [Equation]
Revised: [Equation]

Original: w N
Revised: w N

Original: [Equation]
Revised: [Equation]

Original: −F
Revised: −

Original: [Equation]
Revised: [Equation]

Original: D−
Revised: D

Original: [Equation]
Revised: [Equation]

Original: For any (cid:15) > 0 , by the continuity of F 2 D − 1 cir , we are able to ﬁnd a K > 

Revised: For any (cid:15) > 0 , by the continuity of F 2 D − 1 cir , we are able to ﬁnd a K >

Original: such that | F 2 D − 1 cir ( w N k ) −
Revised: such that | F 2 D − 1 cir ( w N k ) −

Original: F 2 D − 1 cir ( w ) | < (cid:15) for all k ≥ K .
Revised: F 2 D − 1 cir ( w ) | < (cid:15) for all k ≥ K .

Original: Pick a K (cid:48) such that N K (cid:48) ≥ ( BB (cid:48) /(cid:15) ) 2 .
Revised: Pick a K (cid:48) such that N K (cid:48) ≥ ( BB (cid:48) /(cid:15) ) 2 .

Original: Then, for all k ≥ max( K, K (cid:48) ) , we have | F N k conv
Revised: Then, for all k ≥ max( K, K (cid:48) ) , we have | F N k conv

Original: [Equation]
Revised: [Equation]

Original: w N k
Revised: w N k

Original: [Equation]
Revised: [Equation]

Original: The above conclusion holds for all subsequences { w N k } ∞ k =

Revised: The above conclusion holds for all subsequences { w N k } ∞ k =

Original: ⊂ W normal .
Revised: ⊂ W normal .

Original: F 2 D − 1 cir ( w ) is an accumulation point of { F N conv ( w N ) } ∞ N =0 .
Revised: F 2 D − 1 cir ( w ) is an accumulation point of { F N conv ( w N ) } ∞ N =0 .

Original: All the other accumulation points of { F N conv ( w N ) } ∞ N =0 must be + ∞ because F N conv ( w ) =
Revised: All the other accumulation points of { F N conv ( w N ) } ∞ N =0 must be + ∞ because F N conv ( w ) =

Original: F 2 D − 1 cir
Revised: F 2 D − 1 cir

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Secondly, we prove (57).
Revised: Secondly, we prove (57).

Original: We set w N = w for all N = 0 , 1 , 2 , · · · .
Revised: We set w N = w for all N = 0 , 1 , 2 , · · · .

Original: Then (57) is a direct result of (55).
Revised: Then (57) is a direct result of (55).

Original: Step 3: proving (26).
Revised: Step 3: proving (25).

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: We want to show that G ( w ) is strongly convex.
Revised: We want to show that G ( w ) is strongly convex.

Original: Let ˜ w i ∈ R (2 D − 1) 2 be the i th column of W 2 D − 1 cir , i.e.,
Revised: Let ˜ w i ∈ R (2 D − 1) 2 be the i th column of W 2 D − 1 cir , i.e.,

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Let ˜ w ∈ R
Revised: Let ˜ w ∈ R

Original: (2 D − 1) 

Revised: (2 D − 1)

Original: M vectorize W 2 D − 1 cir , i.e.,
Revised: M vectorize W 2 D − 1 cir , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Then G ( w ) can be written as a quadratic form of ˜ w :
Revised: Then G ( w ) can be written as a quadratic form of ˜ w :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: As long as at least one of the matrices { D 2 D − 1 cir , 0 , · · · , D 2 D − 1 cir ,M − 1 } is non-singular, D 2 D − 1 cir is full row rank, which implies that D 2 D − 1 cir ( D 2 D − 1 cir ) T is non-singular.
Revised: As long as at least one of the matrices { D 2 D − 1 cir , 0 , · · · , D 2 D − 1 cir ,M − 1 } is non-singular, D 2 D − 1 cir is full row rank, which implies that D 2 D − 1 cir ( D 2 D − 1 cir ) T is non-singular.

Original: Then Q is positive deﬁnite.
Revised: Then Q is positive deﬁnite.

Original: The transform between w and ˜ w is linear.
Revised: The transform between w and ˜ w is linear.

Original: We denote the transform as T , i.e., ˜ w = T w .
Revised: We denote the transform as T , i.e., ˜ w = T w .

Original: It’s trivial that (cid:107) ˜ w (cid:107) 22 = 0 implies (cid:107) W 2 D − 1 cir (cid:107) 2 F = 0 .
Revised: It’s trivial that (cid:107) ˜ w (cid:107) 22 = 0 implies (cid:107) W 2 D − 1 cir (cid:107) 2 F = 0 .

Original: By the deﬁnition of W 2 D − 1 cir , (cid:107)
Revised: By the deﬁnition of W 2 D − 1 cir , (cid:107)

Original: [Equation]
Revised: [Equation]

Original: implies (cid:107) w (cid:107) 22 = 0 .
Revised: implies (cid:107) w (cid:107) 22 = 0 .

Original: Thus, linear operator T is full column rank.
Revised: Thus, linear operator T is full column rank.

Original: Thus, T T QT is positive deﬁnite, and
Revised: Thus, T T QT is positive deﬁnite, and

Original: [Equation]
Revised: [Equation]

Original: is strongly convex.
Revised: is strongly convex.

Original: Then F 2 D − 1 cir
Revised: Then F 2 D − 1 cir

Original: [Equation]
Revised: [Equation]

Original: has only one minimizer, i.e., W 2 D
Revised: has only one minimizer, i.e., W 2 D

Original: [Equation]
Revised: [Equation]

Original: cir involves only a unique element.
Revised: cir involves only a unique element.

Original: Now we check the conditions of Propositions 7.32(c) and 7.33 in (Rockafellar & Wets, 2009) to apply them.
Revised: Now we check the conditions of Propositions 7.32(c) and 7.33 in (Rockafellar & Wets, 2009) to apply them.

Original: F N conv e −→ F 2 D − 1 cir .
Revised: F N conv e −→ F 2 D − 1 cir .

Original: This is proved in Step 2.
Revised: This is proved in Step 2.

Original: F 2 D − 1 cir is level bounded.
Revised: F 2 D − 1 cir is level bounded.

Original: Since G ( w ) is strongly convex, F 2 D − 1 cir ( w ) =
Revised: Since G ( w ) is strongly convex, F 2 D − 1 cir ( w [Equation] =

Original: (cid:112) G ( w ) + ι W normal ( w ) must be level bounded.
Revised: [Equation] G [Equation] + ι W normal ( w ) must be level bounded.

Original: F 2 D − 1 cir (cid:54)≡ + ∞ .
Revised: F 2 D − 1 cir (cid:54)≡ + ∞ .

Original: Since W normal is nonempty (54), dom F 2 D − 1 cir (cid:54) = ∅ , F 2 D − 1 cir is not constantly + ∞ .
Revised: Since W normal is nonempty (54), dom F 2 D − 1 cir (cid:54) = ∅ , F 2 D − 1 cir is not constantly + ∞ .

Original: All the level set of F N conv are connected.
Revised: All the level set of F N conv are connected.

Original: This can be derived from the convexity of F N conv .
Revised: This can be derived from the convexity of F N conv .

Original: F 2 D − 1 cir and F N conv are all lower semi-continuous and proper.
Revised: F 2 D − 1 cir and F N conv are all lower semi-continuous and proper.

Original: This condition follows from the fact that the functions F 2 D − 1 cir and F N conv are all continuous functions deﬁned on a nonempty closed convex domain W normal .
Revised: This condition follows from the fact that the functions F 2 D − 1 cir and F N conv are all continuous functions deﬁned on a nonempty closed convex domain W normal .

Original: Applying Proposition 7.32(c), we have { F N conv } is eventually level bounded.
Revised: Applying Proposition 7.32(c), we have { F N conv } is eventually level bounded.

Original: If we arbitrarily pick a w N ∈ W N conv and let w cir be the unique point in W 2 D − 1 cir .
Revised: If we arbitrarily pick a w N ∈ W N conv and let w cir be the unique point in W 2 D − 1 cir .

Original: Applying Proposition 7.33, we have w N → w cir .
Revised: Applying Proposition 7.33, we have w N → w cir .

Original: By Deﬁnition 4.1 in (Rockafellar & Wets, 2009), we obtain the convergence of the sequence of sets {W N conv } : lim N →∞ W N conv =
Revised: By Deﬁnition 4.1 in (Rockafellar & Wets, 2009), we obtain the convergence of the sequence of sets {W N conv } : lim N →∞ W N conv =

Original: W 2 D − 1 cir .
Revised: W 2 D − 1 cir .

Revised: D D ISCUSSION OF D EFINITION 2 (11)

Revised: In this section, we want to numerically show that, given typical D and s , there is a ¯ σ min > 0 such that a random generated matrix W ∈ ¯ W ( D , s, ¯ σ min ) .

Original:  D A N EFFICIENT ALGORITHM TO SOLVE (25)
Revised: However, given D and W , it’s intractable to completely check (11): [Equation] s. The reason is that there are extremely large amount of possible S s. For example, we take M = 250 , N = 500 , s = 50 .

Revised: There are totally

Revised: [Equation]

Revised: possible S s satisfying 2 ≤ | S | ≤ s .

Revised: It’s impossible to check (11) on all possible S s.

Revised: Instead of checking all possible S s, we sample 5000 S s from the whole set:

Revised: [Equation]

Revised: where S (cid:48) is the set of all the samples.

Revised: Then we estimate ¯ σ min with the following quantity:

Revised: [Equation]

Revised: Furthermore, we use the same D as that in Section 5 and generate 1000 W s with each entry i.i.d sampled from the normal distribution.

Revised: Then we normalize each column of the generated W s. This technique is commonly used in sparse coding.

Original: In this section, we introduce an algorithm to solve (25) (we copy (25) below to facilitate reading): min
Revised: Finally, we report the distribution of ¯ σ (cid:48) ( D , W ) with the ﬁxed D and the 1000 sampled W s in Figure 5. Figure 5 demonstrates that, with the ﬁxed D , most of the random generated W s have a ¯ σ (cid:48) ( D , W ) within the interval [0 . 25 , 0 . 35] .

Original: [Equation] By the deﬁnition of the Frobenius norm, (cid:107) W T D (cid:107) 2 F =
Revised: Thus, the numerical results support our claim: with high probability, a random generated W satisﬁes

Original: [Equation]
Revised: [Equation]

Original: ( W T D  ) T
Revised: that is, W ∈ ¯ W ( D , s, ¯ σ min ) .

Original: [Equation] F [Equation]
Revised: E A N EFFICIENT ALGORITHM TO SOLVE (24)

Original: D
Revised: In this section, we introduce an algorithm to solve (24) (we copy (24) below to facilitate reading): min

Original: [Equation]
Revised: [Equation]

Original: W
Revised: By the deﬁnition of the Frobenius norm,

Original: [Equation]
Revised: [Equation]

Original: F .
Revised: F .

Original: Thus, the above problem is equivalent with
Revised: Thus, the above problem is equivalent with

Original: [Equation]
Revised: [Equation]

Original: Since the circular convolution is very efﬁcient to calculate in the frequency domain, we consider solving (58) utilizing the fast Fourier transform (FFT).
Revised: Since the circular convolution is very efﬁcient to calculate in the frequency domain, we consider solving (58) utilizing the fast Fourier transform (FFT).

Original: Firstly, we introduce the operators D N cir ( d ) , W N cir ( w ) in the frequency domain.
Revised: Firstly, we introduce the operators D N cir ( d ) , W N cir ( w ) in the frequency domain.

Original: To simplify the notation, we denote the operators as D N cir and W N cir respectively.
Revised: To simplify the notation, we denote the operators as D N cir and W N cir respectively.

Original: Let F be the FFT operator.
Revised: Let F be the FFT operator.

Original: Thus, b = D N cir x is equivalent with
Revised: Thus, b = D N cir x is equivalent with

Original: [Equation]
Revised: [Equation]

Original: Let ˆ b = F b , ˆ x = F x be the frequency domain signals, let ˆ D N cir
Revised: Let ˆ b = F b , ˆ x = F x be the frequency domain signals, let ˆ D N cir

Original: = F D N cir F H be the frequency domain operator.
Revised: = F D N cir F H be the frequency domain operator.

Original: The above equation is:
Revised: The above equation is:

Original: [Equation]
Revised: [Equation]

Original: b = ˆ D N cir ˆ [Equation] The frequency domain operator ˆ D N cir is much cheaper to calculate than the operator D N cir in the spacial domain because it is block diagonal (Wohlberg, 2016).
Revised: The frequency domain operator ˆ D N cir is much cheaper to calculate than the operator D N cir in the spacial domain because it is block diagonal (Wohlberg, 2016).

Original: Speciﬁcally, we zero pad d to N × N and do FFT: ˆ d m = FFT (cid:0) zeropad( d m , N − D ) (cid:1) , then the above operator can be explicitly written as:
Revised: Speciﬁcally, we zero pad d to N × N and do FFT: ˆ d m = FFT (cid:0) zeropad( d m , N − D ) (cid:1) , then the above operator can be explicitly written as:

Original: [Equation]
Revised: [Equation]

Revised: where ¯ · means complex conjugate.

Revised: This is due to D N cir is actually cross-correlation, not convolution (see (18)).

Revised: Cross-correlation is equal to the transpose of convolution.

Revised: Thus, there should be complex conjugate in the frequency domain.

Original: Further, since
Revised: Further, since

Original: [Equation]
Revised: [Equation]

Original: problem (58) is equivalent with
Revised: problem (58) is equivalent with

Original: [Equation]
Revised: [Equation]

Original: which can be efﬁciently solved by the frequency domain FISTA in (Liu et al., 2017).
Revised: which can be efﬁciently solved by the frequency domain FISTA in (Liu et al., 2017).

Original: The details are outlined in Algorithm 1.
Revised: The details are outlined in Algorithm 1.

Original: Input: Dictionary
Revised: Input: Dictionary

Original: [Equation]
Revised: [Equation]

Original: Initialize: Let w 0 =
Revised: Initialize: Let w 0 =

Original: d , w 0 aux = d , γ 

Revised: d , w 0 aux = d , γ

Original: = 1 .
Revised: = 1 .

Original: 1 for j = 0 , 1 , 2 , . . .
Revised: 1 for j = 0 , 1 , 2 , . . .

Original: until convergence do
Revised: until convergence do

Original: Zeropad and FFT:
Revised: Zeropad and FFT:

Original: [Equation]
Revised: [Equation]

Original: Compute frequency domain gradient:
Revised: Compute frequency domain gradient:

Original: [Equation]
Revised: [Equation]

Original: where ¯ · represents the conjugate of a complex number.
Revised: where ¯ · represents the conjugate of a complex number.

Original: Compute the next iterate:
Revised: Compute the next iterate:

Original: [Equation]
Revised: [Equation]

Original: where the set W normal is deﬁned in (53).
Revised: where the set W normal is deﬁned in (53).

Original: Let γ j +1 = (cid:0) 1 + (cid:112) 1 + 4( γ j ) 2 (cid:1) / 2 , then compute the auxiliary variable:
Revised: Let γ j +1 = (cid:0) 1 + (cid:112) 1 + 4( γ j ) 2 (cid:1) / 2 , then compute the auxiliary variable:

Original: [Equation]
Revised: [Equation]

Original: w j ) .

Original: 6 end
Revised: 6 end

Original: Output: w J , where J is the last iterate.
Revised: Output: w J , where J is the last iterate.

Original: E A LGORITHM D ETAILS OF T RAINING R OBUST ALISTA E. M ODEL A RCHITECTURE
Revised: F A LGORITHM D ETAILS OF T RAINING R OBUST ALISTA F. M ODEL A RCHITECTURE

Original: Inspired by the a similar unrolling and truncating fashion in LISTA, we can approximately solve the coherence minimization problem (17) using a similar ﬁnite-layer neural network that is unfolded from iterative algorithms.
Revised: Inspired by the a similar unrolling and truncating fashion in LISTA, we can approximately solve the coherence minimization problem (16) using a similar ﬁnite-layer neural network that is unfolded from iterative algorithms.

Original: Because the linear constraints in (17) are hard to enforce in deep neural networks, we ﬁrst relax it to the following form:
Revised: Because the linear constraints in (16) are hard to enforce in deep neural networks, we ﬁrst relax it to the following form:

Original: [Equation]
Revised: [Equation]

Original: (59) where (cid:12) is the Hadamard product and Q is a weight matrix that put more penalty on errors on diagonals, because entries on the diagonal will be far smaller than off-diagonal.
Revised: where (cid:12) is the Hadamard product and Q is a weight matrix that put more penalty on errors on diagonals, because entries on the diagonal will be far smaller than off-diagonal.

Original: The above relaxed coherence minimization can be solved using the gradient descent algorithm:
Revised: The above relaxed coherence minimization can be solved using the gradient descent algorithm:

Original: [Equation]
Revised: [Equation]

Original: By unfolding (60) and truncate to K steps, and considering the γ ( k ) D T outside the residual as learnable parameters B , we will have a deep neural network W = E
Revised: By unfolding (60) and truncate to K steps, and considering the γ ( k ) D T outside the residual as learnable parameters B , we will have a deep neural network W = E

Original: ( D ) as a coherence minimizer.
Revised: ( D ) as a coherence minimizer.

Original: We call it a Stage 1 encoder as it encodes a dictionary D into a weight matrix, that can be used in the Stage 2 of ALISTA, refered as a decoder .
Revised: We call it a Stage 1 encoder as it encodes a dictionary D into a weight matrix, that can be used in the Stage 2 of ALISTA, refered as a decoder .

Original: One layer of this model is shown in Fig. 5(a).
Revised: One layer of this model is shown in Fig. 6(a).

Original: The illustration of the whole feed-forward robust model is shown in Fig. 5(b).
Revised: The illustration of the whole feed-forward robust model is shown in Fig. 6(b).

Original: The two parts, the encoder and the decoder, can be jointly trained to gain the most from data-driven learning.
Revised: The two parts, the encoder and the decoder, can be jointly trained to gain the most from data-driven learning.

Original: We further adopt pre-training and curriculum learning to stabilize training, as to be discussed below.[Figure]
Revised: We further adopt pre-training and curriculum learning to stabilize training, as to be discussed below.

Original: E. M ODEL T RAINING
Revised: F. M ODEL T RAINING

Original: To stabilize the training process, we train the model in two stages: the pre-training stage and curriculum (joint) training stage.
Revised: To stabilize the training process, we train the model in two stages: the pre-training stage and curriculum (joint) training stage.

Revised: [Figure]

Original: Pre-Training Stage.
Revised: Pre-Training Stage.

Original: We ﬁrst pre-train the encoder and the decoder individually.
Revised: We ﬁrst pre-train the encoder and the decoder individually.

Original: The pre-training of the decoder, e.g., ALISTA, follows the standard training procedure in Section 5.1, without bothering the perturbations of D .
Revised: The pre-training of the decoder, e.g., ALISTA, follows the standard training procedure in Section 5.1, without bothering the perturbations of D .

Original: On the other hand, the encoder will always see perturbed dictionaries ˜
Revised: On the other hand, the encoder will always see perturbed dictionaries ˜

Original: D = D + ε D , where ε D ’s entries are sampled from i.i.d.
Revised: D = D + ε D , where ε D ’s entries are sampled from i.i.d.

Original: normal distribution with zero mean and σ 2 pre variance, and update its weight to minimize loss function deﬁned by (59).
Revised: normal distribution with zero mean and σ 2 pre variance, and update its weight to minimize loss function deﬁned by (59).

Original: The σ pre is a hyperparameter that we manually select for the pre-training stage, with a default value of 0.01.
Revised: The σ pre is a hyperparameter that we manually select for the pre-training stage, with a default value of 0.01.

Original: Curriculum (Joint) Training Stage.
Revised: Curriculum (Joint) Training Stage.

Original: After the pre-training stage, we concatenate these two parts and do joint training.
Revised: After the pre-training stage, we concatenate these two parts and do joint training.

Original: However, a direct end-to-end tuning was observed to cause much instability, due to the randomness in weights.
Revised: However, a direct end-to-end tuning was observed to cause much instability, due to the randomness in weights.

Original: Inspired by the curriculum learning technique, we ﬁrst perturb the dictionaries with smaller standard deviations and gradually increase the perturbation level during training.
Revised: Inspired by the curriculum learning technique, we ﬁrst perturb the dictionaries with smaller standard deviations and gradually increase the perturbation level during training.

Original: Speciﬁcally, starting from a small standard deviation σ t = σ 0 , the curriculum joint training procedure repeats the routine below:
Revised: Speciﬁcally, starting from a small standard deviation σ t = σ 0 , the curriculum joint training procedure repeats the routine below:

Original: • First uniformly sample a batch of standard deviations { σ i } B D i =1 from [0 , σ t ] , where B D is the batch size for perturbations of the original dictionary D .
Revised: • First uniformly sample a batch of standard deviations { σ i } B D i =1 from [0 , σ t ] , where B D is the batch size for perturbations of the original dictionary D .

Original: Use the sampled standard deviations to sample B D perturbations and apply to D and then normalized them to get { ˜ D } B D i =1 .
Revised: Use the sampled standard deviations to sample B D perturbations and apply to D and then normalized them to get { ˜ D } B D i =1 .

Original: • Sample a batch of sparse codes { x j } B x j =1 from a pre-deﬁned Gaussian-Bernoulli distribution; the supports of the sparse codes are decided i.i.d.
Revised: • Sample a batch of sparse codes { x j } B x j =1 from a pre-deﬁned Gaussian-Bernoulli distribution; the supports of the sparse codes are decided i.i.d.

Original: by a Bernoulli distribution to have around 10% non-zero entries; and the magnitudes are sampled from i.i
Revised: by a Bernoulli distribution to have around 10% non-zero entries; and the magnitudes are sampled from i.i.d.

Original: d. standard Gaussian.
Revised: standard Gaussian.

Original: B x is the batch size for sparse codes in training.
Revised: B x is the batch size for sparse codes in training.

Original: • Measure y i,j = ˜ D
Revised: • Measure y i,j = ˜ D

Original: i x j .
Revised: i x j .

Original: Then ( y i,j , x j , ˜ D
Revised: Then ( y i,j , x j , ˜ D

Original: i ) forms a tripelet of training sample.
Revised: i ) forms a tripelet of training sample.

Original: Note that only Robust ALISTA needs ˜ D
Revised: Note that only Robust ALISTA needs ˜ D

Original: i as part of the training samples.
Revised: i as part of the training samples.

Original: • Feed in the data and update the encoder and decoder with learning rates α e and α d , using the Adam Optimizer, respectively.
Revised: • Feed in the data and update the encoder and decoder with learning rates α e and α d , using the Adam Optimizer, respectively.

Original: • Increase σ t to the next larger value, after C training batches.
Revised: • Increase σ t to the next larger value, after C training batches.

Original: Repeat the above steps, until the value of σ t exceeds the pre-deﬁned σ max , that represents the maximal standard deviation to sample the dictionary perturbation.
Revised: Repeat the above steps, until the value of σ t exceeds the pre-deﬁned σ max , that represents the maximal standard deviation to sample the dictionary perturbation.

Original: In the experiment, we have B D = 4 , B x = 16 , C = 50000, α e = 10 − 6 , α d = 10 − 4 and σ max ∈ { 0 . 02 , 0 . 03 } and σ t is obtained by linearly interpolating between [0 , σ max ] for L − 1 times.
Revised: In the experiment, we have B D = 4 , B x = 16 , C = 50000, α e = 10 − 6 , α d = 10 − 4 and σ max ∈ { 0 . 02 , 0 . 03 } and σ t is obtained by linearly interpolating between [0 , σ max ] for L − 1 times.

Original: We choose L = 5 ; hence σ t takes i 5 σ max , i = 1 , 2 , . . .
Revised: We choose L = 5 ; hence σ t takes i 5 σ max , i = 1 , 2 , . . .

Original: , 5 in order.
Revised: , 5 in order.

Original: F V ISUALIZATION OF THE ANALYTIC CONVOLUTIONAL WEIGHTS
Revised: G V ISUALIZATION OF THE ANALYTIC CONVOLUTIONAL WEIGHTS

Original: Fig.
Revised: Fig.

Original: 6 visualizes the dictionary d ( 7 × 7 × 64 ) and the weights ˜ w ∈ W 13cir , used in the convolutional
Revised: 7 visualizes the dictionary d ( 7 × 7 × 64 ) and the weights ˜ w ∈ W 13cir , used in the convolutional

Original: A-LISTA simulation of Section 5.2.
Revised: A-LISTA simulation of Section 5.2.

Original: [Figure]

Original: G R ESULTS OF N ATURAL I MAGE D ENOISING U SING C ONV ALISTA
Revised: H R ESULTS OF N ATURAL I MAGE D ENOISING U SING C ONV ALISTA

Original: The natural image denoising experiment is conducted on the same BSD 500 dataset using the 400image training set, 50-image validation set and 50-image test set.
Revised: The natural image denoising experiment is conducted on the same BSD 500 dataset using the 400image training set, 50-image validation set and 50-image test set.

Original: We convert them all to grayscale,  and then add σ = 20 Gaussian i.i.d.
Revised: We convert them all to grayscale, [Figure] and then add σ = 20 Gaussian i.i.d.

Original: noise.
Revised: noise.

Original: We train both Conv LISTA (i.e., model (21)) and Conv
Revised: We train both Conv LISTA (i.e., model (20)) and Conv

Original: ALISTA (i.e., model (27)).
Revised: ALISTA (i.e., model (26)).

Original: Both networks have 5 layers, with the same dictionary D obtained from the training set by solving (25).
Revised: Both networks have 5 layers, with the same dictionary D obtained from the training set by solving (24).

Original: We reconstruct the denoised images using by convolving the learned feature maps with the original dictionary D .
Revised: We reconstruct the denoised images using by convolving the learned feature maps with the original dictionary D .

Original: The mean-square-error (MSE) between denoised and clean images are adopted as the network training loss, as inspired by (Zhou et al., 2018).
Revised: The mean-square-error (MSE) between denoised and clean images are adopted as the network training loss, as inspired by (Zhou et al., 2018).

Original: Six popular benchmark images (adding σ = 20 noise) are tested and reported in Table 3.
Revised: Six popular benchmark images (adding σ = 20 noise) are tested and reported in Table 4.

Original: The APSNR denotes the average PSNR over all images and the A-Times represents the average inference time (in seconds) for denoising one image.
Revised: The APSNR denotes the average PSNR over all images and the A-Times represents the average inference time (in seconds) for denoising one image.

Original: We compare Conv LISTA and Conv ALISTA, as well as the classical KSVD denoising algorithm
Revised: We compare Conv LISTA and Conv ALISTA, as well as the classical KSVD denoising algorithm

Original: (Elad & Aharon, 2006) and the recent CSC denoising algorithm with gradient regularization (CSC-GR) (Wohlberg, 2018).
Revised: (Elad & Aharon, 2006) and the recent CSC denoising algorithm with gradient regularization (CSC-GR) (Wohlberg, 2018).

Original: The results show that Conv
Revised: The results show that Conv

Original: LISTA and Conv ALISTA (without heavy tuning done for their optimal performance) can perform comparably with KSVD and outperforms CSC-GR, but with tremendously faster inference speeds than KSVD/CSC-GR.
Revised: LISTA and Conv ALISTA (without heavy tuning done for their optimal performance) can perform comparably with KSVD and outperforms CSC-GR, but with tremendously faster inference speeds than KSVD/CSC-GR.

Original: More importantly, Conv LISTA and Conv ALISTA only have marginal performance differences, validating again the analytic weights in convolutional cases.
Revised: More importantly, Conv LISTA and Conv ALISTA only have marginal performance differences, validating again the analytic weights in convolutional cases.

Original: [Table]
Revised: [Table]