Original: S PARSE C ODING WITH G ATED L EARNED ISTA
Revised: S PARSE C ODING WITH G ATED L EARNED ISTA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.
Revised: In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.

Original: Following assumptions made by prior works, we ﬁrst discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.
Revised: Following assumptions made by prior works, we ﬁrst discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.

Original: Speciﬁc design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.
Revised: Speciﬁc design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.

Original: In addition to the gain gates, we further introduce overshoot gates for compensating insufﬁcient step size in LISTA.
Revised: In addition to the gain gates, we further introduce overshoot gates for compensating insufﬁcient step size in LISTA.

Original: Extensive empirical results conﬁrm our theoretical ﬁndings and verify the effectiveness of our method.
Revised: Extensive empirical results conﬁrm our theoretical ﬁndings and verify the effectiveness of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Sparse coding serves as the foundation of many machine learning applications, e.g., the direction-ofarrival estimation (Xu et al., 2012), signal denoising (Elad & Aharon, 2006), and super resolution imaging (Yang et al., 2010).
Revised: Sparse coding serves as the foundation of many machine learning applications, e.g., the direction-ofarrival estimation (Xu et al., 2012), signal denoising (Elad & Aharon, 2006), and super resolution imaging (Yang et al., 2010).

Original: In general, it aims to recover an inherently sparse vector x s ∈ R n from an observation y ∈ R m corrupted by a noise vector
Revised: In general, it aims to recover an inherently sparse vector x s ∈ R n from an observation y ∈ R m corrupted by a noise vector

Original: ε ∈ R m .
Revised: ε ∈ R m .

Original: That is, y =
Revised: That is, y =

Original: Ax s + ε, (1) in which A ∈ R m × n is an over-complete basis matrix.
Revised: Ax s + ε, (1) in which A ∈ R m × n is an over-complete basis matrix.

Original: The problem of recovering x s , however, is a challenging task, in which the main difﬁculties are to incorporate the sparse constraint which is nonconvex and to further determine the indices of its non-zero elements, i.e., the support of the vector.
Revised: The problem of recovering x s , however, is a challenging task, in which the main difﬁculties are to incorporate the sparse constraint which is nonconvex and to further determine the indices of its non-zero elements, i.e., the support of the vector.

Original: A reasonable solution to the problem is to use convex functions as surrogates to relax the constraint of sparsity, among which the most classical one probably is the l 1 -norm penalty.
Revised: A reasonable solution to the problem is to use convex functions as surrogates to relax the constraint of sparsity, among which the most classical one probably is the l 1 -norm penalty.

Original: Such a problem is carefully studied in Lasso (Tibshirani, 1996), and it can be solved via least angle regression (Efron et al., 2004), the iterative shrinkage and thresholding algorithm (ISTA) (Daubechies et al., 2004), etc.
Revised: Such a problem is carefully studied in Lasso (Tibshirani, 1996), and it can be solved via least angle regression (Efron et al., 2004), the iterative shrinkage and thresholding algorithm (ISTA) (Daubechies et al., 2004), etc.

Original: Despite the simplicity, these conventional solvers suffer from critical shortcomings.
Revised: Despite the simplicity, these conventional solvers suffer from critical shortcomings.

Original: Taking ISTA as an example, we know that 1) it converges very slowly with only a sublinear rate (Beck & Teboulle, 2009), 2) the correlation between each of the two columns of A should be relatively low.
Revised: Taking ISTA as an example, we know that 1) it converges very slowly with only a sublinear rate (Beck & Teboulle, 2009), 2) the correlation between each of the two columns of A should be relatively low.

Original: In recent years, deep learning (LeCun et al., 2015) methods have achieved remarkable successes.
Revised: In recent years, deep learning (LeCun et al., 2015) methods have achieved remarkable successes.

Original: Deep neural networks (DNNs) have been proven both effective and efﬁcient in dealing with many tasks, including image classiﬁcation (He et al., 2016), object detection (Girshick, 2015), speech recognition (Hinton et al., 2012), and also sparse coding (Gregor & LeCun, 2010; Wang et al., 2016; Borgerding et al., 2017;
Revised: Deep neural networks (DNNs) have been proven both effective and efﬁcient in dealing with many tasks, including image classiﬁcation (He et al., 2016), object detection (Girshick, 2015), speech recognition (Hinton et al., 2012), and also sparse coding (Gregor & LeCun, 2010; Wang et al., 2016; Borgerding et al., 2017;

Original: He et al., 2017; Zhang & Ghanem, 2018; Chen et al., 2018; Liu et al., 2019; Sulam et al., 2019).
Revised: He et al., 2017; Zhang & Ghanem, 2018; Chen et al., 2018; Liu et al., 2019; Sulam et al., 2019).

Original: The core idea behind deep learning-based sparse coding is to train DNNs to approximate the optimal sparse code.
Revised: The core idea behind deep learning-based sparse coding is to train DNNs to approximate the optimal sparse code.

Original: For instance, an initial work of Gregor and LeCun’s (2010) takes the inspiration from
Revised: For instance, an initial work of Gregor and LeCun’s (2010) takes the inspiration from

Original: ISTA and develops an approximator named learned ISTA (LISTA), which is structurally similar to a recurrent neural network (RNN).
Revised: ISTA and develops an approximator named learned ISTA (LISTA), which is structurally similar to a recurrent neural network (RNN).

Original: It has been demonstrated both empirically and theoretically that LISTA is superior to ISTA (Wang et al., 2016; Moreau & Bruna, 2017; Giryes et al., 2018; Chen et al., 2018).
Revised: It has been demonstrated both empirically and theoretically that LISTA is superior to ISTA (Wang et al., 2016; Moreau & Bruna, 2017; Giryes et al., 2018; Chen et al., 2018).

Original: Nevertheless, it is also uncontroversial that there exists much room for further enhancing it.
Revised: Nevertheless, it is also uncontroversial that there exists much room for further enhancing it.

Original: In this paper, we delve deeply into the foundation of (L)ISTA and discover possible weaknesses of LISTA.
Revised: In this paper, we delve deeply into the foundation of (L)ISTA and discover possible weaknesses of LISTA.

Original: First and foremost, we know from prior arts (Chen et al., 2018; Liu et al., 2019) that LISTA tends to learn large enough biases to achieve no “false positive” in the support of generated codes and further ensure linear convergence, and we prove that this tendency, however, also makes the magnitude of the code components being lower than that of the ground-truth.
Revised: First and foremost, we know from prior arts (Chen et al., 2018; Liu et al., 2019) that LISTA tends to learn large enough biases to achieve no “false positive” in the support of generated codes and further ensure linear convergence, and we prove that this tendency, however, also makes the magnitude of the code components being lower than that of the ground-truth.

Original: That said, there probably exists a requirement of gains in the code estimations.
Revised: That said, there probably exists a requirement of gains in the code estimations.

Original: Second, regarding the optimization procedure of ISTA as to minimize an upper bound of its objective function at each step, we conjecture that the element-wise update of (L)ISTA normally
Revised: Second, regarding the optimization procedure of ISTA as to minimize an upper bound of its objective function at each step, we conjecture that the element-wise update of (L)ISTA normally

Original: “lags behind” the optimal solution, which suggests that it requires overshoots to reach the optimum, just like what has been suggested in fast ISTA (FISTA) (Beck & Teboulle, 2009) and learned FISTA (LFISTA) (Moreau & Bruna, 2017).
Revised: “lags behind” the optimal solution, which suggests that it requires overshoots to reach the optimum, just like what has been suggested in fast ISTA (FISTA) (Beck & Teboulle, 2009) and learned FISTA (LFISTA) (Moreau & Bruna, 2017).

Original: In this paper, our main contributions are summarized as follows:
Revised: In this paper, our main contributions are summarized as follows:

Original: • We discover weaknesses of LISTA by theoretically analyzing its optimization procedure, for mitigating which we introduce gain gates and overshoot gates, akin to update gate and reset gate mechanisms in the gated recurrent unit (GRU) Cho et al.
Revised: • We discover weaknesses of LISTA by theoretically analyzing its optimization procedure, for mitigating which we introduce gain gates and overshoot gates, akin to update gate and reset gate mechanisms in the gated recurrent unit (GRU) Cho et al.

Original: • We provide convergence analyses for LISTA (with or without gates), which further give rise to conditions on which the performance of our method with gain gates can be guaranteed.
Revised: • We provide convergence analyses for LISTA (with or without gates), which further give rise to conditions on which the performance of our method with gain gates can be guaranteed.

Original: A practical case is considered, where the assumption of no “false positive” is relaxed.
Revised: A practical case is considered, where the assumption of no “false positive” is relaxed.

Original: • Insightful expressions for the gates are presented.
Revised: • Insightful expressions for the gates are presented.

Original: In comparison with state-of-the-art sparse coding networks (not limited to previous extensions to LISTA), our method achieves superior performance.
Revised: In comparison with state-of-the-art sparse coding networks (not limited to previous extensions to LISTA), our method achieves superior performance.

Original: It also applies to variants of LISTA, e
Revised: It also applies to variants of LISTA, e

Original: g., LFSITA (Moreau & Bruna, 2017) and ALISTA (Liu et al., 2019).
Revised: g., LFSITA (Moreau & Bruna, 2017) and ALISTA (Liu et al., 2019).

Original: Notations : In this paper, unless otherwise clariﬁed, vectors and matrices are denoted by lowercase and uppercase characters, respectively.
Revised: Notations : In this paper, unless otherwise clariﬁed, vectors and matrices are denoted by lowercase and uppercase characters, respectively.

Original: For vectors/matrices originally introduced without any subscript, adding a subscript (e.g., i ) indicates its element/column at the corresponding position.
Revised: For vectors/matrices originally introduced without any subscript, adding a subscript (e.g., i ) indicates its element/column at the corresponding position.

Original: For instance, for x ∈ R n , x i represents the i -th element of the vector, and W : ,i and W i, : denote the i -th column and row of a matrix W respectively.
Revised: For instance, for x ∈ R n , x i represents the i -th element of the vector, and W : ,i and W i, : denote the i -th column and row of a matrix W respectively.

Original: While for vectors introduced with subscripts already, e.g., x s , we use ( x s ) i to denote its i -th element.
Revised: While for vectors introduced with subscripts already, e.g., x s , we use ( x s ) i to denote its i -th element.

Original: The operator (cid:12) is used to indicate element-wise multiplication of two vectors.
Revised: The operator (cid:12) is used to indicate element-wise multiplication of two vectors.

Original: The support of a vector is denoted as supp ( x ) :
Revised: The support of a vector is denoted as supp ( x ) :

Original: = { i | x i (cid:54) = 0 } .
Revised: = { i | x i (cid:54) = 0 } .

Original: We use sup x s as the simpliﬁed form of sup x s ∈X ( B,s, 0) , see Assumption 1 for the deﬁnition of X ( B, s, 0) .
Revised: We use sup x s as the simpliﬁed form of sup x s ∈X ( B,s, 0) , see Assumption 1 for the deﬁnition of X ( B, s, 0) .

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In general, sparse coding solves the problem that can be formulated as
Revised: In general, sparse coding solves the problem that can be formulated as

Original: [Equation]
Revised: [Equation]

Original: in which f ( x, y ) calculates the residual of approximating y using a linear combination of column-wise features in A .
Revised: in which f ( x, y ) calculates the residual of approximating y using a linear combination of column-wise features in A .

Original: The function f ( x, y ) is convex with respect to x in general.
Revised: The function f ( x, y ) is convex with respect to x in general.

Original: In particular, if ε is a
Revised: In particular, if ε is a

Original: Gaussian vector, then it should be f ( x, y )
Revised: Gaussian vector, then it should be f ( x, y )

Original: = (cid:107)
Revised: = (cid:107)

Original: Ax − y (cid:107) 22 .
Revised: Ax − y (cid:107) 22 .

Original: The term λr ( x ) serves as a regularizer for sparsity and we have r ( x ) = (cid:107) x (cid:107) 1 in Lasso.
Revised: The term λr ( x ) serves as a regularizer for sparsity and we have r ( x ) = (cid:107) x (cid:107) 1 in Lasso.

Original: As mentioned, a variety of algorithms can be applied to solve the problem and our focus in the paper is (L)ISTA.
Revised: As mentioned, a variety of algorithms can be applied to solve the problem and our focus in the paper is (L)ISTA.

Original: We ﬁrst revisit the optimization procedure of
Revised: We ﬁrst revisit the optimization procedure of

Original: ISTA, which is the foundation of LISTA as well.
Revised: ISTA, which is the foundation of LISTA as well.

Original: Given y , let us introduce a scalar γ > 0 that fulﬁlls γI − ∇ 2 x f ( x, y ) (cid:31) 0 , ∀ x , then it can be considered as optimizing an upper bound of the objective function obtained via Taylor expansion.
Revised: Given y , let us introduce a scalar γ > 0 that fulﬁlls γI − ∇ 2 x f ( x, y ) (cid:31) 0 , ∀ x , then it can be considered as optimizing an upper bound of the objective function obtained via Taylor expansion.

Original: To be more speciﬁc, for any presumed x ( t ) , we have
Revised: To be more speciﬁc, for any presumed x ( t ) , we have

Original: [Equation]
Revised: [Equation]

Original: By substituting r ( x ) with (cid:107) x (cid:107) 1 and optimizing the bound in an element-wise manner, we can easily get the one-step update rule that zeros the gradient based on x ( t ) .
Revised: By substituting r ( x ) with (cid:107) x (cid:107) 1 and optimizing the bound in an element-wise manner, we can easily get the one-step update rule that zeros the gradient based on x ( t ) .

Original: It is, x (0) = 0 and
Revised: It is, x (0) = 0 and

Original: [Equation]
Revised: [Equation]

Original: in which s b ( x ) :
Revised: in which s b ( x ) :

Original: = sign ( x )( | x | − b ) + is a shrinking function and ( · ) + is a rectiﬁed linear unit (ReLU) calculating max { 0 , ·} .
Revised: = sign ( x )( | x | − b ) + is a shrinking function and ( · ) + is a rectiﬁed linear unit (ReLU) calculating max { 0 , ·} .

Original: For Gaussian noises, the formulation reduces to
Revised: For Gaussian noises, the formulation reduces to

Original: [Equation]
Revised: [Equation]

Original: The update as shown in Eq.
Revised: The update as shown in Eq.

Original: (4) and (5) can be performed iteratively until convergence.
Revised: (4) and (5) can be performed iteratively until convergence.

Original: However, the convergence of ISTA (along with some other conventional solvers) is known to be slow, and it has been shown that DNNs can be utilized to accelerate the procedure.
Revised: However, the convergence of ISTA (along with some other conventional solvers) is known to be slow, and it has been shown that DNNs can be utilized to accelerate the procedure.

Original: Many researchers have explored the idea since the initial work of Gregor and LeCun’s (i.e., LISTA).
Revised: Many researchers have explored the idea since the initial work of Gregor and LeCun’s (i.e., LISTA).

Original: For LISTA, they design deep architectures following the main procedure of ISTA yet to learn parameters in an end-to-end manner from data (Gregor & LeCun, 2010; Hershey et al., 2014).
Revised: For LISTA, they design deep architectures following the main procedure of ISTA yet to learn parameters in an end-to-end manner from data (Gregor & LeCun, 2010; Hershey et al., 2014).

Original: The inference process of LISTA is similar to that of an RNN and can be formulated as x (0)
Revised: The inference process of LISTA is similar to that of an RNN and can be formulated as x (0)

Original: = 

Revised: =

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where Θ = { U ( t ) , W ( t ) , b ( t ) } t =0 , 1 ,...,d − 1 , is learnable parameters set.
Revised: where Θ = { U ( t ) , W ( t ) , b ( t ) } t =0 , 1 ,...,d − 1 , is learnable parameters set.

Original: Some works (Xin et al., 2016;
Revised: Some works (Xin et al., 2016;

Original: Chen et al., 2018) have proved that W ( t ) and U ( t ) should satisfy the constraint W ( t ) =
Revised: Chen et al., 2018) have proved that W ( t ) and U ( t ) should satisfy the constraint W ( t ) =

Original: I − U ( t ) A , such that
Revised: I − U ( t ) A , such that

Original: [Equation]
Revised: [Equation]

Original: The parameters in Θ are normally learned from a set of training samples by minimizing the difference between the ﬁnal code estimations and ground-truth.
Revised: The parameters in Θ are normally learned from a set of training samples by minimizing the difference between the ﬁnal code estimations and ground-truth.

Original: In this paper, our main assumption for theoretical analyses follows those of prior works (Chen et al., 2018; Liu et al., 2019) in a noiseless case, and noisy cases will be considered in the experiments.
Revised: In this paper, our main assumption for theoretical analyses follows those of prior works (Chen et al., 2018; Liu et al., 2019) in a noiseless case, and noisy cases will be considered in the experiments.

Original: Assumption 1.
Revised: Assumption 1.

Original: The sparse vector x s and noise vector ε are sampled from a set X ( B, s, 0) fulﬁlling:
Revised: The sparse vector x s and noise vector ε are sampled from a set X ( B, s, 0) fulﬁlling:

Original: [Equation]
Revised: [Equation]

Original:  S PARSE C ODING WITH G AIN G ATES AND O VERSHOOT G ATES
Revised:  S PARSE C ODING WITH G AIN G ATES AND O VERSHOOT G ATES

Original: In this section, we will introduce the advocated gain gates and overshoot gates.
Revised: In this section, we will introduce the advocated gain gates and overshoot gates.

Original: Along with thorough discussions for the motivations, their formulations are provided in Section 3.1 and 3.2, respectively.
Revised: Along with thorough discussions for the motivations, their formulations are provided in Section 3.1 and 3.2, respectively.

Original: Figure 1 summarizes the inference process of the standard LISTA and two evolved versions with our gates incorporated.
Revised: Figure 1 summarizes the inference process of the standard LISTA and two evolved versions with our gates incorporated.

Original: Proofs of all our theoretical results are deferred to the appendix.
Revised: Proofs of all our theoretical results are deferred to the appendix.

Original: 3. S PARSE C ODING WITH G AIN G ATES
Revised: 3. S PARSE C ODING WITH G AIN G ATES

Original: Recent works have shown linear convergence of LISTA (Chen et al., 2018; Liu et al., 2019).
Revised: Recent works have shown linear convergence of LISTA (Chen et al., 2018; Liu et al., 2019).

Original: In order to guarantee the convergence, it is also demonstrated that the value of bias terms should be large enough to eliminate all “false positive” in the support of the generated codes.
Revised: In order to guarantee the convergence, it is also demonstrated that the value of bias terms should be large enough to eliminate all “false positive” in the support of the generated codes.

Original: However, this may lead to an issue that the magnitude of the generated code components in LISTA must be smaller than or at most equal to those of the ground-truth.
Revised: However, this may lead to an issue that the magnitude of the generated code components in LISTA must be smaller than or at most equal to those of the ground-truth.

Original: Our result in Proposition 1 makes this formal.
Revised: Our result in Proposition 1 makes this formal.

Original: For clarity of the result, we would like to introduce the following deﬁnition ﬁrst.
Revised: For clarity of the result, we would like to introduce the following deﬁnition ﬁrst.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: (Liu et al., 2019)
Revised: (Liu et al., 2019)

Original: Given a matrix A ∈ R m × n , its generalized mutual coherence is:
Revised: Given a matrix A ∈ R m × n , its generalized mutual coherence is:

Original: [Equation]
Revised: [Equation]

Original: We let W ( A ) denote a set of all matrices that can achieve the generalized mutual coherence µ ( A ) , which means:
Revised: We let W ( A ) denote a set of all matrices that can achieve the generalized mutual coherence µ ( A ) , which means:

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: (Requirement of gains).
Revised: (Requirement of gains).

Original: With U ( t ) ∈ W ( A ) and W ( t ) =
Revised: With U ( t ) ∈ W ( A ) and W ( t ) =

Original: I − U ( t )
Revised: I − U ( t )

Original: A ,
Revised: A ,

Original: if b ( t ) =
Revised: if b ( t ) =

Original: µ ( A ) sup x s (cid:107) x
Revised: µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 is achieved in LISTA to guarantee no “false positive” (i.e., supp ( x ( t ) )
Revised: ( t ) − x s (cid:107) 1 is achieved in LISTA to guarantee no “false positive” (i.e., supp ( x ( t ) )

Original: ⊂ supp ( x s ) ) and further linear convergence (i.e., (cid:107) x
Revised: ⊂ supp ( x s ) ) and further linear convergence (i.e., (cid:107) x

Original: ( t ) − x s (cid:107) 2 ≤ sB exp( ct ) , in which c = log((2 s − 1) µ ( A )) ), then we have for the estimation | x ( t ) i | ≤ | ( x s )
Revised: ( t ) − x s (cid:107) 2 ≤ sB exp( ct ) , in which c = log((2 s − 1) µ ( A )) ), then we have for the estimation | x ( t ) i | ≤ | ( x s )

Original: i | and x ( t ) i
Revised: i | and x ( t ) i

Original: ( x s ) i ≥ 0 , ∀ i ∈ supp ( x s ) .
Revised: ( x s ) i ≥ 0 , ∀ i ∈ supp ( x s ) .

Original: Provided Proposition 1 as the evidence of a potential weakness of LISTA, we believe that if the code components can be enlarged appropriately, then the estimation at each step would be closer to x s , and the convergence of LISTA will be further improved, which inspires us to design a gate to enlarge the generated code components.
Revised: Provided Proposition 1 as the evidence of a potential weakness of LISTA, we believe that if the code components can be enlarged appropriately, then the estimation at each step would be closer to x s , and the convergence of LISTA will be further improved, which inspires us to design a gate to enlarge the generated code components.

Original: Such a gate is named as a gain gate and it acts on the input to the current estimation, akin to a reset gate in GRU (Cho et al., 2014), which is
Revised: Such a gate is named as a gain gate and it acts on the input to the current estimation, akin to a reset gate in GRU (Cho et al., 2014), which is

Original: [Equation]
Revised: [Equation]

Original: in which the gate function g t ( · , ·| Λ ( t ) g ) outputs an n -dimensions vector, and Λ ( t ) g is the set of its learnable parameters.
Revised: in which the gate function g t ( · , ·| Λ ( t ) g ) outputs an n -dimensions vector, and Λ ( t ) g is the set of its learnable parameters.

Original: In the original implementation of LISTA, the output of each layer is obtained by calculating Eq.
Revised: In the original implementation of LISTA, the output of each layer is obtained by calculating Eq.

Original: (4) iteratively.
Revised: (4) iteratively.

Original: It has been proven that the estimation x ( t ) ultimately converges to the ground-truth x s (as t → ∞ ), only if the condition of ( W ( t ) − ( I − U ( t ) A ))
Revised: It has been proven that the estimation x ( t ) ultimately converges to the ground-truth x s (as t → ∞ ), only if the condition of ( W ( t ) − ( I − U ( t ) A ))

Original: → 0 holds.
Revised: → 0 holds.

Original: That said, it is suggested that U ( t ) and W ( t ) are entangled to the end.
Revised: That said, it is suggested that U ( t ) and W ( t ) are entangled to the end.

Original: Yet, with our gated mechanism, the update rule in neural networks has been modiﬁed into Eq.
Revised: Yet, with our gated mechanism, the update rule in neural networks has been modiﬁed into Eq.

Original: (10), making it unclear whether the convergence is guaranteed similarly or not.
Revised: (10), making it unclear whether the convergence is guaranteed similarly or not.

Original: To ﬁgure it out, we perform theoretical analyses in depth, which will further provide guidance for the gate design.
Revised: To ﬁgure it out, we perform theoretical analyses in depth, which will further provide guidance for the gate design.

Original: We are going to explore: whether the learnable matrices are still entangled as in LISTA, and to encourage fast convergence, what properties should the gate function satisfy?
Revised: We are going to explore: whether the learnable matrices are still entangled as in LISTA, and to encourage fast convergence, what properties should the gate function satisfy?

Original: Theorem 1 and 2 give some answers to these questions and they are based on the same assumptions as for Proposition 1.
Revised: Theorem 1 and 2 give some answers to these questions and they are based on the same assumptions as for Proposition 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: If the s -th principal minor of W ( t ) have full rank, then for the gate function bounded from both above and below, we have x s as the ﬁxed point of Eq.
Revised: If the s -th principal minor of W ( t ) have full rank, then for the gate function bounded from both above and below, we have x s as the ﬁxed point of Eq.

Original: (10) only if diag( g t ( x ( t ) , y | Λ ( t ) g ))
Revised: (10) only if diag( g t ( x ( t ) , y | Λ ( t ) g ))

Original: → D and W ( t ) D − ( I − U ( t ) A ) → 0 , as t → ∞ , (11) in which D is an n × n constant diagonal matrix and the function diag( · ) creates a diagonal matrix with the elements of its input on the main diagonal.
Revised: → D and W ( t ) D − ( I − U ( t ) A ) → 0 , as t → ∞ , (11) in which D is an n × n constant diagonal matrix and the function diag( · ) creates a diagonal matrix with the elements of its input on the main diagonal.

Original: From Theorem 1 we can equivalently have ( ˜ W ( t ) − ( I − U ( t ) A ))
Revised: From Theorem 1 we can equivalently have ( ˜ W ( t ) − ( I − U ( t ) A ))

Original: → 0 by deﬁning ˜ W ( t ) :
Revised: → 0 by deﬁning ˜ W ( t ) :

Original: = W ( t ) D , which means the learnable matrices are similarly entangled as in the standard LISTA.
Revised: = W ( t ) D , which means the learnable matrices are similarly entangled as in the standard LISTA.

Original: Besides, we know that as the number of layers increases, each introduced gain gate should ultimately converge to a constant (diagonal) matrix D to guarantee performance.
Revised: Besides, we know that as the number of layers increases, each introduced gain gate should ultimately converge to a constant (diagonal) matrix D to guarantee performance.

Original: Then if W ( t ) → I − U ( t ) A , the gain gate function converges to an identical mapping as t → ∞ , and vice versa.
Revised: Then if W ( t ) → I − U ( t ) A , the gain gate function converges to an identical mapping as t → ∞ , and vice versa.

Original: This inspires us to “split” the gate function into an identical one and a residual one, and we thus advocate, for each index i of the vector, the i -th element of gain gate is
Revised: This inspires us to “split” the gate function into an identical one and a residual one, and we thus advocate, for each index i of the vector, the i -th element of gain gate is

Original: [Equation]
Revised: [Equation]

Original: in which κ t ( x ( t ) , y | Λ ( t ) g ) i is the i -th element of κ t ( x ( t ) , y | Λ ( t ) g ) , and it should decrease as t increases, in order to guarantee convergence in Eq.
Revised: in which κ t ( x ( t ) , y | Λ ( t ) g ) i is the i -th element of κ t ( x ( t ) , y | Λ ( t ) g ) , and it should decrease as t increases, in order to guarantee convergence in Eq.

Original: Let us further study the convergence rate of “LISTA” equipped with such gain gates.
Revised: Let us further study the convergence rate of “LISTA” equipped with such gain gates.

Original: For clarity, we introduce another condition for the function before moving to more details:
Revised: For clarity, we introduce another condition for the function before moving to more details:

Original: [Equation]
Revised: [Equation]

Original: We present theoretical results as follows on the basis of Proposition 1, i.e., we still have U ( t ) ∈
Revised: We present theoretical results as follows on the basis of Proposition 1, i.e., we still have U ( t ) ∈

Original: W
Revised: W

Original: ( A ) ,
Revised: ( A ) ,

Original: W ( t )
Revised: W ( t )

Original: I − t
Revised: I − t

Original: Theorem 2.
Revised: Theorem 2.

Original: If b ( t ) = µ ( A ) sup x s (cid:107) x s − x ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) (cid:107) 1 is achieved, following the update rule in Eq.
Revised: If b ( t ) = µ ( A ) sup x s (cid:107) x s − x ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) (cid:107) 1 is achieved, following the update rule in Eq.

Original: (10), if the conditions in Eq.
Revised: (10), if the conditions in Eq.

Original: (12) and (13) hold for the gate function, there will be (cid:107) x
Revised: (12) and (13) hold for the gate function, there will be (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: (14) in which c = log((
Revised: (14) in which c = log((

Original: [Equation]
Revised: [Equation]

Original: s −
Revised: s −

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: if i
Revised: if i

Original: [Equation]
Revised: [Equation]

Original: otherwise.
Revised: otherwise.

Original: Theorem 2 presents an upper bound of (cid:107) x ( t ) − x s (cid:107) 2 for LISTA with gain gates, and it shows that so long as the gates satisfying conditions in Eq.
Revised: Theorem 2 presents an upper bound of (cid:107) x ( t ) − x s (cid:107) 2 for LISTA with gain gates, and it shows that so long as the gates satisfying conditions in Eq.

Original: (12) and (13) are introduced, the “convergence factor” c + (cid:80) c i of our gated LISTA would be smaller in comparison with that of the standard LISTA (which is ct , see Proposition 1 and Chen et al.
Revised: (12) and (13) are introduced, the “convergence factor” c + (cid:80) c i of our gated LISTA would be smaller in comparison with that of the standard LISTA (which is ct , see Proposition 1 and Chen et al.

Original: ’s work 2018).
Revised: ’s work 2018).

Original: By consolidating all these theoretical cues, we further give principled expressions for the gate function.
Revised: By consolidating all these theoretical cues, we further give principled expressions for the gate function.

Original: One may expect to endow the gates some learning capacities, thus we let
Revised: One may expect to endow the gates some learning capacities, thus we let

Original: [Equation]
Revised: [Equation]

Original: in which µ t ∈ R is a parameter to be learned, b ( t − 1) is the threshold parameter of the ( t − 1) -th layer, and f t ( x ( t )
Revised: in which µ t ∈ R is a parameter to be learned, b ( t − 1) is the threshold parameter of the ( t − 1) -th layer, and f t ( x ( t )

Original: | ν t ) is a newly introduced function constrained not to be greater than 2 / | x ( t ) | .
Revised: | ν t ) is a newly introduced function constrained not to be greater than 2 / | x ( t ) | .

Original: We are going to evaluate different choices for the function f t ( x ( t ) ) in experiments, e.g.,
Revised: We are going to evaluate different choices for the function f t ( x ( t ) ) in experiments, e.g.,

Original: [Equation]
Revised: [Equation]

Original: piece
Revised: piece

Original: [Equation]
Revised: [Equation]

Original: in which ν t ∈ R is a parameter to be learned, and (cid:15) is a tiny positive scalar introduced to avoid zero being divided.
Revised: in which ν t ∈ R is a parameter to be learned, and (cid:15) is a tiny positive scalar introduced to avoid zero being divided.

Original: All the learnable parameters in a gain gate are thus collected as Λ ( t ) g
Revised: All the learnable parameters in a gain gate are thus collected as Λ ( t ) g

Original: = { µ t , ν t } .
Revised: = { µ t , ν t } .

Original: 3.1. N O F ALSE P OSITIVE ?
Revised: 3.1. N O F ALSE P OSITIVE ?

Original: Our previous theoretical results show that the performance of LISTA can be improved by using a gain gate, as long as the gate function satisﬁes conditions in Eq.
Revised: Our previous theoretical results show that the performance of LISTA can be improved by using a gain gate, as long as the gate function satisﬁes conditions in Eq.

Original: (12) and (13), and no “false positive” is encountered.
Revised: (12) and (13), and no “false positive” is encountered.

Original: However, it is not always true in practice.
Revised: However, it is not always true in practice.

Original: Our experimental results also show that when the inverse proportional function is adopted as gain gates in lower layer for LISTA, the performance of our gated LISTA may even degrade.
Revised: Our experimental results also show that when the inverse proportional function is adopted as gain gates in lower layer for LISTA, the performance of our gated LISTA may even degrade.

Original: We conjecture that such contradiction to the theoretical results may be owing to impractical assumptions.
Revised: We conjecture that such contradiction to the theoretical results may be owing to impractical assumptions.

Original: In this subsection, we try to relax the assumption about no “false positive”, and we further found that a tighter bound can be achieved with a more reasonable assumption instead.
Revised: In this subsection, we try to relax the assumption about no “false positive”, and we further found that a tighter bound can be achieved with a more reasonable assumption instead.

Original: Through theoretical analyses as follows, we also demonstrate that the inverse proportional gain function should better be only adopted in higher layers.
Revised: Through theoretical analyses as follows, we also demonstrate that the inverse proportional gain function should better be only adopted in higher layers.

Original: For clarity of the results, we would like to introduce the following deﬁnition ﬁrst.
Revised: For clarity of the results, we would like to introduce the following deﬁnition ﬁrst.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Given a model with Θ , in which b ( t ) =
Revised: Given a model with Θ , in which b ( t ) =

Original: Γ µ ( A ) sup x s (cid:107) x
Revised: Γ µ ( A ) sup x s (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: , y
Revised: , y

Original: [Equation]
Revised: [Equation]

Original: , we introduce ω t +1 ( k | Θ) to characterize its relationship with the false positive rate, which is ω t +1 ( k t +1 | Θ
Revised: , we introduce ω t +1 ( k | Θ) to characterize its relationship with the false positive rate, which is ω t +1 ( k t +1 | Θ

Original: [Equation]
Revised: [Equation]

Original: = sup
Revised: = sup

Original: [Equation]
Revised: [Equation]

Original: |≤| supp
Revised: |≤| supp

Original: [Equation]
Revised: [Equation]

Original: x s
Revised: x s

Original: [Equation]
Revised: [Equation]

Original: | + k
Revised: | + k

Original: [Equation]
Revised: [Equation]

Original: + Γ , in which ˇ x ( t +1) :
Revised: + Γ , in which ˇ x ( t +1) :

Original: = s b ( t ) ( W ( t ) ( x ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) − x s )) , and k t +1 ≥ 0 is the desired maximal number of “false positive” of x ( t +1) .
Revised: = s b ( t ) ( W ( t ) ( x ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) − x s )) , and k t +1 ≥ 0 is the desired maximal number of “false positive” of x ( t +1) .

Original: The above deﬁnition applies to both the standard LISTA and LISTA with gain gates (we can let the gate function be an identity function to achieve a standard LISTA).
Revised: The above deﬁnition applies to both the standard LISTA and LISTA with gain gates (we can let the gate function be an identity function to achieve a standard LISTA).

Original: We ﬁrst analyze the convergence of LISTA without gates.
Revised: We ﬁrst analyze the convergence of LISTA without gates.

Original: We present theoretical results as follows on the basis of similar assumptions (including Assumption 1, U ( t )
Revised: We present theoretical results as follows on the basis of similar assumptions (including Assumption 1, U ( t )

Original: ∈ W ( A ) , and W ( t ) =
Revised: ∈ W ( A ) , and W ( t ) =

Original: I − U ( t ) A ), but with a different requirement for b ( t ) from Proposition 1.
Revised: I − U ( t ) A ), but with a different requirement for b ( t ) from Proposition 1.

Original: Theorem 3.
Revised: Theorem 3.

Original: If b ( t ) =
Revised: If b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 is achieved, and ∃ 0 < k ( t ) 

Revised: ( t ) − x s (cid:107) 1 is achieved, and ∃ 0 < k ( t )

Original: < s such that ω t ( k ( t ) 0 | Θ) < 1 − 1 / ( s − k ( t ) 0 ) , then there exists “false positive” with 0 < k t <
Revised: < s such that ω t ( k ( t ) 0 | Θ) < 1 − 1 / ( s − k ( t ) 0 ) , then there exists “false positive” with 0 < k t <

Original: s and (cid:107) x
Revised: s and (cid:107) x

Original: ( t ) − x s (cid:107) 2 ≤ sB exp( t (cid:88)
Revised: ( t ) − x s (cid:107) 2 ≤ sB exp( t (cid:88)

Original: i =
Revised: i =

Original: c ∗ i ) , in which c ∗ i < log((2 s − 1) µ ( A )) .
Revised: c ∗ i ) , in which c ∗ i < log((2 s − 1) µ ( A )) .

Original: It can be seen that when we relax the assumption about no “false positive” and further reduce the value of the threshold b ( t ) , the error bound of LISTA becomes even lower.
Revised: It can be seen that when we relax the assumption about no “false positive” and further reduce the value of the threshold b ( t ) , the error bound of LISTA becomes even lower.

Original: Obviously, the previous bound of LISTA with gain gates in Theorem 2 is not necessarily lower than the tighter bound of a standard
Revised: Obviously, the previous bound of LISTA with gain gates in Theorem 2 is not necessarily lower than the tighter bound of a standard

Original: LISTA in Theorem 3, which well explains the contradiction of theoretical and empirical results.
Revised: LISTA in Theorem 3, which well explains the contradiction of theoretical and empirical results.

Original: Here we re-deduce the error bound of our gated LISTA with the inverse proportional function in the following theorem.
Revised: Here we re-deduce the error bound of our gated LISTA with the inverse proportional function in the following theorem.

Original: Note that we still have U ( t ) ∈
Revised: Note that we still have U ( t ) ∈

Original: W
Revised: W

Original: ( A ) , W ( t ) =
Revised: ( A ) , W ( t ) =

Original: I − U ( t ) A and Assumption 1.
Revised: I − U ( t ) A and Assumption 1.

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose that min i ∈ supp ( x s ) | ( x s ) i | ≥ σ > 0 , if b ( t ) =
Revised: Suppose that min i ∈ supp ( x s ) | ( x s ) i | ≥ σ > 0 , if b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x

Original: ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) − x s (cid:107) 1 is achieved and ∃ 0 < k ( t ) 

Revised: ( t ) (cid:12) g t ( x ( t ) , y | Λ ( t ) g ) − x s (cid:107) 1 is achieved and ∃ 0 < k ( t )

Original: < s such that ω t ( k ( t ) 0 | Θ) < 1 − 1 / ( s − k ( t ) 0 ) , then (cid:107) x
Revised: < s such that ω t ( k ( t ) 0 | Θ) < 1 − 1 / ( s − k ( t ) 0 ) , then (cid:107) x

Original: ( t ) − x s (cid:107) 2 ≤ sB exp( t − 1 (cid:88) i
Revised: ( t ) − x s (cid:107) 2 ≤ sB exp( t − 1 (cid:88) i

Original: = c (cid:48)
Revised: = c (cid:48)

Original: i + c (cid:48)∗ t ) ,
Revised: i + c (cid:48)∗ t ) ,

Original: in which c (cid:48)∗ t < log((2 s − 1) µ ( A )) .
Revised: in which c (cid:48)∗ t < log((2 s − 1) µ ( A )) .

Original: ∃ t 0 = (cid:100) log( sBσ ) / log( 1 (2 s − 1) µ ( A ) ) (cid:101) if the scaling factor µ i of the gate has µ i = 0 for i ≤ t 0 , 0 < k
Revised: ∃ t 0 = (cid:100) log( sBσ ) / log( 1 (2 s − 1) µ ( A ) ) (cid:101) if the scaling factor µ i of the gate has µ i = 0 for i ≤ t 0 , 0 < k

Original: i < s , then c (cid:48)
Revised: i < s , then c (cid:48)

Original: i =
Revised: i =

Original: c ∗ i , and if 1 − ω
Revised: c ∗ i , and if 1 − ω

Original: i ( s | Θ) < µ i ≤ 1 for i > t 0 , k i = 0 , then c (cid:48)
Revised: i ( s | Θ) < µ i ≤ 1 for i > t 0 , k i = 0 , then c (cid:48)

Original: i < c ∗ i .
Revised: i < c ∗ i .

Original: We can conclude from Theorem 4 that, a) a gain gate expressed by the inverse proportional function should be applied to deeper layers in LISTA, rather than lower layers, b) when using the function, there indeed exists no “false positive” (i.e., k i = 0 ) in deeper layer.
Revised: We can conclude from Theorem 4 that, a) a gain gate expressed by the inverse proportional function should be applied to deeper layers in LISTA, rather than lower layers, b) when using the function, there indeed exists no “false positive” (i.e., k i = 0 ) in deeper layer.

Original: We follow such guidelines in the implementation of our gated LISTA.
Revised: We follow such guidelines in the implementation of our gated LISTA.

Original: In addition, we observe that unlike the inverse proportional function, other considered functions show consistent performance gains on both lower and higher layers, hence we attempt to utilize them on lower layers in alliance with the inverse proportional function powered gain gates on the other layers.
Revised: In addition, we observe that unlike the inverse proportional function, other considered functions show consistent performance gains on both lower and higher layers, hence we attempt to utilize them on lower layers in alliance with the inverse proportional function powered gain gates on the other layers.

Original: In practice, we choose the ReLU-based piece-wise linear function, and it is uniformly applied to the ﬁrst 10 layers.
Revised: In practice, we choose the ReLU-based piece-wise linear function, and it is uniformly applied to the ﬁrst 10 layers.

Original: We will empirically compare different choices between the gain gate functions in Section 4.1.
Revised: We will empirically compare different choices between the gain gate functions in Section 4.1.

Original: 3. S PARSE C ODING WITH O VERSHOOT G ATES
Revised: 3. S PARSE C ODING WITH O VERSHOOT G ATES

Original: Unlike the gain gates that are incorporated before performing estimation at each step, the overshoot gates act more like adjustments to the outputs, which can be viewed as learnable boosts:
Revised: Unlike the gain gates that are incorporated before performing estimation at each step, the overshoot gates act more like adjustments to the outputs, which can be viewed as learnable boosts:

Original: [Equation]
Revised: [Equation]

Original: The gate function o t ( · , ·| Λ ( t ) o ) : { R n , R m } → R n outputs an n -dimensional vector and Λ ( t ) o collects all the trainable parameters in the function, akin to a dedicated update GRU gate (Cho et al., 2014).
Revised: The gate function o t ( · , ·| Λ ( t ) o ) : { R n , R m } → R n outputs an n -dimensional vector and Λ ( t ) o collects all the trainable parameters in the function, akin to a dedicated update GRU gate (Cho et al., 2014).

Original: Our motivation comes from analyses of ISTA, whose update can be viewed as x ( t ) + η ( x ( t +1) − x ( t ) ) , in which η = 1 is a constant step size.
Revised: Our motivation comes from analyses of ISTA, whose update can be viewed as x ( t ) + η ( x ( t +1) − x ( t ) ) , in which η = 1 is a constant step size.

Original: We argue that η = 1 may not be the most suitable choice and the following proposition makes this formal.
Revised: We argue that η = 1 may not be the most suitable choice and the following proposition makes this formal.

Original: We have it to theoretically analyze the update rule of
Revised: We have it to theoretically analyze the update rule of

Original: ISTA and η ∗ := arg
Revised: ISTA and η ∗ := arg

Original: [Equation]
Revised: [Equation]

Original: η ( x ( t +1) − x ( t ) )
Revised: η ( x ( t +1) − x ( t ) )

Original: + x ( t ) (cid:107) 1 .
Revised: + x ( t ) (cid:107) 1 .

Original: Proposition 2.
Revised: Proposition 2.

Original: (Requirement of overshoots)
Revised: (Requirement of overshoots)

Original: For min
Revised: For min

Original: x f ( x, y ) + λ (cid:107) x (cid:107) 1 , in which f ( x, y ) is convex with respect to x and γI − ∇ 2 x f ( x ) (cid:31) 0 holds for all x , if the update rule in Eq.
Revised: x f ( x, y ) + λ (cid:107) x (cid:107) 1 , in which f ( x, y ) is convex with respect to x and γI − ∇ 2 x f ( x ) (cid:31) 0 holds for all x , if the update rule in Eq.

Original: (4) is adopted, then we have η ∗ ≥ 1 .
Revised: (4) is adopted, then we have η ∗ ≥ 1 .

Original: In addition, if supp ( x ( t ) ) ⊂ supp ( x ( t +1) ) , then we further have η ∗ > 1 .
Revised: In addition, if supp ( x ( t ) ) ⊂ supp ( x ( t +1) ) , then we further have η ∗ > 1 .

Original: See also Figure 3 for an illustration of the issue with η = 1 as concerned.
Revised: See also Figure 3 for an illustration of the issue with η = 1 as concerned.

Original: Since the optimization procedure of ISTA inspires the network architecture in LISTA, the theoretical result in Propositionthat requires a boost in η for superior performance also inspires us to design speciﬁc overshoot gates for LISTA.
Revised: Since the optimization procedure of ISTA inspires the network architecture in LISTA, the theoretical result in Propositionthat requires a boost in η for superior performance also inspires us to design speciﬁc overshoot gates for LISTA.

Original: Having noticed that an essential principle we have obtained is to let η ≥ 1 (or η > 1 ), we may expect the output of the gate function to be greater than or at least equal to 1 .
Revised: Having noticed that an essential principle we have obtained is to let η ≥ 1 (or η > 1 ), we may expect the output of the gate function to be greater than or at least equal to 1 .

Original: To achieve the goal, we can try different expressions for it, e.g., the sigmoid-based function: o t (
Revised: To achieve the goal, we can try different expressions for it, e.g., the sigmoid-based function: o t (

Original: [Equation]
Revised: [Equation]

Original: ( t ) ,
Revised: ( t ) ,

Original: [Equation]
Revised: [Equation]

Original: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:88)
Revised: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) , the inverse-proportional-based function: o t ( x ( t ) , y | Λ ( t ) o )
Revised: (cid:12)(cid:12)(cid:12)(cid:12) , the inverse-proportional-based function: o t ( x ( t ) , y | Λ ( t ) o )

Original: = 1 +
Revised: = 1 +

Original: [Equation]
Revised: [Equation]

Original: with σ ( · ) being the sigmoid function, Λ ( t )
Revised: with σ ( · ) being the sigmoid function, Λ ( t )

Original: o
Revised: o

Original: = { a o , W o , U o } and Λ ( t ) o
Revised: = { a o , W o , U o } and Λ ( t ) o

Original: = { a o } for the two types of functions respectively, and (cid:15) being a tiny positive constant introduced to avoid zero being divided.
Revised: = { a o } for the two types of functions respectively, and (cid:15) being a tiny positive constant introduced to avoid zero being divided.

Original: The principle of our overshoot gate is similar to that of some momentum-based methods, e.g.,
Revised: The principle of our overshoot gate is similar to that of some momentum-based methods, e.g.,

Original: FISTA (Beck & Teboulle, 2009) and LFISTA (Moreau & Bruna, 2017).
Revised: FISTA (Beck & Teboulle, 2009) and LFISTA (Moreau & Bruna, 2017).

Original: The fundamental difference between these methods and ours is that, (L)FISTA considers that the scaling factor in a momentum term should be independent of the current inputs (including the previous estimation and y ), i.e., being time or at least input invariant, while the output of the overshoot gate is a function of both the previous estimation and y , hence being time-and-input-varying.
Revised: The fundamental difference between these methods and ours is that, (L)FISTA considers that the scaling factor in a momentum term should be independent of the current inputs (including the previous estimation and y ), i.e., being time or at least input invariant, while the output of the overshoot gate is a function of both the previous estimation and y , hence being time-and-input-varying.

Original: The design of our overshoot gate may endow the sparse coding network higher capability to learn from its inputs.
Revised: The design of our overshoot gate may endow the sparse coding network higher capability to learn from its inputs.

Original: Experimental comparisons in
Revised: Experimental comparisons in

Original: Section 7 in the Appendix conﬁrm the superiority of our method.
Revised: Section 7 in the Appendix conﬁrm the superiority of our method.

Original: We also note that our convergence analyses in Section 3.1 generalize to η > 1 cases with a constant η , i.e., linear convergence can still be guaranteed, but the asymptotic behavior with learnable and adaptive overshoots should be further explored in future studies.
Revised: We also note that our convergence analyses in Section 3.1 generalize to η > 1 cases with a constant η , i.e., linear convergence can still be guaranteed, but the asymptotic behavior with learnable and adaptive overshoots should be further explored in future studies.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we perform experiments to conﬁrm our theoretical results and evaluate the performance of our gated sparse coding networks.
Revised: In this section, we perform experiments to conﬁrm our theoretical results and evaluate the performance of our gated sparse coding networks.

Original: Validations of our theoretical results are performed on synthetic data, and the performance of our method in sparse coding is tested on both synthetic and real data.
Revised: Validations of our theoretical results are performed on synthetic data, and the performance of our method in sparse coding is tested on both synthetic and real data.

Original: We set m = 250 , n = 500 , and sample the elements of the dictionary matrix
Revised: We set m = 250 , n = 500 , and sample the elements of the dictionary matrix

Original: A randomly from a standard Gaussian distribution in simulations.
Revised: A randomly from a standard Gaussian distribution in simulations.

Original: The position of non-zero elements of the sparse vector x s is determined by a Bernoulli sampling with a probability of 0.1 (which means approximately 90% of the elements are set to be zero).
Revised: The position of non-zero elements of the sparse vector x s is determined by a Bernoulli sampling with a probability of 0.1 (which means approximately 90% of the elements are set to be zero).

Original: Different noise levels and condition numbers are considered in the sparse coding simulations.
Revised: Different noise levels and condition numbers are considered in the sparse coding simulations.

Original: We randomly synthesize in-stream x s and ε to obtain y for training, and we let two extra sets consisting of 1000 samples each as the validation and test sets, just like in prior works.
Revised: We randomly synthesize in-stream x s and ε to obtain y for training, and we let two extra sets consisting of 1000 samples each as the validation and test sets, just like in prior works.

Original: (Chen et al., 2018; Liu et al., 2019; Borgerding et al., 2017).
Revised: (Chen et al., 2018; Liu et al., 2019; Borgerding et al., 2017).

Original: For the proposed gated LISTA and other deep learning-based methods, we set d = 16 and let { b ( t ) } not be shared between different layers under all circumstances.
Revised: For the proposed gated LISTA and other deep learning-based methods, we set d = 16 and let { b ( t ) } not be shared between different layers under all circumstances.

Original: The weight matrices { W ( t ) , U ( t ) } are not shared either in our method and the coupled constraints W ( t ) =
Revised: The weight matrices { W ( t ) , U ( t ) } are not shared either in our method and the coupled constraints W ( t ) =

Original: I − U ( t ) A, ∀ t , are imposed.
Revised: I − U ( t ) A, ∀ t , are imposed.

Original: For all gates, ν t is initialized as 1 . 0 , and then we let the initial value of µ t in the inverse proportional function powered gain gate be 1 . 0 too, since Eq.
Revised: For all gates, ν t is initialized as 1 . 0 , and then we let the initial value of µ t in the inverse proportional function powered gain gate be 1 . 0 too, since Eq.

Original: (12) and (13) indicate 0 ≤ µ t ≤ 2 .
Revised: (12) and (13) indicate 0 ≤ µ t ≤ 2 .

Original: Other learnable parameters in our gates are uniformly initialized as 5 . 

Revised: Other learnable parameters in our gates are uniformly initialized as 5 .

Original: according to their suggested range of the gates.
Revised: according to their suggested range of the gates.

Original: The training batch size is 64 .
Revised: The training batch size is 64 .

Original: We use Adam (Cho et al., 2014) and let β 1 = 0 .
Revised: We use Adam (Cho et al., 2014) and let β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: The hyper-parameters are tuned on the validation set and ﬁxed for all our experiments in the sequel.
Revised: The hyper-parameters are tuned on the validation set and ﬁxed for all our experiments in the sequel.

Original: Our training follows it of Chen et al.’s
Revised: Our training follows it of Chen et al.’s

Original: That said, the sparse coding network is trained progressively to update more layers, and we cut the learning rate for currently optimized layers when no decrease in the validation loss can be observed for 4000 iterations, with a base learning rate of 0 . 0005 .
Revised: That said, the sparse coding network is trained progressively to update more layers, and we cut the learning rate for currently optimized layers when no decrease in the validation loss can be observed for 4000 iterations, with a base learning rate of 0 . 0005 .

Original: Training on current layers stops when the validation loss does not decrease any more with the learning rate being cut to 0 .
Revised: Training on current layers stops when the validation loss does not decrease any more with the learning rate being cut to 0 .

Original: More details are explained in Section 8 in the appendix.
Revised: More details are explained in Section 8 in the appendix.

Original: Our training objective for a network with d intermediate update steps is
Revised: Our training objective for a network with d intermediate update steps is

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in which Θ = { W ( t ) , U ( t ) , b ( t ) } t =0 ,...,d − 1 ∪ Λ (0) ∪ . . .
Revised: in which Θ = { W ( t ) , U ( t ) , b ( t ) } t =0 ,...,d − 1 ∪ Λ (0) ∪ . . .

Original: Λ ( d − 1) is the set of all learnable parameters in the sparse coding network that generates x ( d ) given y .
Revised: Λ ( d − 1) is the set of all learnable parameters in the sparse coding network that generates x ( d ) given y .

Original: Note that in comparison with the parameter set in a standard LISTA, it also contains the parameters in gate functions.
Revised: Note that in comparison with the parameter set in a standard LISTA, it also contains the parameters in gate functions.

Original: In practice, we are given a set of training samples and opt to minimize an empirical loss instead of the one in Eq.
Revised: In practice, we are given a set of training samples and opt to minimize an empirical loss instead of the one in Eq.

Original: Our evaluation metric for sparse coding is the normalized MSE (NMSE) (Chen et al., 2018):
Revised: Our evaluation metric for sparse coding is the normalized MSE (NMSE) (Chen et al., 2018):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( (cid:107)
Revised: ( (cid:107)

Original: [Equation]
Revised: [Equation]

Original: s (cid:107) 22 / (cid:107)
Revised: s (cid:107) 22 / (cid:107)

Original: [Equation]
Revised: [Equation]

Original: s (cid:107) 22 ) .
Revised: s (cid:107) 22 ) .

Original: 4. S IMULATION E XPERIMENTS 4.1. V ALIDATION OF T HEORETICAL R ESULTS
Revised: 4. S IMULATION E XPERIMENTS 4.1. V ALIDATION OF T HEORETICAL R ESULTS

Original: Validation of Proposition 1: We ﬁrst conﬁrm Proposition 1 .
Revised: Validation of Proposition 1: We ﬁrst conﬁrm Proposition 1 .

Original: In order to ensure that LISTA fulﬁlls the assumption about no “false positive”, we introduce an auxiliary loss into the learning object as:
Revised: In order to ensure that LISTA fulﬁlls the assumption about no “false positive”, we introduce an auxiliary loss into the learning object as:

Original: [Equation]
Revised: [Equation]

Original: We formally introduce the false positive rate (FPR) as FPR = | supp ( x
Revised: We formally introduce the false positive rate (FPR) as FPR = | supp ( x

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: ∪ supp ( x s ) |−| supp ( x s ) | | supp ( x ( t ) ) | and try to approach no “false positive” (i.e., LISTA-nfp) by setting λ = 5 . 0 in the experiment.
Revised: ∪ supp ( x s ) |−| supp ( x s ) | | supp ( x ( t ) ) | and try to approach no “false positive” (i.e., LISTA-nfp) by setting λ = 5 . 0 in the experiment.

Original: 1 Check Figurefor an illustrative comparison between different models, we see LISTA-nfp achieves almost no “false positive” in practice in Figure 4(a), but its convergence is slower as demonstrated in Figure 4(c), which is consistent with our result in Theorem 3 .
Revised: 1 Check Figurefor an illustrative comparison between different models, we see LISTA-nfp achieves almost no “false positive” in practice in Figure 4(a), but its convergence is slower as demonstrated in Figure 4(c), which is consistent with our result in Theorem 3 .

Original: In addition, we also see in Figure 4(b) that without “false positive”, the code components in LISTA estimations are almost always less than those of the ground-truth, which conﬁrms our Proposition 1.
Revised: In addition, we also see in Figure 4(b) that without “false positive”, the code components in LISTA estimations are almost always less than those of the ground-truth, which conﬁrms our Proposition 1.

Original: Validation of Theorem 1 : We aim to calculate (cid:107) W ( t ) D − ( I − U ( t ) A ) (cid:107) 2 using a gated LISTA with the introduced ReLU-based piece-wise linear gain gate function 2 .
Revised: Validation of Theorem 1 : We aim to calculate (cid:107) W ( t ) D − ( I − U ( t ) A ) (cid:107) 2 using a gated LISTA with the introduced ReLU-based piece-wise linear gain gate function 2 .

Original: To accomplish this task, we need to ﬁrst evaluate the output of our gate function, which is expect to converge to 1 as shown in the theorem.
Revised: To accomplish this task, we need to ﬁrst evaluate the output of our gate function, which is expect to converge to 1 as shown in the theorem.

Original: We show such a trend indeed exists in Figure 5(a).
Revised: We show such a trend indeed exists in Figure 5(a).

Original: Consequently, the matrix D is supposed
Revised: Consequently, the matrix D is supposed

Original: [Figure]
Revised: [Figure]

Original: to be an identity matrix in the end and we can calculate (cid:107) W ( t ) − ( I − U ( t ) A ) (cid:107) 2 as a surrogate.
Revised: to be an identity matrix in the end and we can calculate (cid:107) W ( t ) − ( I − U ( t ) A ) (cid:107) 2 as a surrogate.

Original: In
Revised: In

Original: Figure 5(b), it converges to zero in the end and the results conﬁrm the theorem.
Revised: Figure 5(b), it converges to zero in the end and the results conﬁrm the theorem.

Original: Validation of Theorem 2 : We apply three kinds of gated LISTA with an alliance of gain gate functions (i.e., what has been introduced in Section 3.1.1), the exponential function, and the inverse proportional function respectively to verify our theoretical results.
Revised: Validation of Theorem 2 : We apply three kinds of gated LISTA with an alliance of gain gate functions (i.e., what has been introduced in Section 3.1.1), the exponential function, and the inverse proportional function respectively to verify our theoretical results.

Original: They were named as GLISTA (which is the abbreviation of gated LISTA), GLISTA-exp, GLISTA-inv, respectively.
Revised: They were named as GLISTA (which is the abbreviation of gated LISTA), GLISTA-exp, GLISTA-inv, respectively.

Original: From Figure 4(c), we see that when the models with such gain gates has no “false positive”, all of them are superior to the standard LISTA without “false positive” as well, which is consistent with the conclusion of
Revised: From Figure 4(c), we see that when the models with such gain gates has no “false positive”, all of them are superior to the standard LISTA without “false positive” as well, which is consistent with the conclusion of

Original: Theorem 2.
Revised: Theorem 2.

Original: In addition, from Figure 4(a), we can also see that there actually exist “false positives” in lower layers of GLISTA, but even without the auxiliary loss term, the evaluated FPR of our GLISTA and it variants approach zero in higher layers, which is in good agreement with Theorem 4 .
Revised: In addition, from Figure 4(a), we can also see that there actually exist “false positives” in lower layers of GLISTA, but even without the auxiliary loss term, the evaluated FPR of our GLISTA and it variants approach zero in higher layers, which is in good agreement with Theorem 4 .

Original: 4.1. C OMPARISON WITH C OMPETITORS
Revised: 4.1. C OMPARISON WITH C OMPETITORS

Original: Empirical analyses for the gate functions: It should be interesting to compare the performance of our gates with different expressions.
Revised: Empirical analyses for the gate functions: It should be interesting to compare the performance of our gates with different expressions.

Original: We test LISTA with different overshoot gate functions introduced in Section 3.2 in Figure 6(a).
Revised: We test LISTA with different overshoot gate functions introduced in Section 3.2 in Figure 6(a).

Original: Both of them are incorporated with their learnable parameters being shared among layers.
Revised: Both of them are incorporated with their learnable parameters being shared among layers.

Original: It can be seen from Figure 6(a) that the accelerations in convergence and gain in ﬁnal performance are obvious, just as expected.
Revised: It can be seen from Figure 6(a) that the accelerations in convergence and gain in ﬁnal performance are obvious, just as expected.

Original: For LISTA with gain gates, one can check
Revised: For LISTA with gain gates, one can check

Original: Figure 6(b).
Revised: Figure 6(b).

Original: It can be seen that the performance degrades a lot if either the bias term or the µ t term is removed.
Revised: It can be seen that the performance degrades a lot if either the bias term or the µ t term is removed.

Original: We also try different f t ( · ) functions, including the ReLU-based piece-wise linear one and some possibly more nonlinear ones as mentioned in Section 3.1.
Revised: We also try different f t ( · ) functions, including the ReLU-based piece-wise linear one and some possibly more nonlinear ones as mentioned in Section 3.1.

Original: We conﬁrm that gate functions whose outputs are relatively closer to the boundary condition may perform better.
Revised: We conﬁrm that gate functions whose outputs are relatively closer to the boundary condition may perform better.

Original: Yet, it is worth noting that when the outputs of inverse proportional function reach that boundary condition and being applied uniformly to all layers, the performance degrades (see LISTA-inv- (cid:15) in Figure 6(b)).
Revised: Yet, it is worth noting that when the outputs of inverse proportional function reach that boundary condition and being applied uniformly to all layers, the performance degrades (see LISTA-inv- (cid:15) in Figure 6(b)).

Original: These results suggest an alliance of gain gate functions in practice.
Revised: These results suggest an alliance of gain gate functions in practice.

Original: We further test a combination of gain gates and overshoot gates, despite the mechanism with solely gain gates is already good enough.
Revised: We further test a combination of gain gates and overshoot gates, despite the mechanism with solely gain gates is already good enough.

Original: See
Revised: See

Original: Figure 6(c), when overshoots are further incorporated, the convergence on lower layers becomes faster while the overall convergence is not affected much, leading to similar ﬁnal performance when the model is very deep and superior performance when the model is relatively shallow.
Revised: Figure 6(c), when overshoots are further incorporated, the convergence on lower layers becomes faster while the overall convergence is not affected much, leading to similar ﬁnal performance when the model is very deep and superior performance when the model is relatively shallow.

Original: Compared with other state-of-the-art methods: We consider four state-of-the-arts: LISTA with support selections (namely LISTA-C-S and LISTA-S, with and without the coupled constraint)
Revised: Compared with other state-of-the-art methods: We consider four state-of-the-arts: LISTA with support selections (namely LISTA-C-S and LISTA-S, with and without the coupled constraint)

Original: (Chen et al., 2018), analytic LISTA with support selections (ALISTA-S)
Revised: (Chen et al., 2018), analytic LISTA with support selections (ALISTA-S)

Original: (Liu et al., 2019), and learned AMP (LAMP) (Borgerding et al., 2017) for comparison, and their ofﬁcial implementations are directly used.
Revised: (Liu et al., 2019), and learned AMP (LAMP) (Borgerding et al., 2017) for comparison, and their ofﬁcial implementations are directly used.

Original: The hyper-parameters are set following the papers (Borgerding et al., 2017; Chen et al., 2018).
Revised: The hyper-parameters are set following the papers (Borgerding et al., 2017; Chen et al., 2018).

Original: We compare our GLISTA with these competitive methods under different levels of noises (including the signal-to-noise ratios (SNRs) being equal to 40dB, 20dB, and 10dB) and different condition numbers (including 3, 30, and 100, with SNR=40dB).
Revised: We compare our GLISTA with these competitive methods under different levels of noises (including the signal-to-noise ratios (SNRs) being equal to 40dB, 20dB, and 10dB) and different condition numbers (including 3, 30, and 100, with SNR=40dB).

Original: See Figure 7 for comparisons between LISTA,
Revised: See Figure 7 for comparisons between LISTA,

Original: LAMP, LISTA-S, LISTA-C-S, ALISTA-S, and our GLISTA in some of the settings.
Revised: LAMP, LISTA-S, LISTA-C-S, ALISTA-S, and our GLISTA in some of the settings.

Original: Obviously, the introduced gates facilitate LISTA signiﬁcantly, and the concerned NMSE diminishes the fastest using
Revised: Obviously, the introduced gates facilitate LISTA signiﬁcantly, and the concerned NMSE diminishes the fastest using

Original: GLISTA.
Revised: GLISTA.

Original: See our Appendix for comparisons of ﬁnal performance after multiple runs and the results in other settings (i.e., SNR: 20dB, 40dB, and condition number: 3).
Revised: See our Appendix for comparisons of ﬁnal performance after multiple runs and the results in other settings (i.e., SNR: 20dB, 40dB, and condition number: 3).

Original: We know from these results that using the gain gates solely can already outperforms existing state-of-the-arts, while incorporating the overshoot gates additionally may further boost the performance, as testiﬁed.
Revised: We know from these results that using the gain gates solely can already outperforms existing state-of-the-arts, while incorporating the overshoot gates additionally may further boost the performance, as testiﬁed.

Original: Applying our method to variants of LISTA: We also try adopting the introduced gates into some variants of LISTA to verify their “generalization ability”.
Revised: Applying our method to variants of LISTA: We also try adopting the introduced gates into some variants of LISTA to verify their “generalization ability”.

Original: Speciﬁcally, we incorporate the gain gates to
Revised: Speciﬁcally, we incorporate the gain gates to

Original: LFISTA (Moreau & Bruna, 2017) and ALISTA (Liu et al., 2019) to obtain GFLISTA and AGLISTA, respectively.
Revised: LFISTA (Moreau & Bruna, 2017) and ALISTA (Liu et al., 2019) to obtain GFLISTA and AGLISTA, respectively.

Original: Since ALISTA is suggested to be implemented with support set selection in the original
Revised: Since ALISTA is suggested to be implemented with support set selection in the original

Original: [Figure]
Revised: [Figure]

Original: paper, i.e. ALISTA-S, we also compare with it.
Revised: paper, i.e. ALISTA-S, we also compare with it.

Original: The experiment is performed under different levels of noises (40dB, 20dB, and 10dB).
Revised: The experiment is performed under different levels of noises (40dB, 20dB, and 10dB).

Original: As can be seen in Table 1 in which average results along with their standard deviations calculated over ﬁve runs are reported, models with our gain gates perform signiﬁcantly better, which veriﬁes that our method generalizes well
Revised: As can be seen in Table 1 in which average results along with their standard deviations calculated over ﬁve runs are reported, models with our gain gates perform signiﬁcantly better, which veriﬁes that our method generalizes well

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. P HOTOMETRIC S TEREO A NALYSIS
Revised: 4. P HOTOMETRIC S TEREO A NALYSIS

Original: We now test on a more practical task, i
Revised: We now test on a more practical task, i

Original: [Table]
Revised: [Table]

Original: e., photometric stereo analysis, using sparse coding.
Revised: e., photometric stereo analysis, using sparse coding.

Original: For a 3D object with Lambertian surface, if there are q different light conditions, a camera or some other kinds of sensors can obtain q different observations, all with noises caused by shadows and specularities.
Revised: For a 3D object with Lambertian surface, if there are q different light conditions, a camera or some other kinds of sensors can obtain q different observations, all with noises caused by shadows and specularities.

Original: The observations can be represented as a vector o ∈ R q for estimating the norm vector v ∈ R 3 at any position on the surface.
Revised: The observations can be represented as a vector o ∈ R q for estimating the norm vector v ∈ R 3 at any position on the surface.

Original: It is generally formulated as o = ρLv + e , in which L ∈ R q × 3 represents the normalized light directions ( q directions), e ∈ R q is a noise which is often sparse, ρ ∈ R represents the albedo reﬂectivity.
Revised: It is generally formulated as o = ρLv + e , in which L ∈ R q × 3 represents the normalized light directions ( q directions), e ∈ R q is a noise which is often sparse, ρ ∈ R represents the albedo reﬂectivity.

Original: Our task is to obtain v from o and L which is also known.
Revised: Our task is to obtain v from o and L which is also known.

Original: The estimation of e can be considered as a sparse coding problem, and one can use L † ( o − e ) to recover n given the estimation.
Revised: The estimation of e can be considered as a sparse coding problem, and one can use L † ( o − e ) to recover n given the estimation.

Original: More detailed descriptions of the task can be found in Xin et al.
Revised: More detailed descriptions of the task can be found in Xin et al.

Original: ’s paper (2016).
Revised: ’s paper (2016).

Original: In the sparse coding problem, we have Q ∈ R ( q − 3)
Revised: In the sparse coding problem, we have Q ∈ R ( q − 3)

Original: × q (the orthogonal complement of L ) as the dictionary matrix (i.e., A in Eq.
Revised: × q (the orthogonal complement of L ) as the dictionary matrix (i.e., A in Eq.

Original: e as the sparse code to be estimated, and Qo as the observation (i.e., y in Eq.
Revised: e as the sparse code to be estimated, and Qo as the observation (i.e., y in Eq.

Original: We mainly follow settings in Xin et al.
Revised: We mainly follow settings in Xin et al.

Original: ’s
Revised: ’s

Original: work, e.g. the vectors of L are randomly selected from the hemispherical surface, except that we test with q = 15 , 25 , 35 , and let 40% of the elements of e be non-zero.
Revised: work, e.g. the vectors of L are randomly selected from the hemispherical surface, except that we test with q = 15 , 25 , 35 , and let 40% of the elements of e be non-zero.

Original: We use GLISTA here to estimate e and the ﬁnal result for v is calculated as L † ( o − e ∗ ) , where L † ∈ R 3 × q is the pseudo-inverse of L and e ∗ is the estimation.
Revised: We use GLISTA here to estimate e and the ﬁnal result for v is calculated as L † ( o − e ∗ ) , where L † ∈ R 3 × q is the pseudo-inverse of L and e ∗ is the estimation.

Original: Our method is compared with LISTA and two traditional methods where no explicit training is introduced, i.e. the original least square (LS) and least L1, in Table 2.
Revised: Our method is compared with LISTA and two traditional methods where no explicit training is introduced, i.e. the original least square (LS) and least L1, in Table 2.

Original: Our evaluation metric is the mean ( ± standard deviation) error in degree and it is calculated using the bunny picture (Xin et al., 2016).
Revised: Our evaluation metric is the mean ( ± standard deviation) error in degree and it is calculated using the bunny picture (Xin et al., 2016).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we study LISTA for solving sparse coding problems.
Revised: In this paper, we study LISTA for solving sparse coding problems.

Original: We discover its potential weak- nesses and introduce gated mechanisms to address them accordingly.
Revised: We discover its potential weak- nesses and introduce gated mechanisms to address them accordingly.

Original: In particular, we theoretically prove that LISTA with gain gates can achieve faster convergence than the standard LISTA.
Revised: In particular, we theoretically prove that LISTA with gain gates can achieve faster convergence than the standard LISTA.

Original: We also discover that LISTA (with or without gates) can obtain lower reconstruction errors under a weaker assumption of “false positive” in its code estimations.
Revised: We also discover that LISTA (with or without gates) can obtain lower reconstruction errors under a weaker assumption of “false positive” in its code estimations.

Original: It helps us improve the convergence analyses to achieve more solid theoretical results, which have been perfectly conﬁrmed in simulation experiments.
Revised: It helps us improve the convergence analyses to achieve more solid theoretical results, which have been perfectly conﬁrmed in simulation experiments.

Original: The effectiveness of our introduced gates is veriﬁed in a variety of sparse coding experiments and the state-of-the-art performance is achieved.
Revised: The effectiveness of our introduced gates is veriﬁed in a variety of sparse coding experiments and the state-of-the-art performance is achieved.

Original: In the future, we aim to extend the method to convolutional neural networks to deal with more complex tasks.
Revised: In the future, we aim to extend the method to convolutional neural networks to deal with more complex tasks.

Original: A PPENDIX P ROOF OF T HEOREMS AND P ROPOSITIONS
Revised: A PPENDIX P ROOF OF T HEOREMS AND P ROPOSITIONS

Original: Before we delve deeply into the proof, we ﬁrst give some importance notations.
Revised: Before we delve deeply into the proof, we ﬁrst give some importance notations.

Original: We deﬁne S as the support of the vector x s , i.e. S = supp ( x s ) , and let |S| denote the number of elements in the set S .
Revised: We deﬁne S as the support of the vector x s , i.e. S = supp ( x s ) , and let |S| denote the number of elements in the set S .

Original: For a vector that shares the same size with x s , say z , we denote by z S ∈ R |S| a vector that keeps the elements with indices of z in S and removes the others.
Revised: For a vector that shares the same size with x s , say z , we denote by z S ∈ R |S| a vector that keeps the elements with indices of z in S and removes the others.

Original: If the vectors have been introduced with subscripts already, e.g. x s , we use ( x s ) S to denote vectors obtained in such a manner.
Revised: If the vectors have been introduced with subscripts already, e.g. x s , we use ( x s ) S to denote vectors obtained in such a manner.

Original: For a square matrix with the same number of row and column as the size of x s , say M , M ( S , S ) is its principal minor with the index set formed by removing rows and columns whose indices are not in S .
Revised: For a square matrix with the same number of row and column as the size of x s , say M , M ( S , S ) is its principal minor with the index set formed by removing rows and columns whose indices are not in S .

Original: Assume a vector x with no zero elements, sign ( · ) is deﬁned as ( sign ( x ))
Revised: Assume a vector x with no zero elements, sign ( · ) is deﬁned as ( sign ( x ))

Original: i =
Revised: i =

Original: x i / | x i | , i.e. ( sign ( x ))
Revised: x i / | x i | , i.e. ( sign ( x ))

Original: i = 1 when x i > 0 , and ( sign ( x ))
Revised: i = 1 when x i > 0 , and ( sign ( x ))

Original: i = − 1 when x i < 0 .
Revised: i = − 1 when x i < 0 .

Original: 6. P ROOF OF P ROPOSITION
Revised: 6. P ROOF OF P ROPOSITION

Original: Recall that the update rule of LISTA is x (0)
Revised: Recall that the update rule of LISTA is x (0)

Original: = 

Revised: =

Original: and x ( t +1) =
Revised: and x ( t +1) =

Original: s b ( t ) ( W ( t ) x
Revised: s b ( t ) ( W ( t ) x

Original: ( t )
Revised: ( t )

Original: + U ( t ) y ) , t = 0 , · · · , d − 1 .
Revised: + U ( t ) y ) , t = 0 , · · · , d − 1 .

Original: Proof.
Revised: Proof.

Original: Recall the deﬁnition of S is S = supp ( x s ) .
Revised: Recall the deﬁnition of S is S = supp ( x s ) .

Original: For the shrinking function z = s b ( t ) ( x ) = sign( x )( | x |− b ( t ) )
Revised: For the shrinking function z = s b ( t ) ( x ) = sign( x )( | x | − b ( t ) )

Original: + = x − bh ( z ) , where h ( z )
Revised: + = x − b ( t ) h ( z ) , where h ( z )

Original: = 

Revised: =

Original: if z > 0 , h ( z ) = − 1 if z < 0 , and h ( z ) ∈ [ − 1 , 1] if z = 0 .
Revised: if z > 0 , h ( z ) = − 1 if z < 0 , and h ( z ) ∈ [ − 1 , 1] if z = 0 .

Original: We use Mathematical Induction to prove supp( x ( t ) )
Revised: We use Mathematical Induction to prove supp( x ( t ) )

Original: ⊂ S , ∀ t = 0 , 1 , . . .
Revised: ⊂ S , ∀ t = 0 , 1 , . . .

Original: , d − 1 .
Revised: , d − 1 .

Original: We assume supp( x ( t ) )
Revised: We assume supp( x ( t ) )

Original: ⊂ S .
Revised: ⊂ S .

Original: From the calculation of x ( t +1) i , as W ( t ) =
Revised: From the calculation of x ( t +1) i , as W ( t ) =

Original: I − U ( t )
Revised: I − U ( t )

Original: A there is 
Revised: A there is x

Original: [Equation]
Revised: [Equation]

Original: h
Revised: t

Revised: [Equation]

Revised: [Equation]

Revised: i = s

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original:  i ) .
Revised: + ( x [Equation] ) i − b ( t ) h ( x ( t +1) i ) .

Original: For ∀ i / ∈ S , ( x s ) i = 0 .
Revised: For ∀ i / ∈ S , ( x s ) i = 0 .

Original: Let’s assume x ( t +1)
Revised: Let’s assume x ( t +1)

Original: i (cid:54) = 0 , then h ( x ( t +1) i ) =
Revised: i (cid:54) = 0 , then h ( x ( t +1) i ) =

Original: sign( x ( t +1) i ) .
Revised: sign( x ( t +1) i ) .

Original: Multiply the two sides of the Eq.
Revised: Multiply the two sides of the Eq.

Original: (23) by sign( x ( t +1) i ) , as the b ( t ) =
Revised: (23) by sign( x ( t +1) i ) , as the b ( t ) =

Original: µ ( A ) sup x s (cid:107) x
Revised: µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 , there will be | x 
Revised: ( t ) − x s (cid:107) 1 , there will be | x ( t +1)

Original: [Equation]
Revised: i | = (( I − U ( t ) A )( x ( t ) − x s ))

Original:  t + [Equation] i [Equation]
Revised: i sign( x ( t +1) i ) − b ( t ) =

Original: +
Revised: (( I −

Original: [Equation]
Revised: [Equation]

Original:  the inequality holds for (cid:126)a(cid:126)b ≤ (cid:107) a (cid:107) ∞ (cid:107) b (cid:107) 1 and (cid:107) I − U ( t ) A (cid:107) ∞ ≤ µ ( A ) .
Revised: 24) the inequality holds for (cid:126)a · (cid:126)b ≤ (cid:107) (cid:126)a (cid:107) ∞ (cid:107) (cid:126)b (cid:107) 1 and (cid:107) I − U ( t ) A (cid:107) ∞ ≤ µ ( A ) (because of U ( t ) ∈ W ( A ) ).

Original:  Eq.
Revised: From Eq.

Original: (24) is in conﬂict with x ( t +1)
Revised: (24), we know | x ( t +1) i | ≤ 0 actually is | x ( t +1) i | = 0 which is in conﬂict with x ( t +1)

Original: i (cid:54) = 0 .
Revised: i (cid:54) = 0 .

Original: Therefore, the x ( t +1) i = 0 , when i / ∈ S , i.e., supp( x ( t +1) )
Revised: Therefore, the x ( t +1) i = 0 , when i / ∈ S , i.e., supp( x ( t +1) )

Original: ⊂ S .
Revised: ⊂ S .

Original: As x (0) = 

Revised: As x (0) =

Original: ⊂ S , the supp( x ( t ) )
Revised: ⊂ S , the supp( x ( t ) )

Original: ⊂ S , ∀ t .
Revised: ⊂ S , ∀ t .

Original: The no “false positive” property has been proved.
Revised: The no “false positive” property has been proved.

Original: According to Eq.
Revised: According to Eq.

Original: (23), as support set of x s and x ( t ) are the subsets of S , there is
Revised: (23), as support set of x s and x ( t ) are the subsets of S , there is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: As supp( x ( t +1) )
Revised: As supp( x ( t +1) )

Original: ⊂ S , accumulate all | x ( t +1)
Revised: ⊂ S , accumulate all | x ( t +1)

Original: i − ( x s ) i | in Eq.
Revised: i − ( x s ) i | in Eq.

Original: (25) with i ∈ S , there is
Revised: (25) with i ∈ S , there is

Original: [Equation]
Revised: [Equation]

Original: The second equation is because of U ( t )
Revised: The second equation is because of U ( t )

Original: ∈ W ( A ) , so that | W i, : A : ,j | ≤ µ ( A ) when i (cid:54) =
Revised: ∈ W ( A ) , so that | W i, : A : ,j | ≤ µ ( A ) when i (cid:54) =

Original: j and | W i, : A : ,j | = 1 when i = j .
Revised: j and | W i, : A : ,j | = 1 when i = j .

Original: Substitute b ( t ) =
Revised: Substitute b ( t ) =

Original: µ ( A ) sup x s
Revised: µ ( A ) sup x s

Original: [Equation]
Revised: [Equation]

Original: x [Equation] t ) − x s (cid:107) 1 into Eq.
Revised: x ( t ) − x s (cid:107) 1 into Eq.

Original: (26), and take the supremum of Eq.
Revised: (26), and take the supremum of Eq.

Original: (26), there is
Revised: As |S| ≤ s there is

Original: [Equation]
Revised: [Equation]

Original: Let c = log((2 s − 1) µ ( A )) [Equation] the l 2 error bound of t -th layer in LISTA should be calculated as
Revised: Let c = log((2 s − 1) µ ( A )) , the l 2 error bound of t -th layer in LISTA should be calculated as

Original: [Equation]
Revised: [Equation]

Original: where the last Inequality is deduced for ( x s ) i ≤ B [Equation] and (cid:107) x s (cid:107) 0 ≤ s .
Revised: where the last inequality is deduced since ( x s ) i ≤ B , and (cid:107) x s (cid:107) 0 ≤ s .

Original: The linear convergence has been proved.
Revised: The linear convergence has been proved.

Original: Refer to the Eq.
Revised: Refer to the Eq.

Original: (25), we concentrate on{+as x ( t +1) i = 
 when+} i  ∈ S 
Revised: (25), {+as x ( t +1) i = 
 when+} i / ∈ S , | x ( t +1)

Original: [Equation]
Revised: i | ≤ | ( x s ) i | and x ( t +1) ( x s ) i ≥ 0 hold certainly. We only concentrate on i ∈ S x ( t +1)

Original:  ) i  ) .
Revised: i − ( x s ) i = (( I − U ( t ) A )( x ( t ) − x s )) i − b ( t ) h

Original:  If x ( t +1) i = 0 , there must be | x ( t +1)
Revised: ( x ( t +1) i ) . If x ( t +1) i = 0 , there must be | x ( t +1)

Original: i | = 0 ≤ ( x s ) i , and x ( t +1)
Revised: i | = 0 ≤ ( x s ) i , and x ( t +1)

Original: i ( x s ) i = 0 .
Revised: i ( x s ) i = 0 .

Original: If x ( t +1)
Revised: If x ( t +1)

Original: i > 0 , the according to Eq.
Revised: i > 0 , the according to Eq.

Original: (29), x ( t +1)
Revised: (29), x ( t +1)

Original: i − ( x s )
Revised: i − ( x s )

Original: i =
Revised: i =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i − b ( t ) =
Revised: i − b ( t ) =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i − µ ( A ) sup x s (cid:107) x
Revised: i − µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 ≤ 0 , i.e. , 0 < x ( t +1) i ≤ ( x s ) i , | x ( t +1) i | ≤ | ( x s ) i | , and x ( t +1)
Revised: ( t ) − x s (cid:107) 1 ≤ 0 , i.e., 0 < x ( t +1) i ≤ ( x s ) i , | x ( t +1) i | ≤ | ( x s ) i | , and x ( t +1)

Original: i ( x s ) i > 0 .
Revised: i ( x s ) i > 0 .

Original: If x ( t +1) i < 0 , the according to Eq.
Revised: If x ( t +1) i < 0 , the according to Eq.

Original: (29), x ( t +1)
Revised: (29), x ( t +1)

Original: i − ( x s )
Revised: i − ( x s )

Original: i =
Revised: i =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i + b ( t )
Revised: i + b ( t )

Original: =
Revised: =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i + µ ( A ) sup x s (cid:107) x
Revised: i + µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 ≥ 0 , i.e. , 0 > x ( t +1)
Revised: ( t ) − x s (cid:107) 1 ≥ 0 , i.e., 0 > x ( t +1)

Original: i ≥ ( x s ) i , | x ( t +1)
Revised: i ≥ ( x s ) i , | x ( t +1)

Original: i | ≤ | ( x s ) i | , and x ( t +1)
Revised: i | ≤ | ( x s ) i | , and x ( t +1)

Original: i ( x s ) i > 0 .
Revised: i ( x s ) i > 0 .

Original: In conclusion, we can obtain | x ( t +1) i | ≤ | ( x s ) i | and x ( t +1) ( x s ) i ≥ 0 for all the situations.
Revised: In conclusion, we can obtain | x ( t +1) i | ≤ | ( x s ) i | and x ( t +1) ( x s ) i ≥ 0 for all the situations.

Original: 6. P ROOF OF T HEOREM
Revised: 6. P ROOF OF T HEOREM

Original: Recall that the update rule of LISTA with gain gates is x (0)
Revised: Recall that the update rule of LISTA with gain gates is x (0)

Original: = 

Revised: =

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: We assume that b ( t ) is a vector, i.e., b ( t ) ∈ R n , in our proof to make it more general.
Revised: We assume that b ( t ) is a vector, i.e., b ( t ) ∈ R n , in our proof to make it more general.

Original: According to deﬁnition of the shrinking function s b ( t ) ( · ) and y =
Revised: According to deﬁnition of the shrinking function s b ( t ) ( · ) and y =

Original: Ax s , Eq.
Revised: Ax s , Eq.

Original: (10) isx
Revised: (10) is

Original: [Equation]
Revised: [Equation]

Revised: Concentrate on the situation of t → ∞ .

Original: Deﬁne g t ( x s ) =
Revised: Deﬁne g κ ( x s ) =

Original: ( g t ( x s , y | Λ ( t ) g )) when t → ∞ .
Revised: g t ( x s , y | Λ ( t ) g ) when t → ∞ .

Original: In Assumption 1, ∀ x s satisfying (cid:107) x s (cid:107) 

Revised: In the main body of Theorem 1, ∀ x s satisfying (cid:107) x s (cid:107)

Original: ≤ s is the ﬁxed point of Eq.
Revised: ≤ s is the ﬁxed point of Eq.

Original: (10) when t → ∞ .
Revised: (10) when t → ∞ .

Original: Eq.
Revised: Eq.

Original: (31) is
Revised: (31) is

Original: [Equation]
Revised: [Equation]

Original: The equation group of the indices in S in Eq.
Revised: The equation group of the indices in S in Eq.

Original: (32) is
Revised: (32) is

Original: [Equation]
Revised: [Equation]

Original: Let ( x s ) S → 0 but ( x s ) S (cid:54) = 

Revised: Let ( x s ) S → 0 but ( x s ) S (cid:54) =

Original: so that h (( x s ) S ) = sign (( x s ) S ) .
Revised: so that h (( x s ) S ) = sign (( x s ) S ) .

Original: As W ( t ) , U ( t ) , A and g κ ( x s )
Revised: As W ( t ) , U ( t ) , A and g κ ( x s )

Original: =
Revised: =

Original: g t ( x s , y | Λ ( t ) g ) are bounded, the right hand side of Eq.
Revised: g t ( x s , y | Λ ( t ) g ) are bounded, the right hand side of Eq.

Original: (33) is also tend to 0 , which is
Revised: (33) is also tend to 0 , which is

Original: [Equation]
Revised: [Equation]

Original: As the S can be selected arbitrarily as long as |S| ≤ s , b ( t ) also satisﬁes
Revised: As the S can be selected arbitrarily as long as |S| ≤ s , b ( t ) also satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Substitute the b ( t ) S of Eq.
Revised: Substitute the b ( t ) S of Eq.

Original: (34) into Eq.
Revised: (34) into Eq.

Original: (33), ( x s ) S is
Revised: (33), ( x s ) S is

Original: [Equation]
Revised: [Equation]

Original:  where the W ( t ) ( S , S ) is deﬁned at start of this section.
Revised: 36) where the W ( t ) ( S , S ) is deﬁned at start of this section.

Original: Eq.
Revised: Eq.

Original: (36) is
Revised: (36) is

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: [Equation] [Equation] [Equation] (37) where M = ( W ( t ) ( S , S )) − 1 ( I − U ( t ) A )( S , S ) .
Revised: 37) where M = ( W ( t ) ( S , S )) − 1 ( I − U ( t ) A )( S , S ) .

Original: The i -th row and j -th column element in M is denoted as m ij .
Revised: The i -th row and j -th column element in M is denoted as m ij .

Original: From Eq.
Revised: From Eq.

Original: (37), ( g κ ( x s ))
Revised: (37), ( g κ ( x s ))

Original: S is 
Revised: S is ( g

Original: [Equation]
Revised: [Equation]

Original: 
Revised: =   ((

Original: [Equation]
Revised: [Equation]

Original:  [Equation] m 1 |S| m
Revised: (( x s ) S ) −

Original: [Equation]
Revised: [Equation]

Original:  m [Equation] m 2 |S| ... m |S|m |S|... m |S||S| 
Revised: (( x s ) S ) − 1 |S|     mm... m 1 |S| mm... m 2 |S| ... m |S|m |S|... m |S||S| 

Original:    (( x s ) S )(
Revised:    (( x s ) S )(( x

Original: [Equation]
Revised: [Equation]

Revised: ) S )...

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  , where this equation should hold for all x s in Assumption 1.
Revised:  , where this equation should hold for all x s in Assumption 1.

Original: Assume ( x s ) S → 0 , for g κ ( x s ) is bounded, we can conclude that m ij = 0 , if i (cid:54) =
Revised: Assume ( x s ) S → 0 , for g κ ( x s ) is bounded, we can conclude that m ij = 0 , if i (cid:54) =

Original: j .
Revised: j .

Original: From Eq.
Revised: From Eq.

Original: (38), the ﬁnal form of g κ (( x s ) S ) is formulated as
Revised: (38), the ﬁnal form of g κ (( x s ) S ) is formulated as

Original: [Equation]
Revised: [Equation]

Original: From Eq.
Revised: From Eq.

Original: (38), we can conclude that g κ ( x s )
Revised: (38), we can conclude that g κ ( x s )

Original: i is a constant if i ∈ S , as the S could be arbitrary subset of { 1 , · · · , n } as long as |S| ≤ s .
Revised: i is a constant if i ∈ S , as the S could be arbitrary subset of { 1 , · · · , n } as long as |S| ≤ s .

Original: We could deduce that g κ ( x s ) i is constant ∀ i ∈ { 1 , . . . , n } and g κ ( x s ) must be constant vector, i.e.
Revised: We could deduce that g κ ( x s ) i is constant ∀ i ∈ { 1 , . . . , n } and g κ ( x s ) must be constant vector, i.e.

Original: [Equation]
Revised: [Equation]

Original: where D is an n × n constant diagonal matrix.
Revised: where D is an n × n constant diagonal matrix.

Original: The ﬁrst part of conclusion of Theorem 1 has been proved.
Revised: The ﬁrst part of conclusion of Theorem 1 has been proved.

Original: Substitute b ( t ) in Eq.
Revised: Substitute b ( t ) in Eq.

Original: (34) and diag( g κ ( x s )) in Eq.
Revised: (34) and diag( g κ ( x s )) in Eq.

Original: (39) into Eq.
Revised: (39) into Eq.

Original: (32), Eq.
Revised: (32), Eq.

Original: (32) is.
Revised: (32) is.

Original: [Equation]
Revised: [Equation]

Original: where Z = W ( t ) D + U ( t )
Revised: where Z = W ( t ) D + U ( t )

Original: A =
Revised: A =

Original: [ Z 1 , Z 2 , . .
Revised: [ Z 1 , Z 2 , . .

Original: , Z n ] and the Z i is the i -th column of Z .
Revised: , Z n ] and the Z i is the i -th column of Z .

Original: Give a x s satisfying only the i -th element of x s is non-zero and all the other elements are equal to zero, i.e., x s = [0 , 0 , . .
Revised: Give a x s satisfying only the i -th element of x s is non-zero and all the other elements are equal to zero, i.e., x s = [0 , 0 , . .

Original: . , ω, . .
Revised: . , ω, . .

Original: . , 0] T = ωe
Revised: . , 0] T = ωe

Original: i , in which e i is basis vector with only the i -th element being 1 and ω (cid:54) = 0 .
Revised: i , in which e i is basis vector with only the i -th element being 1 and ω (cid:54) = 0 .

Original: Substitute the x s = ωe i into Eq.
Revised: Substitute the x s = ωe i into Eq.

Revised: (40), there is

Original: [Equation]
Revised: [Equation]

Original: As the Eq.
Revised: As the Eq.

Original: (41) should hold for ∀ ω (cid:54) = 0 , we can deduce that Z i =
Revised: (41) should hold for ∀ ω (cid:54) = 0 , we can deduce that Z i =

Original: e i .
Revised: e i .

Original: As the i is selected arbitrarily,
Revised: As the i is selected arbitrarily,

Original: Z = W ( t ) D + U ( t )
Revised: Z = W ( t ) D + U ( t )

Original: A =
Revised: A =

Original: [ Z 1 , Z 2 , . .
Revised: [ Z 1 , Z 2 , . .

Original: , Z n ] =
Revised: , Z n ] =

Original: [ e 1 , e 2 , . . .
Revised: [ e 1 , e 2 , . . .

Original: , e n ] = I .
Revised: , e n ] = I .

Original: Thus we have completed the proof and get
Revised: Thus we have completed the proof and get

Original: [Equation]
Revised: [Equation]

Original: 6. P ROOF OF T HEOREM
Revised: 6. P ROOF OF T HEOREM

Original: Recall that the update rule of LISTA with gain gates is x (0)
Revised: Recall that the update rule of LISTA with gain gates is x (0)

Original: = 

Revised: =

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We simplify the g t ( x ( t ) , y | Λ ( t ) g ) as g t ( x ( t ) ) , and κ t ( x ( t ) , y | Λ ( t ) g ) as κ t ( x ( t ) ) .
Revised: We simplify the g t ( x ( t ) , y | Λ ( t ) g ) as g t ( x ( t ) ) , and κ t ( x ( t ) , y | Λ ( t ) g ) as κ t ( x ( t ) ) .

Original: According to the deﬁnition of gain gate in Eq.
Revised: According to the deﬁnition of gain gate in Eq.

Original: (43), we have
Revised: (43), we have

Original: [Equation]
Revised: [Equation]

Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Simplify the x ( t ) (cid:12) g
Revised: Simplify the x ( t ) (cid:12) g

Original: ( x ( t ) ) − x s as ∆ g x ( t ) .
Revised: ( x ( t ) ) − x s as ∆ g x ( t ) .

Original: For the i -th equation in Eq.
Revised: For the i -th equation in Eq.

Original: (44), and i / ∈ S , give the value of b ( t ) =
Revised: (44), and i / ∈ S , give the value of b ( t ) =

Original: µ ( A ) sup x s (cid:107) x
Revised: µ ( A ) sup x s (cid:107) x

Original: ( t ) (cid:12) g ( x ( t ) )
Revised: ( t ) (cid:12) g ( x ( t ) )

Original: − x s (cid:107) 1 , there is
Revised: − x s (cid:107) 1 , there is

Original: [Equation]
Revised: [Equation]

Original: With almost the same proof process in Theorem 1, we could deduce that
Revised: With almost the same proof process in Proposition 1, we could deduce that

Original: [Equation]
Revised: [Equation]

Original: 46) which is the no “false poistive” property.
Revised: which is the no “false poistive” property.

Original: Recall the Eq.
Revised: Recall the Eq.

Original: and substitute the 1 + κ t +1 ( x ( t +1) )
Revised: and substitute the 1 + κ t +1 ( x ( t +1) )

Original: =
Revised: =

Original: g t +1 ( x ( 
Revised: g t +1 ( x ( t +1) ) into it:

Original: [Equation]
Revised: [Equation]

Revised: We shall calculate the non-zero | ∆ g x ( t +1) i | with the index i .

Original: [Equation]
Revised: The i could be seperated to two parts. One is i ∈ S but i / ∈ supp ( x ( t +1) ) , another one part is i ∈ S and i ∈ supp( x ( t +1) ) .

Revised: Two kinds of i are discussed respectively.

Original: For i ∈ S
Revised: For i ∈ S

Original: but i / ∈ supp( x ( t +1) ) ,  x ( t +1)i
Revised: but i / ∈ supp( x ( t +1) ) , there must be x ( t +1)

Original:  = 0  .
Revised: i = 0 and − 1 ≤ h ( x ( t +1) i ) ≤ 1 .

Original: Select the i -th equation in Eq.
Revised: Select the i -th equation in Eq.

Original: (47), there is
Revised: (47), there is

Original: [Equation]
Revised: [Equation]

Original:  x ( [Equation] +
Revised: For i ∈ S and i ∈ supp( x ( t +1) ) , there must be x ( t +1)

Original: [Equation]{+(cid:54) = 
+}
Revised: {+(cid:54) = 
+}

Original: For i ∈ S and i ∈ supp( x ( t +1) ) , select the i -th equation in Eq.
Revised: and h ( x ( t +1) ) = sign( x ( t +1) ) . Select the i -th equation in Eq.

Original: (47), there is| ∆ g x ( t + [Equation] i
Revised: (47), there is

Original: [Equation]
Revised: [Equation]

Original: According to the condition in Eq.
Revised: According to the condition in Eq.

Original: (12) and (13), the 0 < κ t ( x ) (cid:12) | x | < 2 b ( t − 1) .
Revised: (12) and (13), the 0 < κ t ( x ) (cid:12) | x | < 2 b ( t − 1) .

Original: Then, | κ t ( x ) (cid:12) | x | − b ( t − 1) | < b ( t − 1) , there must ∃ η < 1 , so that | κ t ( x ) (cid:12) | x | − b ( t − 1) | ≤ ηb ( t − 1) <
Revised: Then, | κ t ( x ) (cid:12) | x | − b ( t − 1) | < b ( t − 1) , there must ∃ η < 1 , so that | κ t ( x ) (cid:12) | x | − b ( t − 1) | ≤ ηb ( t − 1) <

Original: b ( t − 1) .
Revised: b ( t − 1) .

Original: Substituting it to Eq.
Revised: Substituting it to Eq.

Original: (49), there is
Revised: (49), there is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Accumulate all the | ∆ g x ( t +1)
Revised: Accumulate all the | ∆ g x ( t +1)

Original: i | with all i ∈ S , and deﬁne s ( t ) =
Revised: i | with all i ∈ S , and deﬁne s ( t ) =

Original: | supp( x ( t ) ) | as the number of non-zeros elements in x ( t ) there is
Revised: | supp( x ( t ) ) | as the number of non-zeros elements in x ( t ) there is

Original: [Equation]
Revised: [Equation]

Original: Take the supremum of Eq.
Revised: Take the supremum of Eq.

Revised: (51), let s ( t ) ∗ denote the inﬁmum of s ( t ) with all of the x s ∈ X ( B, s, 0) , and c t = log((2 s − 1 − s ( t ) ∗

Original: [Equation]
Revised: [Equation]

Original: let c t = log((2 |S| − 1 −s ( t ) (1 − η ))
Revised: 1 − η ))

Original: µ ( A ))  there is
Revised: µ ( A )) , there is

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (52) gives the upper bound of sup x s (cid:107) ∆ g x ( t +1) (cid:107) 1 .{+, next we shall deduce the relationship between (cid:107) x ( t ) − x s (cid:107) 
 and it.+}
Revised: (52) gives the upper bound of sup x s (cid:107) ∆ g x ( t +1) (cid:107) 1 {+, next we shall deduce the relationship between (cid:107) x ( t ) − x s (cid:107) 
 and it.+}

Original: For the last layer ( t -th layer), from Eq.
Revised: For the last layer ( t -th layer), from Eq.

Original: (44), we have
Revised: (44), we have

Original: [Equation]
Revised: [Equation]

Original: Using almost the same process in Eq.
Revised: Using almost the same process in Eq.

Original: (26) and Eq.
Revised: (26) and Eq.

Original: (27), we could deduce Eq.
Revised: (27), we could deduce Eq.

Original: (53) that
Revised: (53) that

Original: [Equation]
Revised: [Equation]

Original: where  c = log((2 s − 1) µ ( A )) .
Revised: where the third inequality sign holds because of Eq. (52) and the last equation holds because of c = log((2 s − 1) µ ( A )) .

Original: As  c t ≤ c , there will be  (cid:107) x ( t ) − x s (cid:107) 1 ≤ exp( ct ) sB .
Revised: As at least c t ≤ c satisﬁes, there will be sup x s (cid:107) x ( t ) − x s (cid:107) 1 ≤ exp( ct ) sB .

Original: Set t [-
-] = (cid:100) log( sB (cid:107) x s (cid:107) 1 ) / log( 1 (2 s − 1) µ ( A ) ) (cid:101) .
Revised: Set t [-
-]0 = (cid:100) log( sB (cid:107) x s (cid:107) 1 ) / log( 1 (2 s − 1) µ ( A ) ) (cid:101) .

Original: When i > t 0 , exp( ci ) sB < (cid:107) x  s (cid:107) 1 , (cid:107)x
Revised: When i > t 0 , sup x s (cid:107) x ( i ) − x s (cid:107) 1 ≤ exp( ci ) sB < (cid:107)

Original:  ( i ) − x s (cid:107) 1 < (cid:107) x s (cid:107) 1 , then the s ( i )  = {+log(
+}
Revised: x s (cid:107) 1 , (cid:107) x ( i ) − x s (cid:107) 1 < (cid:107) x s (cid:107) 1 , then the s ( i ) ∗ = | supp( x ( i ) ) | > 0 , c i = {+log(
+}

Original: | supp( x ( i ) ) | > 0 , c i = log(2 |S| − 1 + s ( i )  (1 − η )) µ ( A ) < log((2 s − 1) µ ( A ))
Revised: s − 1 + s ( i ) ∗ (1 − η )) µ ( A ) < log((2 s − 1) µ ( A ))

Original: =
Revised: =

Original: c .
Revised: c .

Original: In conclusion, from Eq.
Revised: In conclusion, from Eq.

Original: , there is
Revised: , there is

Revised: [Equation]

Revised: t )

Original: [Equation]
Revised: [Equation]

Original: 55) where c = log((2 s − 1) µ ( A )) , c i = c
Revised: 55) where c = log((2 s − 1) µ ( A )) , c i = c

Original: when i ≤ t 0 , and c i < c when i > t 0 .
Revised: when i ≤ t 0 , and c i < c when i > t 0 .

Original: 6. P ROOF OF T HEOREM
Revised: 6. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: For the t -th layer of the LISTA, according to the Eq.
Revised: For the t -th layer of the LISTA, according to the Eq.

Original: (23), we have
Revised: (23), we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As we have remove the no false positive assumption, supp( x ( t ) i ) (cid:42) S .
Revised: As we have remove the no false positive assumption, supp( x ( t ) i ) (cid:42) S .

Original: Deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ supp( x ( t ) ) but i / ∈ S .
Revised: Deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ supp( x ( t ) ) but i / ∈ S .

Revised: In order to calculate all of non-zero x ( t +1)

Revised: i − ( x s ) i with index

Revised: i , we divide the i into two kinds, i ∈ S and i ∈ S ( t +1) .

Original: If i ∈ S , from Eq.
Revised: If i ∈ S , from Eq.

Original: (56), we can deduce the same formulation as Eq.
Revised: (56), we can deduce the same formulation as Eq.

Revised: (25): | x ( t +1)

Revised: i −

Original: [Equation]
Revised: [Equation]

Revised: x s

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: b

Revised: [Equation]

Original: If i ∈ S ( t +1) ,then the ( x s ) i = 0 .{+
 but x ( t +1) i , then h ( x ( t +1) i )+}
Revised: If i ∈ S ( t +1) , then the ( x s ) i = {+
 but x ( t +1) i , then h ( x ( t +1) i )+}

Revised: =

Revised: sign( x ( t +1) i ) .

Original: From Eq.
Revised: From Eq.

Original: (56), there is x ( t +1) i
Revised: (56), there is x ( t +1) i

Original: =
Revised: =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i − b ( t ) h
Revised: i − b ( t ) h

Original: ( x ( t +1) i ), x ( t +1)
Revised: ( x ( t +1) i )

Original: i =
Revised: =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i − b ( t ) sign( x ( t +1) i ) .
Revised: i − b ( t ) sign( x ( t +1) i ) .

Original: Multiply sign( x ( t +1) i ) on Eq.
Revised: Multiply sign( x ( t +1) i ) on Eq.

Original: (58), we have | x ( t +1)
Revised: (58), we have | x ( t +1)

Original: i | =
Revised: i | =

Original: (( I − U ( t ) A )( x ( t ) − x s ))
Revised: (( I − U ( t ) A )( x ( t ) − x s ))

Original: i sign( x ( t +1) i ) − b ( t ) , | x ( t+1) i | + b ( t ) =
Revised: i sign( x ( t +1) i ) − b ( t ) [Equation] | x ( t + [Equation]

Original: (( I − U ( t ) A )( x ( t ) − x s )) i sign( x ( t +1) i ) , ( | x ( t +1) i | + b ( t ) )sign( x ( t +1) i ) =
Revised: [Equation]

Original:  (( I − U ( t ) A )( x ( t ) − x s ))
Revised: which means (( I − U ( t ) A )( x ( t ) − x s ))

Original: i, (59) which means (( I − U ( t ) A )( x ( t ) − x s )) i have the same sign with b ( t  ) sign( x ( t +1) i  ) .
Revised: i have the same sign with sign( x ( t +1) i ) because | x ( t +1) i | >and b ( t ) > 0 , i.e., [Equation]

Original: From the Eq.
Revised: From the Eq.

Original: (58), there is x ( t +1) i − ( x s ) i = x ( t +1) i = sign( x ( t +1) i )( (cid:88)
Revised: (58), substitute Eq.

Original: j ∈ supp( x ( t ) ) ( I − U ( t ) A ) ij ( x ( t ) j − ( x s ) j ) − b ( t ) ) , | x ( t +1) i − ( x s ) i | = | x ( t +1)
Revised: (60) into Eq.

Original: i | = x ( t +1) i sign( x ( t +1) i [Equation]
Revised: (58), there is

Original: [Equation]
Revised: [Equation]

Original: Accumulate all the | x ( t +1)
Revised: Accumulate all the | x ( t +1)

Original: i − ( x s ) i | with i ∈ supp( x ( t +1) ) 
Revised: i − ( x s ) i | with i ∈ supp( x ( t +1) ) ∪ supp( x s )

Original: =
Revised: =

Original: S ( t +1) + S , there is
Revised: S ( t +1) + S , there is

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: t +

Revised: [Equation]

Revised: − x s

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Original: Substitute the b ( t ) =
Revised: Substitute the b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 into Eq.
Revised: ( t ) − x s (cid:107) 1 into Eq.

Original: (61), and take its supremum:
Revised: (62), and take its supremum of right part:

Original: [Equation]
Revised: [Equation]

Revised: Take the supremum of left part of (63), there is

Original: [Equation]
Revised: [Equation]

Original: Let c ∗ t +1{++
+} = log (( |S ( t +1) | + |S| + ( |S| − |S ( t +1) | ) ω
Revised: 64) where c ∗ t {++
+} = sup x s log(( |S ( t +1) | + |S| + ( |S| − |S ( t +1) | ) ω

Original: t +1 ( k t +1 | Θ))
Revised: t +1 ( k t +1 | Θ))

Original: µ ( A )) , and substitute it to Eq. (62), sup [Equation]
Revised: µ ( A )) .

Original: According to the deﬁnition of ω t ( · ) , b ( t ) =
Revised: According to the deﬁnition of ω t ( · ) , b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) x

Original: ( t ) − x s (cid:107) 1 , so that the number of false positive is less or equal than k t +1 , i.e. |S ( t +1) | ≤ k t +1 .
Revised: ( t ) − x s (cid:107) 1 , so that the number of false positive is less or equal than k t +1 , i.e. |S ( t +1) | ≤ k t +1 .

Original: As assumption of ω t +1 , 0 <
Revised: According to the previous proof, when b ( t ) =

Original: [Equation] < s [-, ω k +
-]
Revised: µ ( A ) sup x s [-, ω k +
-](cid:107) x

Original: (k t +10 | Θ) < 1 − 1 / ( s − k t +10 ) 3 .
Revised: ( t ) − x s (cid:107) 1 , the number of false positive satisﬁes k t +1 = 0 .

Original: Let k t +1 = k t +10 , c ∗ t +1 = log (( |S ( t +1) | + |S| + ( |S| − |S ( t +1) | ) ω
Revised: That means that ω (0 | Θ) ≤ 1 and ω ( k | Θ) ≤ 1 when k > 0 3 .

Original:  t +1 ( k t +1 | Θ)) µ ( A )) = log(( |S ( t +1) | (1 − ω t +1 ( k t +1 | Θ))
Revised: The c ∗ t +1 should be

Original: + |S| (1 + ω t +1 ( k t +1 | Θ)))

Original: µ ( A )) ≤ log( k t +1 (1 − ω t +1 ( k t +1 | Θ))

Original: + s (1 + ω t +1 ( k t +1 | Θ))

Original: µ ( A ) <

Original: [Equation]
Revised: [Equation]

Original: [Equation] t [Equation]
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  t [Equation] [Equation] [Equation]
Revised: As assumption of ω t +1 ( ·| Θ) , ∃ k t +10 , s.t. 0 < k t + < s , and ω t +1 ( k t +10 | Θ) < 1 − 1 / ( s − k t +10 ) .

Original: t [Equation] ( A )) [Equation]
Revised: Select the value of b ( t ) so that k t +1 =

Original: [Equation] t [Equation] [Equation]
Revised: k t +10 , we substitute it to Eq.

Original: t
Revised: (65), there will be

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: k t

Original: [Equation] [Equation] ( A [Equation]
Revised: Recall the Eq.

Original: = log((
Revised: (64), we have

Original: [Equation]
Revised: [Equation]

Original: s − 1) µ ( A )) .

Original: In conclusion, the l 2 error bound of the t -th layer of LISTA is
Revised: The l 2 error bound of the t -th layer of LISTA is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where c ∗ i < log((2 s − 1) µ ( A )) .
Revised: where c ∗ i < log((2 s − 1) µ ( A )) .

Original: 6. P ROOF OF T HEOREM
Revised: 6. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: For the t -th layer given in Eq.
Revised: For the t -th layer given in Eq.

Original: (10), according to Eq.
Revised: (10), according to Eq.

Original: [Equation]
Revised: [Equation]

Original: ) )) − b ( t ) h ( x ( t +1) )
Revised: and

Original: + x s , (66) and ∆ g x ( t +1) =
Revised: [Equation] [Equation] [Equation]

Original: ( I − U ( t ) A )
Revised: [Equation]

Original: (∆ g x ( t ) ) − b ( t ) h
Revised: t [Equation] [Equation]

Original: ( x ( t+1) )
Revised: + x [Equation] t

Original: + x ( t +1) (cid:12) κ t ( x ( t +1) ) .
Revised: [Equation] [Equation] [Equation] [Equation]

Original: As the no false positive is not ﬁt for x ( t ) , x ( t ) (cid:42) S .
Revised: As the no false positive is not ﬁt for x ( t ) , x ( t ) (cid:42) S .

Original: We still deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ supp( x ( t ) i ) but i / ∈ S and deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ S and i ∈ supp( x ( t )i ) .
Revised: We still deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ supp( x ( t ) i ) but i / ∈ S and deﬁne S ( t ) as ∀ i ∈ S ( t ) satisﬁes i ∈ S and i ∈ supp( x ( t ) ) .

Revised: In order to calculate the non-zero ∆ g x ( t +1) i , we divide the i into three situations: i ∈ S ( t +1) , i / ∈ S ( t +1) but i ∈ S , and i ∈ S ( t +1) .

Original: For i ∈ S ( t +1) ,  x ( t +1)
Revised: For i ∈ S ( t +1) , there must be x ( t +1)

Original: i (cid:54) = 0 , and ( x s ) i (cid:54) = 0 .
Revised: i (cid:54) = 0 , and ( x s ) i (cid:54) = 0 .

Original: Substitute the form of k t into i -th equation of Eq.
Revised: Substitute the form of κ t into i -th equation of Eq.

Original: (67): ∆ g x ( t +1) i = (( I − U ( t ) A )(∆ g x ( t ) )) i − b ( t ) sign( x ( t +1) ) + µ t +1 b ( t ) sign( x ( t +1) i ) = (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: [Equation]

Original: i − (1 − µ t +1 )

Original: b ( t ) sign( x ( t +1) ) , | ∆ g x ( t +1) i | ≤ | (( I − U ( t ) A )(∆ g x ( t ) )) i | + (1 − µ t +1 )
Revised: we have assume µ t ≤

Original: b ( t ) .

Original: For i / ∈ S ( t +1)
Revised: For i / ∈ S ( t +1)

Original: but i ∈ S , x ( t +1) i = 0 , and ( x s ) i (cid:54) = 0 .
Revised: but i ∈ S , x ( t +1) i = 0 , and ( x s ) i (cid:54) = 0 .

Original: The i -th equation of Eq.
Revised: The i -th equation of Eq.

Original: (67) is ∆ g x ( t +1)
Revised: (70) is [Equation]

Original: i =
Revised: [Equation]

Original: (( I − U ( t ) A )(∆ g x ( t ) )) i − b ( t ) sign( x ( t +1) i ) , | ∆ g x ( t +1) i | ≤ | (( I − U ( t ) A )(∆ g x ( t ) )) i | + b ( t ) .
Revised: [Equation]

Original: For i ∈ S ( t +1) , ( x s ) i = 0 .
Revised: For i ∈ S ( t +1) , ( x s ) i = 0 .

Original: ∆ g x ( t +1) i = x ( t +1)

Original: i g t ( x ( t +1) i ) = (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: [Equation]

Original: i − (1 − µ t+1 ) b (t ) sign( x ( t +1)  )  , (70) Multiply sign( x ( t +1) i ) on Eq. | x ( t +1)
Revised: As g t ( x ( t +1) i ) ≥ 1 , the sign of x ( t +1) i g t ( x ( t +1) i ) is the same as that of x ( t +1)

Original: i g t ( x ( t +1) i ) | = (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: i .

Original: i sign( x ( t +1)  ) − (1 − µ t +1 )
Revised: Multiply sign( x ( t +1) i ) on Eq.

Original: b ( t ) ,
Revised: (73), there is

Original: | x ( t +1) i g t ( x ( t +1) i ) | + (1 − µ t +1 ) b ( t ) =
Revised: [Equation]

Original:  (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: which means the (( I − U ( t ) A )(∆ g x ( t ) ))

Original: i sign( x ( t +1) ) , (71) which means the (( I − U ( t ) A )(∆ g x ( t ) ))

Original: i should have the same sign with sign( x ( t +1) ) , i.e.
Revised: i should have the same sign with sign( x ( t +1) ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: Accumulate all the | ∆ g x ( t +1)
Revised: Accumulate all the | ∆ g x ( t +1)

Original: i | with i ∈ supp( x ( t +1) )  , there is (cid:107)
Revised: i | with i ∈ supp( x ( t +1) ) ∪ supp( x s ) , there is [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: µ

Revised: [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Substitute the b ( t ) =
Revised: Substitute the b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107) 1 into Eq.
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107) 1 into Eq.

Original: Take the supremum of  Eq.
Revised: Take the supremum of the right part of Eq.

Original: [Equation]
Revised: [Equation]

Original:  Letc (cid:48) t + = log(( |S| + |S ( t +1)
Revised: x [Equation] Let

Original: | + (( | S ( t +1) | − |S ( t +1) | )(1 − µ t +1 ) + ( S − | S ( t +1) | ))
Revised: [Equation] +1) | − |S ( t +1) | )(1 − µ t +1 ) + ( |S| − | S ( t +1) | ))

Original: ω t +1 ( k t +1 | Θ))
Revised: ω t +1 ( k t +1 | Θ))

Original: µ ( A )) .
Revised: µ ( A )) .

Original: The sup x s(cid:107) ∆ g x ( t +1) (cid:107) 1 satisﬁes
Revised: Take the supremum of the left part of Eq. (77), the sup x s

Original: [Equation]
Revised: [Equation]

Revised: 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Consider about the relationship between x ( t +1) − x s and ∆ g x ( t ) :
Revised: After the upper bound of (cid:107) ∆ g x ( t ) (cid:107) 1 is deduced, we shall consider about the relationship between x ( t +1) − x s and ∆ g x ( t ) :

Original: If i ∈ S ( t +1) , ( x s ) i = 0 , and x ( t +1) i (cid:54) = 0 .
Revised: If i ∈ S ( t +1) , ( x s ) i = 0 , and x ( t +1) i (cid:54) = 0 .

Original: The i -th equation in Eq.
Revised: The i -th equation in Eq.

Original: (66) is 
Revised: (69) is x ( t +

Original: [Equation]
Revised: [Equation]

Original: According to the similar analyses in previous, the sign of (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: According to the similar analyses in previous, the sign of (( I − U ( t ) A )(∆ g x ( t ) ))

Original: i is the same as sign( x ( t +1) i ) , x ( t +1)
Revised: i is the same as sign( x ( t +1) i ) , x ( t +1)

Original: i − ( x s ) i satisﬁes
Revised: i − ( x s ) i satisﬁes

Original: [Equation]
Revised: [Equation]

Original: i [Equation] 77) i ∈ S , the x ( t +1) i − ( x s ) i satisﬁes
Revised: If i ∈ S , the x ( t +1) i − ( x s ) i satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Accumulate all the | x ( t +1)
Revised: Accumulate all the | x ( t +1)

Original: i − ( x s ) i | with i ∈ supp( x ( t +1) )  , there is (cid:107) x
Revised: i − ( x s ) i | with i ∈ supp( x ( t +1) ) ∪ supp( x s ) , there is (cid:107) x

Original: ( t +1) − x s (cid:107) 1 ≤ (cid:88) i ∈ supp( x ( t +1) )
Revised: ( t +1) − x s (cid:107) 1 ≤ (cid:88) i ∈ supp( x ( t +1) )

Original: | (( I − U ( t ) A )(∆ g x ( t ) ))
Revised: | (( I − U ( t ) A )(∆ g x ( t ) ))

Original: i | + ( |S| − S ( t +1)  ) | b ( t ) | ≤ ( |S| + S ( t +1)  ) µ ( A ) (cid:107) ∆ g x ( t ) (cid:107) 

Revised: i | + ( |S| − |S ( t +1) | ) | b ( t ) | ≤ ( |S| + |S ( t +1) | ) µ ( A ) (cid:107) ∆ g x ( t ) (cid:107)

Original: + ( |S| − S ( t +1)  ) b ( t ) .
Revised: + ( |S| − |S ( t +1) | ) b ( t ) .

Original: Substitute the b ( t ) =
Revised: Substitute the b ( t ) =

Original: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107) 1 into Eq.
Revised: ω t +1 ( k t +1 | Θ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107) 1 into Eq.

Original: (79), take the supremum of (cid:107)  x
Revised: (83), take the supremum of (cid:107) ∆ g x ( t ) (cid:107) 1 : (cid:107) x

Original: ( t +1) − x s (cid:107) 1and (cid:107) ∆ g x ( t ) (cid:107) 1 : sup x s (cid:107) x ( t +1) − x s (cid:107) 1 ≤ ( |S| + S ( t +1)  ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107)+ ( |S| − S ( t +1)  ) 
Revised: ( t +1) − x s (cid:107) 1 ≤ ( |S| + |S ( t +1) | ) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107)+ ( |S| − |S ( t +1) | ) ω

Original: ω t +1 ( k t +1 | Θ) sup x s (cid:107) ∆ g x ( t ) (cid:107)≤ ( |S| + S ( t +1) 
Revised: t +1 ( k t +1 | Θ) sup x s (cid:107) ∆ g x ( t ) (cid:107)≤ ( |S| + |S ( t +1) | +

Original: + ( |S| − S ( t +1) ) ω t +1 ( k t +1 | Θ)) µ ( A ) sup x s (cid:107) ∆ g x ( t ) (cid:107) 1 .
Revised: [Equation]

Original:  Let c (cid:48)∗ = log(( |S| + S ( t ) +
Revised: x [Equation] Let c (cid:48)∗ be

Original: ( |S| − S ( t ) ) ω t ( k t | Θ)) µ ( A )) .
Revised: [Equation]

Original: Substitute Eq.
Revised: Substitute Eq.

Original: (75) into Eq.
Revised: (79) and (85) into Eq.

Original: The lerror bound of LISTA with gain gate should be 
Revised: (84), the l 2 error bound of LISTA with gain gate should be (cid:107)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: t

Original: [Equation]
Revised: [Equation]

Original: sB (81)
Revised: Then we shall discuss the value of c (cid:48) i and c (cid:48) ∗ .

Original: Let t 

Revised: Let t

Original: = (cid:100) log( sBσ ) / log( 1 (2 s − 1)
Revised: = (cid:100) log( sBσ ) / log( 1 (2 s − 1)

Original: µ ( A ) ) (cid:101) .
Revised: µ ( A ) ) (cid:101) .

Original: When i ≤ t 0 , as µ i = 0 , c (cid:48) i = log(( |S| + |S ( i ) | + ( S −
Revised: When i ≤ t 0 , as µ i = 0 , which means the gain gate does not exist, there is

Original: S ( i ) )
Revised: [Equation]

Original:  ω i[-+
-] ( k i +1 | Θ)) µ ( A )) .
Revised: ∃ k i 0 , s.t. 0 < k i 0 < s and ω i[-+
-] ( k i 0 | Θ) < 1 − 1 / ( s − k i 0 ) .

Original: According to the prove mean process of Theorem 4, let k i
Revised: According to main process in the proof Theorem 3, let k i

Original: = k
Revised: = k

Original: i 0 , 0 < k i < s , and c (cid:48)
Revised: i 0 , 0 < k i < s , and c (cid:48)

Original: i =
Revised: i =

Original: c ∗ i < log((2 s − 1) µ ( A )) .
Revised: c ∗ i < log((2 s − 1) µ ( A )) .

Original: When i > t 0 ,  (cid:107) x
Revised: When i > t 0 , sup x s (cid:107) x

Original: ( i ) − x s (cid:107) 1 <
Revised: ( i ) − x s (cid:107) 1 <

Original: sB exp( ci ) ≤ σ  .
Revised: sB exp( ci ) ≤ σ ≤ min i ∈ supp( x s ) | ( x s ) i | .

Original: As the minimal absolute value of x s is less or equal than σ , S ( t ) =
Revised: As the minimal absolute value of x s is less or equal than σ , S ( t ) =

Original: S .
Revised: S .

Original: When k i = 
Revised: Select the b ( i ) so that k i = 0 , ω i ( k i | Θ) ≤ 1 . Recall the form in Eq.

Original:  |S( i) | =  0 ,  c (cid:48)
Revised: As sup x s |S i | = k i = 0 , and c (cid:48)

Original: i = |S| + |S| (1 − µ i ) .
Revised: i is [Equation]

Original: As 1 − ω i ( s | Θ) <
Revised: As 1 − ω i ( s | Θ) <

Original: µ i ≤ 1 , 1 − µ i < ω
Revised: µ i ≤ [Equation]

Original: i ( s | Θ) .
Revised: A [Equation]

Original: Recall the form the c ∗ i ≤ log(( |S| + |S ( i ) |{+
+} + ( S − S ( i ) )
Revised: < [Equation] < log(( s + k i {+
+} + ( s − k

Original: ω
Revised: i

Original: i ( k i | Θ) µ ( A )) . As |S| + |S| (1 − µ i ) < |S| + |S| ω i ( s | Θ) < |S| + |S| ω i ( k i | Θ) ≤ |S| + |S ( i ) | + ( |S| − |S
Revised: [Equation]

Original: ( i ) | ) ω 
Revised: ) ω i

Original: i ( k i | Θ) .
Revised: [Equation]

Original: The c (cid:48)
Revised: (89) i.e., c (cid:48)

Original: i < c ∗ i .
Revised: i < c ∗ i .

Revised: As ω i ( ·| Θ) is the monotone decreasing function, the second “<” in Eq.

Original: 6. P ROOF OF P ROPOSITION{+(89)holds since k i 0 < s and the last “<” holds since k i 0 > 0 . |S| ≤ s , and |S ( t ) | ≤ k t . According to the assumption of ω t ( ·| Θ) , ∃ k i 0 , s.t. 
+}
Revised: {+(89)holds since k i 0 < s and the last “<” holds since k i 0 > 0 . |S| ≤ s , and |S ( t ) | ≤ k t . According to the assumption of ω t ( ·| Θ) , ∃ k i 0 , s.t. 
+}

Revised: <

Revised: k

Revised: i 0 < s and ω i ( k i 0 | Θ) < 1 − 1 / ( s − k i 0 ) .

Revised: Select b ( t ) to let k t = k t 0 .

Revised: According to the similar derivation in Theorem 3, c (cid:48)∗ in Eq.

Revised: (85) should satisﬁes c (cid:48)∗ ≤ log(( s + k t

Revised: +( s − k t ) ω t ( k t | Θ))

Revised: µ ( A ))

Revised: <

Revised: log((2 s − 1) µ ( A )) .

Revised: All of the conclusions in Theorem 4 have been proven.

Revised: 6. P ROOF OF P ROPOSITION

Original: Recall that the update rule of ISTA is x (0)
Revised: Recall that the update rule of ISTA is x (0)

Original: = 

Revised: =

Original: andx ( [Equation] [Equation] −
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We have the following theorem which analyzes the update rule of ISTA and
Revised: We have the following theorem which analyzes the update rule of ISTA and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to the analysis in Section 2 in the main paper, x ( t +1) is the solution of minimizing the upper bound U ( x ) , 
Revised: According to the analysis in Section 2 in the main paper, x ( t +1) is the solution of minimizing the upper bound U ( x ) , [Equation]

Original: [Equation]
Revised: [Equation]

Original: The sub-gradient of U ( x ) is
Revised: The sub-gradient of U ( x ) is

Original: [Equation]
Revised: [Equation]

Original: [Equation] As the x ( t +1) is the optimal solution to minimizing Eq. 
Revised: As the x ( t +1) is the optimal solution to minimizing Eq. [Equation]

Original: (84), ∂ x U ( x ( t +1) ) satisﬁes
Revised: 92), ∂ x U ( x ( t +1) ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where r ( x ) = (cid:107) x (cid:107) 1 .
Revised: where r ( x ) = (cid:107) x (cid:107) 1 .

Original: According to the deﬁnition of the sub-gradient, ( ∂ x r ( x )) i ∈ [ − 1 , 1] when x i = 0 , ( ∂ x r ( x ))
Revised: According to the deﬁnition of the sub-gradient, ( ∂ x r ( x )) i ∈ [ − 1 , 1] when x i = 0 , ( ∂ x r ( x ))

Original: i = − 1 when x i < 0 , and ( ∂ x r ( x ))
Revised: i = − 1 when x i < 0 , and ( ∂ x r ( x ))

Original: i = 1 when x i > 0 .
Revised: i = 1 when x i > 0 .

Original: From the Eq.
Revised: From the Eq.

Original: (86), there must exists r 1 ∈ r ( x ) such that
Revised: (94), there must exists r 1 ∈ r ( x ) such that

Original: [Equation]
Revised: [Equation]

Original: (87) where ( r 1 ) i = 

Revised: where ( r 1 ) i =

Original: if x ( t +1) i > 0 , ( r 1 ) i = − 1 if x ( t +1) i < 0 , and − 1 ≤ ( r 1 ) i ≤ 1 if x ( t +1)
Revised: if x ( t +1) i > 0 , ( r 1 ) i = − 1 if x ( t +1) i < 0 , and − 1 ≤ ( r 1 ) i ≤ 1 if x ( t +1)

Original: i < 0 .
Revised: i < 0 .

Original: According to the deﬁnition of η ∗ in Eq.
Revised: According to the deﬁnition of η ∗ in Eq.

Original: (83), we deﬁne a new function θ ( η ) as
Revised: (91), we deﬁne a new function θ ( η ) as

Original: [Equation]
Revised: [Equation]

Original: Notice that θ ( η ) is the line search function of f ( x, y ) + λ (cid:107) x (cid:107) 1 .
Revised: Notice that θ ( η ) is the line search function of f ( x, y ) + λ (cid:107) x (cid:107) 1 .

Original: According to the law of convex optimization, as f ( x, y ) + λ (cid:107) x (cid:107) 1 is a convex function, the θ ( η ) must be also a convex function about η .
Revised: According to the law of convex optimization, as f ( x, y ) + λ (cid:107) x (cid:107) 1 is a convex function, the θ ( η ) must be also a convex function about η .

Original: The sub-gradient of θ ( η ) is
Revised: The sub-gradient of θ ( η ) is

Original: [Equation]
Revised: [Equation]

Original: The η ∗ actually is the value to minimize θ ( η ) in Eq.
Revised: The η ∗ actually is the value to minimize θ ( η ) in Eq.

Original: There must be0 ∈ ∂ x θ ( η ∗ ) .
Revised: There must be

Original: [Equation]
Revised: [Equation]

Original: From Eq.
Revised: From Eq.

Original: (89), the sub-gradient function of θ ( η )
Revised: (97), the sub-gradient function of θ ( η )

Original: when η = 1 is
Revised: when η = 1 is

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: where the last equation holds for Lagrange’s mean value theorem and ζ ∈ R n .
Revised: where the last equation holds for Lagrange’s mean value theorem and ζ ∈ R n .

Original: Substitute ∇ x f ( x ( t ) ) in Eq.
Revised: Substitute ∇ x f ( x ( t ) ) in Eq.

Original: (87) into Eq.
Revised: (95) into Eq.

Original: (91), the ∂ η θ (1) is
Revised: (99), the ∂ η θ (1) is

Original: [Equation]
Revised: [Equation]

Original: where the sub-gradient ∂r ( x ( t +1) ) is a set
Revised: where the sub-gradient ∂r ( x ( t +1) ) is a set.

Original: [Equation]

Original: For all r ∗ ∈ ∂r ( x ( t +1) ) ,
Revised: For all r ∗ ∈ ∂r ( x ( t +1) ) ,

Original: [Equation]
Revised: [Equation]

Original: We prove η ∗ ≥ 1 according to the counter-evidence.
Revised: We prove η ∗ ≥ 1 according to the counter-evidence.

Original: According to the properties of convex function and the sub-gradient, assume η ∗ < 1 , s.t. 0 ∈
Revised: According to the properties of convex function and the sub-gradient, assume η ∗ < 1 , s.t. 0 ∈

Original: ∂ η θ ( η ∗ )
Revised: ∂ η θ ( η ∗ )

Original: , ∀ r θ ∈ ∂
Revised: , ∀ r θ ∈ ∂

Original: η θ (1)
Revised: η θ (1)

Original: , there will be r θ > 0 .
Revised: , there will be r θ > 0 .

Original: However, as r 1 ∈ ∂r ( x ( t +1) ) , substitute r = r 1 ∈ ∂r ( x ( t +1) ) into Eq.
Revised: However, as r 1 ∈ ∂r ( x ( t +1) ) , substitute r = r 1 ∈ ∂r ( x ( t +1) ) into Eq.

Original: The corresponding element in sub-gradient when r = r 1 is r θ =
Revised: The corresponding element in sub-gradient when r = r 1 is r θ =

Original: ( x ( t +1) − x ( t ) )
Revised: [Equation] x [Equation]

Original: T ( ∇ 2 x f ( ζ ) − γI )( x ( t +1) − x ( t ) )

Original: ∈
Revised: ∂

Original: ∂ η θ (1) .
Revised: [Equation] θ (1) .

Original: According to given condition γI − ∇ 2 x f ( x ) (cid:31) 0 , r θ < 0 , which is in contrast to r θ > 0 .
Revised: According to given condition γI − ∇ 2 x f ( x ) (cid:31) 0 , r θ < 0 , which is in contrast to r θ > 0 .

Original: Therefore, the conclusion η ∗ ≥ 1 is obtained.
Revised: Therefore, the conclusion η ∗ ≥ 1 is obtained.

Original: Moreover, consider about the last term of Eq.
Revised: Moreover, consider about the last term of Eq.

Original: (92), i.e.
Revised: (100), i.e.

Original: [Equation]
Revised: [Equation]

Original:  If supp ( x ( t ) ) ⊂ supp ( x ( t +1) ) , there are two situations about index i . 1) i ∈ supp ( x ( t +1) ) , there will be x ( t +1)
Revised: i [Equation] If supp ( x ( t ) ) ⊂ supp ( x ( t +1) ) , there are two situations about index i . 1) i ∈ supp ( x ( t +1) ) , there will be x ( t +1)

Original: i (cid:54) = 

Revised: i (cid:54) =

Original: and ( ∂r ( x ( t +1) ))
Revised: and ( ∂r ( x ( t +1) ))

Original: i =
Revised: i =

Original: ( r 1 )
Revised: ( r 1 )

Original: i
Revised: i

Original: = sign ( x ( t +1) i ) .
Revised: = sign ( x ( t +1) i ) .

Original: 2) i / ∈ supp ( x ( t +1) ) and i / ∈ supp ( x ( t ) ) , there will be x ( t +1)
Revised: 2) i / ∈ supp ( x ( t +1) ) and i / ∈ supp ( x ( t ) ) , there will be x ( t +1)

Original: i =
Revised: i =

Original: x ( t ) i = 0 .
Revised: x ( t ) i = 0 .

Original: Both conditions will make the term ( x ( t +1)i − [Equation] t )
Revised: Both conditions will make the term ( x ( t +1)

Original: i

Original: [Equation]

Original:  ∂r [Equation] x [Equation]
Revised: i − x ( t ) i )(( ∂r ( x ( t +1) ))

Original: [Equation]

Original: ) i − ( r 1 ) i ) in Eq.
Revised: i − ( r 1 ) i ) in Eq.

Original: (94) be 0 .
Revised: (102) be 0 .

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (94) is
Revised: (102) is

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: According to the given condition γI − ∇ 2 x f ( x ) (cid:31) 0 , ∂ η θ (1) should be a number but not a set and ∂ η θ (1) =
Revised: According to the given condition γI − ∇ 2 x f ( x ) (cid:31) 0 , ∂ η θ (1) should be a number but not a set and ∂ η θ (1) =

Original: ( x ( t +1) − x ( t ) )
Revised: ( x ( t +1) − x ( t ) )

Original: T ( ∇ 2 x f ( ζ ) − γI )( x ( t +1) − x ( t ) ) < 0 .
Revised: T ( ∇ 2 x f ( ζ ) − γI )( x ( t +1) − x ( t ) ) < 0 .

Original: As the θ ( η ) is convex function, there must be η ∗ > 1 because of 0 ∈ ∂ η θ ( η ∗ ) .
Revised: As the θ ( η ) is convex function, there must be η ∗ > 1 because of 0 ∈ ∂ η θ ( η ∗ ) .

Original: The conclusion η ∗ > 1 is derived.
Revised: The conclusion η ∗ > 1 is derived.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  M ORE S IMULATION E XPERIMENTS
Revised:  M ORE S IMULATION E XPERIMENTS

Original: Validation of Proposition 2 : Some more experimental results are given here due to the length limit of the main body of our paper.
Revised: Validation of Proposition 2 : Some more experimental results are given here due to the length limit of the main body of our paper.

Original: One might also be interested in our Proposition 2, hence we ﬁrst conduct an experiment to conﬁrm it.
Revised: One might also be interested in our Proposition 2, hence we ﬁrst conduct an experiment to conﬁrm it.

Original: We adopt ISTA with an adaptive overshoot and compare it with the standard ISTA for sparse coding.
Revised: We adopt ISTA with an adaptive overshoot and compare it with the standard ISTA for sparse coding.

Original: The adaptation is obtained via enlarging the step size from 1.
Revised: The adaptation is obtained via enlarging the step size from 1.

Original: through backtracking line search (see section 8 for more details).
Revised: through backtracking line search (see section 8 for more details).

Original: Figure 8 demonstrates that our overshoot mechanism facilitates ISTA optimization, and such a result conﬁrms Proposition 2.
Revised: Figure 8 demonstrates that our overshoot mechanism facilitates ISTA optimization, and such a result conﬁrms Proposition 2.

Original: Comparison with similar methods : As mentioned in the main body of the paper, the overshoot gates is proposed do address insufﬁcient step size, which is similar to the motivation of (L)FISTA.
Revised: Comparison with similar methods : As mentioned in the main body of the paper, the overshoot gates is proposed do address insufﬁcient step size, which is similar to the motivation of (L)FISTA.

Original: LIHT and support select can also be considered as special cases of our gain gates (by letting µ t =in the inverse proportional function).
Revised: LIHT and support select can also be considered as special cases of our gain gates (by letting µ t =in the inverse proportional function).

Original: We compare these similar methods with our overshoot and gain gates in Figure 9.
Revised: We compare these similar methods with our overshoot and gain gates in Figure 9.

Original: It can be seen that when compared with LISTA, LFISTA converges faster in lower layers, and our overshoot gates also show such advantage.
Revised: It can be seen that when compared with LISTA, LFISTA converges faster in lower layers, and our overshoot gates also show such advantage.

Original: When applying to deeper layers,
Revised: When applying to deeper layers,

Original: LFISTA converges quite slow while the overshoot gates still perform well, which indicates that the time-varying property is beneﬁcial in practice.
Revised: LFISTA converges quite slow while the overshoot gates still perform well, which indicates that the time-varying property is beneﬁcial in practice.

Original: LISTA with our gain gates is obviously better than
Revised: LISTA with our gain gates is obviously better than

Original: LIHT as shown in Figure 9(b), and sufﬁcient experimental results in the paper also prove that the gain gate outperforms support select (e.g., in LISTA-C-S and LISTA-S).
Revised: LIHT as shown in Figure 9(b), and sufﬁcient experimental results in the paper also prove that the gain gate outperforms support select (e.g., in LISTA-C-S and LISTA-S).

Original: Comparison under less challenging settings : Now we also give sparse coding results under the described less challenging settings on the noise level and the condition number in Figure 10.
Revised: Comparison under less challenging settings : Now we also give sparse coding results under the described less challenging settings on the noise level and the condition number in Figure 10.

Original: Compared with LISTA-CP, LAMP, LISTA-SS, and LISTA-CP-SS, our gated LISTA (GLISTA) performs remarkably better with less ill-posed dictionary matrices and less noises.
Revised: Compared with LISTA-CP, LAMP, LISTA-SS, and LISTA-CP-SS, our gated LISTA (GLISTA) performs remarkably better with less ill-posed dictionary matrices and less noises.

Original: Table 3 and 4 report the statistical means and standard deviations of ﬁve runs using different methods.
Revised: Table 3 and 4 report the statistical means and standard deviations of ﬁve runs using different methods.

Original: It can be seen that the improvement achieved by our GLISTA is signiﬁcant
Revised: It can be seen that the improvement achieved by our GLISTA is signiﬁcant

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  P ROGRESSIVE T RAINING AND A DAPTIVE O VERSHOOT
Revised:  P ROGRESSIVE T RAINING AND A DAPTIVE O VERSHOOT

Original: Our training mostly follows it of Chen et al.’s
Revised: Our training mostly follows it of Chen et al.’s

Original: (2018), and some key steps are listed here: 1)
Revised: (2018), and some key steps are listed here: 1)

Original: The model is trained progressively to include more layers during the training phase.
Revised: The model is trained progressively to include more layers during the training phase.

Original: At the very beginning, only learnable parameters in the ﬁrst layer is considered, and parameters in the second layer is only included once training on the ﬁrst update converges, so as the third and higher layers.
Revised: At the very beginning, only learnable parameters in the ﬁrst layer is considered, and parameters in the second layer is only included once training on the ﬁrst update converges, so as the third and higher layers.

Original: 2) Training after including the t -th layer is split into three stages, with an initial learning rate of 0 . 0005 to optimize its own learnable parameters ﬁrst, and learning rates of 0 . 0001 and 0 .
Revised: 2) Training after including the t -th layer is split into three stages, with an initial learning rate of 0 . 0005 to optimize its own learnable parameters ﬁrst, and learning rates of 0 . 0001 and 0 .

Original: 00001 to jointly optimize all learnable parameters from the 0 -th to t -th layers in the second and third stages, respectively.
Revised: 00001 to jointly optimize all learnable parameters from the 0 -th to t -th layers in the second and third stages, respectively.

Original: We move to the next stage once no performance gain is observed on the validation set for 4000 iterations.
Revised: We move to the next stage once no performance gain is observed on the validation set for 4000 iterations.

Original: 3) With the three stages done on the t -th layer, training moves to include the ( t + 1) -th and the same three stages of training are performed.
Revised: 3) With the three stages done on the t -th layer, training moves to include the ( t + 1) -th and the same three stages of training are performed.

Original: We perform an adaptive overshoot in the experiment to conﬁrm Proposition 2.
Revised: We perform an adaptive overshoot in the experiment to conﬁrm Proposition 2.

Original: The algorithm is summarized in Algorithm 1.
Revised: The algorithm is summarized in Algorithm 1.

Original: Most of input variables are introduced in the main body of our paper and τ is given as the step size for performing line search.
Revised: Most of input variables are introduced in the main body of our paper and τ is given as the step size for performing line search.

Original: The whole algorithm procedure is very similar to the famous backtracking line search.
Revised: The whole algorithm procedure is very similar to the famous backtracking line search.

Original: The step size η for sparse coding is updated by τ until the objective function f ( x, y )
Revised: The step size η for sparse coding is updated by τ until the objective function f ( x, y )

Original: + λr ( x ) does not decrease any more.
Revised: + λr ( x ) does not decrease any more.

Original: Algorithm 1 ISTA with adaptive overshoot.
Revised: Algorithm 1 ISTA with adaptive overshoot.

Original: Input: The dictionary matrix A , an observation y , an initial step size η 0 = 1 . 

Revised: Input: The dictionary matrix A , an observation y , an initial step size η 0 = 1 .

Original: for sparse coding, a step size τ = 1 . 05 for line search, and a maximal number of iteration.
Revised: for sparse coding, a step size τ = 1 . 05 for line search, and a maximal number of iteration.

Original: Output: output result 1: x (0) = 0 ; 2: for t =
Revised: Output: output result 1: x (0) = 0 ; 2: for t =

Original: , · · · , K − 1 do 3: ˜ x ( t ) =
Revised: , · · · , K − 1 do 3: ˜ x ( t ) =

Original: s λ/γ (( I − A T A/γ ) x
Revised: s λ/γ (( I − A T A/γ ) x

Original: ( t − 1) + A T y/γ ) ; 4: x p = ˜ x ( t ) , η = η 0 τ ; 5: x c = τ
Revised: ( t − 1) + A T y/γ ) ; 4: x p = ˜ x ( t ) , η = η 0 τ ; 5: x c = τ

Original: (˜ x ( t ) − x ( t ) ) − x ( t ) ; 6: while f o ( x p , y ) ≥ f o ( x c , y ) do 7: x p = x c , η = τη ; 8: x c = η (˜ x ( t ) − x ( t ) )
Revised: (˜ x ( t ) − x ( t ) ) − x ( t ) ; 6: while f o ( x p , y ) ≥ f o ( x c , y ) do 7: x p = x c , η = τη ; 8: x c = η (˜ x ( t ) − x ( t ) )

Original: − x ( t ) ; 9: x ( t ) = x p ; 10: return x ( K − 1)
Revised: − x ( t ) ; 9: x ( t ) = x p ; 10: return x ( K − 1)