Original: P ROVABLE L EARNING - BASED A LGORITHM F OR S PARSE R ECOVERY
Revised: P ROVABLE L EARNING - BASED A LGORITHM F OR S PARSE R ECOVERY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recovering sparse parameters from observational data is a fundamental problem in machine learning with wide applications.
Revised: Recovering sparse parameters from observational data is a fundamental problem in machine learning with wide applications.

Original: Many classic algorithms can solve this problem with theoretical guarantees, but the theoretical performances depend on choosing the correct hyperparameters.
Revised: Many classic algorithms can solve this problem with theoretical guarantees, but their performances rely on choosing the correct hyperparameters.

Original: Besides, they do not fully exploit the particular problem distribution of interest.
Revised: Besides, hand-designed algorithms do not fully exploit the particular problem distribution of interest.

Original: In this work, we propose  PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm) to learn algorithms automatically from data.
Revised: In this work, we propose a deep learning method for algorithm learning called PLISA ( P rovable L earningbased I terative S parse recovery A lgorithm).

Original: PLISA is designed by unrolling a classic path-following algorithm, with some components being more ﬂexible and learnable.
Revised: PLISA is designed by unrolling a classic path-following algorithm for sparse recovery, with some components being more ﬂexible and learnable.

Original: With this structure, we theoretically show the improved recovery accuracy achievable by PLISA .
Revised: We theoretically show the improved recovery accuracy achievable by PLISA .

Original: Furthermore, we analyze the empirical Rademacher complexity of PLISA to characterize its generalization ability to solve new problems outside the training set.
Revised: Furthermore, we analyze the empirical Rademacher complexity of PLISA to characterize its generalization ability to solve new problems outside the training set.

Original: This paper contains novel theoretical contributions to the area of learning-based algorithms in the sense that (i) PLISA is generically applicable to a broad class of sparse estimation problems, (ii) generalization analysis has received less attention so far, and (iii) our analysis makes novel connections between the generalization ability and algorithmic properties such as stability and convergence of the unrolled algorithm, which leads to a tighter bound that can explain the empirical observations.
Revised: This paper contains novel theoretical contributions to the area of learning-based algorithms in the sense that (i) PLISA is generically applicable to a broad class of sparse estimation problems, (ii) generalization analysis has received less attention so far, and (iii) our analysis makes novel connections between the generalization ability and algorithmic properties such as stability and convergence of the unrolled algorithm, which leads to a tighter bound that can explain the empirical observations.

Original: The techniques could potentially be applied to analyze other learning-based algorithms in the literature.
Revised: The techniques could potentially be applied to analyze other learning-based algorithms in the literature.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: The problem of recovering a sparse vector β ∗ from ﬁnite observations Z 1: n ∼ ( P β ∗ ) n is fundamental in machine learning, covering a broad family of problems including compressed sensing, sparse regression analysis, graphical model estimation, etc.
Revised: The problem of recovering a sparse vector β ∗ from ﬁnite observations Z 1: n ∼ ( P β ∗ ) n is fundamental in machine learning, covering a broad family of problems including compressed sensing, sparse regression analysis, graphical model estimation, etc.

Original: It has also found applications in various domains.
Revised: It has also found applications in various domains.

Original: For example, in magnetic resonance imaging, sparse signals need to be reconstructed from measurements taken by a scanner.
Revised: For example, in magnetic resonance imaging, sparse signals need to be reconstructed from measurements taken by a scanner.

Original: In computational biology, estimating a sparse graph struc - ture from gene expression data is important for understanding gene regulatory networks
Revised: In computational biology, estimating a sparse graph structure from gene expression data is important for understanding gene regulatory networks.

Original: Various classic algorithms are available for solving sparse recovery problems.
Revised: Various classic algorithms are available for solving sparse recovery problems.

Original: Many of them come with theoretical guarantees for the recovery accuracy.
Revised: Many of them come with theoretical guarantees for the recovery accuracy.

Original: However, the theoretical performance often relies on choosing the correct hyperparameters, such as regularization parameters and the learning rate, which may depend on unknown constants.
Revised: However, the theoretical performance often relies on choosing the correct hyperparameters, such as regularization parameters and the learning rate, which may depend on unknown constants.

Original: Furthermore, in practice, similar problems may need to be solved repeatedly, but it is hard for classic algorithms to fully utilize this information.
Revised: Furthermore, in practice, similar problems may need to be solved repeatedly, but it is hard for classic algorithms to fully utilize this information.

Original: To alleviate these limitations, we consider the approach of learning-to-learn and propose a neural algorithm, called PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm).
Revised: To alleviate these limitations, we consider the approach of learning-to-learn and propose a neural algorithm, called PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm).

Original: PLISA is a deep learning model that takes the observations Z 1: n as the input and outputs an estimation for β ∗ .
Revised: PLISA is a deep learning model that takes the observations Z 1: n as the input and outputs an estimation for β ∗ .

Original: To make use of classic techniques developed by domain experts, we design the architecture of PLISA by unrolling and modifying a classic path-following algorithm proposed by Wang et al.
Revised: To make use of classic techniques developed by domain experts, we design the architecture of PLISA by unrolling and modifying a classic path-following algorithm proposed by Wang et al.

Original: (2014b).

Original: To beneﬁt from learning, some components in this classic algorithm are made more ﬂexible with careful design and treated as learnable parameters in PLISA .
Revised: To beneﬁt from learning, some components in this classic algorithm are made more ﬂexible with careful design and treated as learnable parameters in PLISA .

Original: These parameters can be learned by optimizing the performances on a set of training problems.
Revised: These parameters can be learned by optimizing the performances on a set of training problems.

Original: The learned PLISA can then be used for solving other problems in the target distribution.
Revised: The learned PLISA can then be used for solving other problems in the target distribution.

Original: With the algorithm design problem converted to a deep learning problem, wenaturally ask the two important questions:
Revised: With the algorithm design problem converted to a deep learning problem, we ask the two fundamental questions in learning theory:

Original: Capacity: What’s the recovery accuracy achievable by PLISA ?
Revised: Capacity: What’s the recovery accuracy achievable by PLISA ?

Original: Can the ﬂexible components in PLISA lead to an algorithm which effectively improves the recovery performance?
Revised: Can the ﬂexible components in PLISA lead to an algorithm which effectively improves the recovery performance?

Original: Generalization: How well can the learned PLISA solve new problems outside the training set?
Revised: Generalization: How well can the learned PLISA solve new problems outside the training set?

Original: Is the generalization behavior related to the algorithmic properties of PLISA ?
Revised: Is the generalization behavior related to the algorithmic properties of PLISA ?

Original: Aiming at supplying rigorous answers to these questions, we conduct theoretical analysis of PLISA to provide guarantees for its recovery performance, measure its empirical Rademachar complexity, andcharacterize its generalization ability.
Revised: Aiming at supplying rigorous answers to these questions, we conduct theoretical analysis for PLISA to provide guarantees for its representation and generalization ability.

Original: The results and the techniques in our analysis can distinguish our work from existing studies on algorithm learning, which has become an active research direction recently.
Revised: The results and the techniques in our analysis can distinguish our work from existing studies on algorithm learning.

Original: We summarize ournew and novel contributions into the following three aspects.
Revised: We summarize our novel contributions into the following three aspects.

Original: Theoretical understanding.
Revised: Theoretical understanding.

Original: In contrast to the plethora of empirical studies on algorithm learning, there have been relatively few studies devoted to the theoretical understanding.
Revised: In contrast to the plethora of empirical studies on algorithm learning, there have been relatively few studies devoted to the theoretical understanding.

Original: Existing theoretical efforts primarily focus on analyzing the convergence rate achievable by the neural algorithm
Revised: Existing theoretical efforts primarily focus on analyzing the convergence rate achievable by the neural algorithm

Original: (Chen et al., 2018; Liu et al., 2019a; Zhang & Ghanem, 2018; Wu et al., 2020), but the generalization error bound has received less attention so far.
Revised: (Chen et al., 2018; Liu et al., 2019a; Zhang & Ghanem, 2018; Wu et al., 2020), but the generalization error bound has received less attention so far.

Original: A substantial body of works only argue intuitively that algorithm unrolling architectures can generalize well because they contain a small number of parameters.
Revised: A substantial body of works only argue intuitively that algorithm unrolling architectures can generalize well because they contain a small number of parameters.

Original: In comparison, we provide theoretical guarantees for both the capacity and the generalization ability of PLISA , which are more solid arguments.
Revised: In comparison, we provide theoretical guarantees for both the capacity and the generalization ability of PLISA , which are more solid arguments.

Original: Novel connection.

Original: The algorithmic structure in PLISA can make it behaves differently from conventional neural networks.

Original: Therefore, we largely utilize the analysis techniques in classic algorithms to derive its generalization bound.

Original: By combining the analysis tools of deep learning theory and optimization algorithms, our result reveals a novel connection between the generalization ability of

Original: PLISA and the algorithmic properties including the convergence rate and stability of the unrolled algorithm.

Original: Beneﬁt from this connection, our generalization bound is tight in the sense that it matches the interesting behavior of PLISA observed in experiments - the generalization gap could decrease in the number of layers, which is rarely observed in conventional neural networks.

Original: General setting.
Revised: General setting.

Original: The problem setting in this paper is new and more challenging.
Revised: The problem setting in this paper is new and more challenging.

Original: Existing works mainly focus on a speciﬁc problem.
Revised: Existing works mainly focus on a speciﬁc problem.

Original: For example, the compressed sensing problem with a ﬁxed design matrix is the mostly investigated one.
Revised: For example, the compressed sensing problem with a ﬁxed design matrix is the mostly investigated one.

Original: PLISA , however, is generic and is applicable to various sparse recovery problems as long as they satisfy certain conditions in Assumption 3.1.
Revised: PLISA , however, is generic and is applicable to various sparse recovery problems as long as they satisfy certain conditions in Assumption C.1.

Original:  PLISA : L EARNING T O S OLVE S PARSE E STIMATION P ROBLEMS
Revised: Novel connection The algorithmic structure in PLISA can make it behaves differently from conventional neural networks.

Original: [Equation] input [Equation] [Equation]
Revised: Therefore, we largely utilize the analysis techniques in classic algorithms to derive its generalization bound.

Original: [Figure] [Equation] ∗ … … [Figure]
Revised: By combining the analysis tools of deep learning theory and optimization algorithms, our result reveals a novel connection between the generalization ability of PLISA and the algorithmic properties including the convergence rate and stability of the unrolled algorithm.

Original: [Equation] [Figure]
Revised: Beneﬁt from this connection, our generalization bound is tight in the sense that it matches the interesting behavior of PLISA observed in experiments - the generalization gap could decrease in the number of layers, which is rarely observed in conventional neural networks.  PLISA : L EARNING T O S OLVE S PARSE E STIMATION P ROBLEMS

Original: A sparse estimation problem is to recover β ∗ from ﬁnite observations Z 1: n sampled from P β ∗ .
Revised: A sparse estimation problem is to recover β ∗ from ﬁnite observations Z 1: n sampled from P β ∗ .

Original: As a concrete example, in a sparse linear regression problem, n observations { Z i = ( x i , y i ) } ni =1 are sampled from a linear model y
Revised: As a concrete example, in a sparse linear regression problem, n observations { Z i = ( x i , y i ) } ni = are sampled from a linear model y

Original: = x (cid:62) β ∗ + (cid:15) , and an algorithm needs to estimate β ∗ from n observations.
Revised: = x (cid:62) β ∗ + (cid:15) , and an algorithm needs to estimate β ∗ from n observations.

Original: Therefore, an oracle algorithm can be viewed as a function that maps Z 1: n to β ∗ :
Revised: Classic algorithms recover β ∗ by minimizing a regularized empirical loss: [Equation] where L n is an empirical loss that measures the “ﬁt” between the parameter β and observations Z 1: n , and P ( λ, β ) is a sparsity regularization with coefﬁcient λ .

Original: Oracle algorithm (function): Z 1: n (cid:55)→ β ∗ .
Revised: When L n is the least square loss and P ( λ, β ) is λ (cid:107) β (cid:107) 1 , the optimization is known as LASSO and can be solved by the well-known algorithm ISTA (Daubechies et al., 2004). Based on the idea of algorithm unrolling , Gregor & LeCun (2010) proposed LISTA, a neural algorithm that interprets ISTA as layers of neural networks. It has been demonstrated that LISTA outperforms ISTA thanks to its learnable components.

Original: Based on the equivalent view of algorithms and functions, we propose to approximate the oracle algorithm by  a deep learning model, called 
Revised: Since then, designing neural algorithms by unrolling ISTA has become an active research topic. However, existing works mostly focus on the compressed sensing problem with a ﬁxed design matrix only. To enable for more general applicability, we design the architecture of PLISA by unrolling a classic path-following algorithm called APF (Wang et al., 2014) instead of ISTA.

Original:  PLISA θ .
Revised: APF is applicable to nonconvex losses and nonconvex penalty functions, covering a considerably larger range of objectives than LASSO. Designing the architecture based on APF allows PLISA to be applicable to a broader class of problems such as nonlinear sparse regression, graphical model estimation, etc. Furthermore, employing nonconvexity can potentially lead to better statistical properties (Fan & Li, 2001; Fan et al., 2009; Loh & Wainwright, 2015), for which we will explain more in Section 3.

Original: In this section, we ﬁrst describe the architecture of  PLISA θ and explain its design principles.
Revised: In the following, we will introduce APF and the architecture of our proposed PLISA .

Original: After that, we  describe how to optimize the parameters in PLISAθ under the learning-to-learn setting.
Revised: After that, we will describe how to optimize the parameters in PLISA under the learning-to-learn setting.

Original: 2. A RCHITECTURE OF PLISA θ
Revised: 2. A B RIEF I NTRODUCTION TO APF

Original: A naive architecture could be a recurrent neural network that takes Z 1: n as the input and outputs a d -dimensional vector.
Revised: We brieﬂy introduce the classic algorithm APF (Wang et al., 2014), and its details are presented in Algorithm 3 in Appendix I.

Original: However, a favorable design choice in the literature of algorithm learning is algorithm unrolling , which unrolls and truncates a classic iterative algorithm to design the architecture. Based on this idea, we unroll a classic path-following algorithm proposed by Wang et al. (2014b) to design PLISA θ .
Revised: The key idea of path-following algorithms is creating a sequence of T many sub-objectives to gradually approach the target objective that is supposed to be more difﬁcult to solve. More speciﬁcally, APF approximates the local minimizers of a sequence of sub-objectives:

Original: This classic algorithm is presented in Algorithm 2 in Appendix H.
Revised: [Equation] where λ 1 > λ 2 > · · · > λ T is a decreasing sequence of regularization parameters.

Original: The key idea of path-following algorithms iscreating a sequence of objectives to gradually approach the target objective that is supposed to be more difﬁcult to solve.
Revised: The last parameter λ T is the target regularization parameter. As a result, APF contains T blocks, and each block contains an iterative algorithm that minimizes one sub-objective in Eq.

Original: Following this idea, PLISA θ is designed to approximate the local minimizers of a sequence of objectives for t =  1 , · · · , T :
Revised: The output of the ( t − 1) -th block, denoted by β t − 1 , is used as the initialization of the t -th block, i.e., (cid:101) β 0 t = β t − 1 . Then the t -th block minimizes the t -th sub-objective by the modiﬁed proximal gradient algorithm: for k = 1 , · · · , K ,

Original: [Equation]
Revised: [Equation]

Revised: + ∇ β Q ( λ t , (cid:101) β k − 1 t ) (cid:1)(cid:17) .

Revised: (3) output of t -th block: β t = (cid:101) β Kt .

Revised: The notation T δ ( β ) :

Revised: = sign( β ) max {| β | − δ, 0 } is the soft-thresholding function, and the function

Revised: Q is the concave component of P deﬁned as Q ( λ, β ) :

Revised: = P ( λ, β ) − λ (cid:107) β (cid:107) 1 .

Revised: The number of steps K in each block is determined by certain stopping criteria.

Revised: It can be seem that the update steps in each block is similar to the ISTA algorithm, but it is modiﬁed in order to incorporate nonconvexity.

Revised: 2. A RCHITECTURE OF PLISA

Revised: [Figure]

Revised: The architecture of PLISA is designed by unrolling the APF algorithm, and augmenting some learnable parameters θ .

Revised: Therefore, the architecture of PLISA θ contains T blocks and each block contains K layers deﬁned by the K -step algorithm in Eq.

Revised: 3 (See Figure 2).

Revised: Note that in PLISA θ , both K and T are pre-deﬁned.

Revised: The architecture of

Revised: PLISA θ is different from APF as summarized below:

Revised: Element-wise and learnable regularization parameters.

Revised: Most classic algorithms including APF employ a uniform regularization parameter λ t across all entries of β , but PLISA θ uses a ddimensional vector λ t =

Revised: [ λ t, 1 , · · · , λ t,d ] (cid:62) to enforce different levels of sparsity to different entries in β .

Revised: Furthermore, the regularization parameters in PLISA θ are learnable, which will be optimized during the training.

Revised: Learnable penalty function: Classic algorithms use a pre-deﬁned sparse penalty function P , but PLISA θ parameterizes it as a combination of q different penalty functions and learns the weights of each penalty.

Revised: In other words, PLISA θ can learn to select a speciﬁc combination of the penalty functions from training data.

Revised: Learnable step size: The step sizes in APF are selected by line-search but they are learnable in PLISA θ .

Revised: Experimentally, we ﬁnd the learned step sizes lead to a much faster algorithm.

Revised: In later sections of this paper, we will show how such differences can make PLISA θ perform better than APF both empirically and theoretically.

Revised: Algorithm 1 and 2 present the mathematical details of the architecture, follow which we explain some notations and deﬁnitions.

Revised: Red-colored symbols indicate learnable parameters in PLISA θ .

Original: #cells: T , #layers per cell: K
Revised: #blocks: T , #layers per block: K

Original: Parameters: θ = { η , λ ∗ , w , α }
Revised: Parameters: θ = { η , λ ∗ , w , α }

Original: Input: samples Z 1: n β 0 ← 0 , λ 0 ← ∇ β L n ( Z 1: n , 0 )
Revised: Input: samples Z 1: n β 0 ← 0 , λ 0 ← ∇ β L n ( Z 1: n , 0 )

Original: For t = 1 , . . .
Revised: For t = 1 , . . .

Original: , T do λ t ← max { σ ( η ) ◦ λ t − 1 , λ ∗ } β t ← Cell K w ,α ( Z 1: n , β t − 1 , λ t ) by K
Revised: , T do λ t ← max { σ ( η ) ◦ λ t − 1 , λ ∗ } β t ← Block K w ,α ( Z 1: n , β t − 1 , λ t ) return β T

Original: many update steps in Eq.return β T with regularization parameters λ 1 , · · · , λ T and empirical loss L n which measures the “ﬁt” between a parameter β and observations Z 1: n. As an example, L n can be the least squared loss in linear regression problems. The second term in the objective, P w ,is a sparsity penalty function, which is set to be λ (cid:107) β (cid:107) 1 [-in many classic algorithms. Consequently, the architecture of PLISA θ contains T recurrent cells (Fig. 
-]
Revised: Input: Z 1: n , β t − 1 [-in many classic algorithms. Consequently, the architecture of PLISA θ contains T recurrent cells (Fig. 
-], λ t (cid:101) β 0 t ← β t −

Original: and Algorithm 1). Each cell contains a K -step algorithm that minimizes one objective in Eq. 1.
Revised: For k = 1 , . . . , K do g kt ← ∇ β L n ( Z 1: n , (cid:101) β k − 1 t )

Original: PLISA θ is different from the classic path-following algorithm in Wang et al. (2014a) in the sense that it contains learnable parameters θ = { η , λ ∗ , w , α } so that the regularization parameters, the sparsity penalty function, and the step size are learned from data. Furthermore, classic algorithms employ a uniform regularization parameter λ across all entries of β , but PLISA θ uses a d -dimensional vector λ t =
Revised: + ∇ β Q w ( λ t , (cid:101) β

Original: [ λ t, 1 , · · ·, λ t,d ] (cid:62) to enforce different levels of sparsity to different entries. Moreover, the classic algorithm determines the step sizes by line-search.
Revised: k − 1 t ) (cid:101) β kt ← T α · λ t (cid:16) (cid:101) β k − 1 t − α · g kt (cid:17) return β t = (cid:101) β

Original: Experimentally, we ﬁnd it much slower than PLISA θ that contains learned step sizes. We explain the details in the following. Red-colored symbols are used to indicate learnable parameters in PLISA θ . The concrete computations in PLISA θ are presented in Algorithm 1.
Revised: Kt

Original: (i) Entry-wise regularzation parameters.
Revised: Regularzation parameters.

Original: Given input samples Z 1: n , the  regularization parameters are initialized by a vector λ 0 := ∇ β L n ( Z 1: n , 0 ) . Then PLISA θ will update it sequentially by
Revised: In PLISA θ , the element-wise regularization parameters are initialized by a vector λ 0 := ∇ β L n ( Z 1: n , 0 ) , and then updated sequentially by

Original: [Equation]
Revised: [Equation]

Original:  where σ ( · ) , ◦ , and max {· , ·} are element-wise sigmoid function, multiplication, and maximization, respectively.
Revised: [Equation] where σ ( · ) , ◦ , and max {· , ·} are element-wise sigmoid function, multiplication, and maximization.

Original: { η , λ ∗ } are  learnable parameters.
Revised: { η , λ ∗ } are both d -dimensional learnable parameters.

Original: Eq.
Revised: Eq.

Original: 2 will create adecreasing sequence λ 1 , · · · , λ T through the decrease ratio σ ( η ) , until they reach theentry values of the learnable target vector λ ∗ .
Revised: 5 creates a sequence λ 1 , · · · , λ T through the decrease ratio σ ( η ) , until they reach the target regularization parameters λ ∗ .

Original: (ii) Penalty function.
Revised: Penalty function.

Original: PLISA θ parameterizes the penalty function as follows,
Revised: PLISA θ parameterizes the penalty function as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:102)
Revised: where (cid:102)

Original: [Equation]
Revised: [Equation]

Original:  In other words, P w is a learnable convex combination of q penalty functions ( P (1) , · · · , P ( q ) ) .
Revised: exp [Equation] In other words, P w is a learnable convex combination of q penalty functions ( P (1) , · · · , P ( q ) ) .

Original: The weights of these functions are determined by learnable parameters w =
Revised: The weights of these functions are determined by learnable parameters w =

Original: [ w 1 , · · · , w q ] .
Revised: [ w 1 , · · · , w q ] .

Original: In this paper, we focus on learning the combination of three well-known penalty functions:
Revised: In this paper, we focus on learning the combination of three well-known penalty functions:

Original: [Equation]
Revised: [Equation]

Original: ) , where P (1) is convex, and MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given in Appendix B. In fact, one can include any other penalty functions as long as they satisfy a set of conditions speciﬁed in Appendix B. (iii) Cell K w ,α and the step size. Each cell in PLISA θ contains a K -step
Revised: where P (1) is convex, and MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given in Appendix B. One can include any other penalty functions as long as they satisfy a set of conditions speciﬁed in Appendix B. Q w ( λ , β )

Original: algorithm that iteratively minimizes the cell-speciﬁc objective in Eq.
Revised: =

Original: The t -th cell Cell K w ,α initialize (cid:101) β 0 t by the output of last cell β t − and
Revised: P w ( λ , β ) −(cid:107) λ ◦ β (cid:107)represents the concave component of P w .

Original: computes the following

Original: modiﬁed proximal

Original: gradient steps: for k = 1 , · · · , K ,

Original: [Equation]

Original: cell output:

Original: [Equation]

Original: [ T δ ( β )]

Original: j := sign( β j ) max {| β j | − δ j , 0 } is the entry-wise soft-thresholding function.

Original: The function

Original: Q w ( λ , β ) :

Original: = P w ( λ , β ) − (cid:107) λ ◦ β (cid:107) 1 represents the concave component of P w .

Original: The analytical form of ∇ β Q w ( λ t , β ) can be found in Appendix B.
Revised: The analytical form of ∇ β Q w ( λ t , β ) are in Appendix B.

Original: The step size α is a learnable parameter in θ .

Original: Discussion.

Original: We choose the path-following algorithm in (Wang et al., 2014b) as the basis to design PLISA θ because this algorithm is applicable to nonconvex losses L n and nonconvex penalty functions P w .

Original: This allows PLISA θ to be applicable to a broader class of problems (e.g., nonlinear sparse regression problems may involve nonconvex loss).

Original: Furthermore, employing nonconvex L n and P w can potentially lead to better statistical properties (Fan & Li, 2001; Fan et al., 2009; Loh &

Original: Wainwright, 2015), for which we will explain more in Section 3 and in experiments.

Original: 2. L EARNING - TO - LEARN S ETTING
Revised: L EARNING - TO - LEARN S ETTING

Original: Now we describe how to train the parameters θ in PLISA θ under the learning-to-learn setting.
Revised: Now we describe how to train the parameters θ in PLISA θ under the learning-to-learn setting.

Original: Training set.
Revised: Training set.

Original: Similar to other works in this domain, we assume the access to m problems from the target problem-space P , and use them as the training set:
Revised: Similar to other works in this domain, we assume the access to m problems from the target problem-space P , and use them as the training set:

Original: [Equation]
Revised: [Equation]

Original: Here each estimation problem is represented by a pair of observations and the corresponding true parameter to be recovered.
Revised: Here each estimation problem is represented by a pair of observations and the corresponding true parameter to be recovered.

Original: A different problem i can contain a different number n i of observations.
Revised: A different problem i can contain a different number n i of observations.

Original: Training loss.
Revised: Training loss.

Original: Since the intermediate outputs β t ( Z 1: n ; θ ) of PLISA θ are also estimates of β ∗ , a common design of the training loss is the weighted sum of the intermediate estimation errors (Chen et al., 2021).
Revised: Since the intermediate outputs β t ( Z 1: n ; θ ) of PLISA θ are also estimates of β ∗ , a common design of the training loss is the weighted sum of the intermediate estimation errors (Chen et al., 2021).

Original: More speciﬁcally, we employ the following training loss:
Revised: More speciﬁcally, we employ the following training loss:

Original: [Equation]
Revised: [Equation]

Original: where γ < 1 is a discounting factor If γ = 0 then the loss is  estimated 
Revised: where γ < 1 is a discounting factor If γ = 0 then the loss is only estimated at the last layer.

Original: Generalization error.
Revised: Generalization error.

Original: The ultimate goal of algorithm learning is to minimize the estimation error on expectation over all problems in the target problem distribution:
Revised: The ultimate goal of algorithm learning is to minimize the estimation error on expectation over all problems in the target problem distribution:

Original: [Equation]
Revised: [Equation]

Original: where P ( P ) is a distribution in the target problem-space P .
Revised: where P ( P ) is a distribution in the target problem-space P .

Original: Let θ ∗ ∈ arg min
Revised: Let θ ∗ ∈ arg min

Original: L γ =0 train ( D m ; θ ) be a minimizer of the training loss.
Revised: L γ =0 train ( D m ; θ ) be a minimizer of the training loss.

Original: It is well-known that the generalization error can be bounded by:
Revised: It is well-known that the generalization error can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: We will theoretically characterize these two terms in Theorem 3.1 and Theorem 4.1.
Revised: We will theoretically characterize these two terms in Theorem 3.1 and Theorem 4.1.

Original:  C APACITY OF PLISA
Revised:  C APACITY OF PLISA

Original: Can PLISA θ achieve a small training error without using too many layers (cells)?
Revised: Can PLISA θ achieve a small training error without using too many layers?

Original: How can designs of element-wise regularization and learnable penalty functions help PLISA θ to achieve a smaller training error compared to classic algorithms?
Revised: How can designs of element-wise regularization and learnable penalty functions help PLISA θ to achieve a smaller training error compared to classic algorithms?

Original: We answer this question theorectically in this section.
Revised: We answer this question theorectically in this section.

Original: 3. P ROBLEM S PACE A SSUMPTIONS

Original: Before stating the theorem, we follow the notations in Wang et al.

Original: (2014b)

Original: ; Loh & Wainwright (2015) to describe some classic assumptions on the estimation problems.

Original: Assumption 3.1 (Problem Space) .

Original: Let s ∗ , ˜ s be positive integers and ρ − , ρ + be positive constants such that ˜ s >

Original: (121( ρ + /ρ − )

Original: + 144( ρ + /ρ − ) 2 ) s ∗ .

Original: Assume for every estimation problem ( Z 1: n , β ∗ ) in the space P , the following conditions are satisﬁed.

Original: [Equation]

Original: ) (cid:107) β ∗ (cid:107) 0 ≤ s ∗ and (cid:107) β ∗ (cid:107) ∞

Original: ≤ B 1 ; (b) For any nonzero v ∈ R d with sparsity (cid:107) v (cid:107) 0 ≤ s ∗ + 2˜ s , it holds v (cid:62) ∇ 2 β L n ( Z 1: n , β ) v (cid:107) v (cid:107)∈ [ ρ − , ρ + ] ; (c) 8 | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ≤ | [ ∇ β L n ( Z 1: n , 0 )]

Original: j | ≤ B 2 , ∀ j = 1 , · · · , d .

Original: Condition (a) assumes β ∗ is s ∗ -sparse and B 1 -bounded.

Original: Condition (b) is commonly referred to as ‘sparse eigenvalue condition’ (Zhang, 2010b; Wang et al., 2014b), which is weaker than the wellknown restricted isometry property (RIP) in compressed sensing (Candes & Tao, 2005).

Original: Note that the class of functions satisfying conditions of this type is much larger than the class of convex losses.

Original: In the special case when L n ( Z 1: n , β ) is strongly convex in β , condition (b) holds with ˜ s → ∞ .

Original: The last condition bounds the gradient of the empirical loss L n at the true parameter β ∗ and 0 .

Original:  F IRST M AIN R ESULT : C APACITY
Revised: 3. F IRST M AIN R ESULT : C APACITY

Original: Let β t ( Z 1: n ; θ ) be the output of the t -th cell in PLISA θ .
Revised: Let β t ( Z 1: n ; θ ) be the output of the t -th block in PLISA θ .

Original: Let x ∨ a denote entry-wise maximal value max { x , a } .
Revised: Let x ∨ a denote entry-wise maximal value max { x , a } .

Original: Let ( x ) S denote the sub-vector of x with entries indexed by the set S .
Revised: Let ( x ) S denote the sub-vector of x with entries indexed by the set S .

Original: Theorem 3.1 (Capacity) .
Revised: Theorem 3.1 (Capacity) .

Original: Assume the problem space P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume the problem space P satisﬁes Assumption C.1 and D m ⊆ P .

Original: Let
Revised: Let

Original: T be the number of cells in PLISA θ and let K be the number of layers in each cell.
Revised: T be the number of blocks in PLISA θ and let K be the number of layers in each block.

Original: For any ε > 0 , there exists a set of parameters θ = { η , λ ∗ , w , α } such that the estimation error of every problem ( Z 1: n , β ∗ )
Revised: For any ε > 0 , there exists a set of parameters θ = { η , λ ∗ , w , α } such that the estimation error of every problem ( Z 1: n , β ∗ )

Original: ∈
Revised: ∈

Original: D m is bounded as follows, ∀ T > t 0 ,
Revised: D m is bounded as follows, ∀ T > t 0 ,

Original: [Equation]
Revised: [Equation]

Original: optimization error
Revised: optimization error

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: statistical error (10) where S ∗ :=
Revised: statistical error ( [Equation] where S ∗ :

Original:  supp( β ∗ ) is the support indices of β ∗ , c θ , c (cid:48) θ , and C θ are some positive values depending on the chosen θ , and κ m is a condition number which reveals the similarity of the problems in D m .
Revised: = supp( β ∗ ) is the support indices of β ∗ , c θ , c (cid:48) θ , and C θ are some positive values depending on the chosen θ , and κ m is a condition number which reveals the similarity of the problems in D m .

Original: Note that K and t 0 are required to be larger than certain values, but we will elaborate in
Revised: Note that K and t 0 are required to be larger than certain values, but we will elaborate in

Original: Appendix D that the required lower bounds are small.
Revised: Appendix E that the required lower bounds are small.

Original: See Appendix D for the proof of this theorem.
Revised: See Appendix E for the proof of this theorem.

Original: This estimation error can be interpreted as a combination of the optimization error (in Eq. 9) and the statistical error (in Eq. 10).
Revised: This estimation error can be interpreted as a combination of the optimization error (in Eq. 10) and the statistical error (in Eq. 11).

Original: The optimization error decreases linearly in both K and T .
Revised: The optimization error decreases linearly in both K and T .

Original: The statistical erroroccurs because of the randomness in Z 1: n .
Revised: The statistical erroroccurs because of the randomness in Z 1: n .

Original: The gradient at the true parameter ∇ β L n ( Z 1: n , β ∗ ) characterizes how well the ﬁnite samples Z 1: n can represent the distribution P β ∗ .
Revised: The gradient at the true parameter ∇ β L n ( Z 1: n , β ∗ ) characterizes how well the ﬁnite samples Z 1: n can represent the distribution P β ∗ .

Original: A direct consequence of Theorem 3.1 is that the training error can be small without using too many layers and cells in PLISA θ .
Revised: A direct consequence of Theorem 3.1 is that the training error can be small without using too many layers and blocks in PLISA θ .

Original: Furthermore, based on Theorem 3.1, we will  elaborate on how the entry-wise regularization and learnable penalty function can effectively reduce thestatistical error and therefore the training error in the following.
Revised: We will also elaborate on how the entry-wise regularization and learnable penalty function can effectively reduce the training error in the following.

Original: (i Impact of entry wise regularization.
Revised: (i) Impact of entry-wise regularization.

Original: Restricting the regularization to be uniform across entries will lead to an error bound that replaces the statistical error (cid:107) ( ∇ β L n ( Z 1: n , β ∗ )
Revised: Restricting the regularization to be uniform across entries will lead to an error bound that replaces the statistical error (cid:107) ( ∇ β L n ( Z 1: n , β ∗ )

Original: ∨ ε ) S ∗ (cid:107) 2 in Eq.by √ s ∗ ( (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞ ∨ ε ) .
Revised: ∨ ε ) S ∗ (cid:107) 2 in Eq.by √ s ∗ ( (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞ ∨ ε ) .

Original: To understand how the former has improved the latter, we can consider the sparse linear regression problem.
Revised: To understand how the former has improved the latter, we can consider the sparse linear regression problem.

Original: If the design matrix is normalized such that max 1 ≤ j ≤ d (cid:107) ([ x 1 ] j , · · · , [ x n ] j ) (cid:107) 2 ≤ √ n , then (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ (cid:107) 2 ≤ C (cid:112) s ∗ /n with high probability.
Revised: If the design matrix is normalized such that max 1 ≤ j ≤ d (cid:107) ([ x 1 ] j , · · · , [ x n ] j ) (cid:107) 2 ≤ √ n , then (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ (cid:107) 2 ≤ C (cid:112) s ∗ /n with high probability.

Original: In comparison, √ s ∗ (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞
Revised: In comparison, √ s ∗ (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞

Original: ≤ C (cid:112) s ∗ log d/n with high probability is a slower statistical rate due to the term log d . (ii) Impact of learnable penalty function.
Revised: ≤ C (cid:112) s ∗ log d/n with high probability is a slower statistical rate due to the term log d . (ii) Impact of learnable penalty function.

Original: To explain the the beneﬁt of using learnable penalty function, we give a more reﬁned bound for the statistical error in Eq.
Revised: To explain the the beneﬁt of using learnable penalty function, we give a more reﬁned bound for the statistical error in Eq.

Original: 10 in the following lemma.
Revised: 11 in the following lemma.

Original: Lemma 3.1 (Reﬁned bound) .
Revised: Lemma 3.1 (Reﬁned bound) .

Original: Assume the same conditions and parameters θ in Theorem 3.1.
Revised: Assume the same conditions and parameters θ in Theorem 3.1.

Original: Assume T → ∞ so that the optimization error can be ignored.
Revised: Assume T → ∞ so that the optimization error can be ignored.

Original: For simplicity, assume (cid:102) w 3 = 0 and only consider the weights (cid:102)w 1 and (cid:102)
Revised: For simplicity, assume (cid:102) w 3 = 0 and only consider the weights (cid:102)

Original: w  2 for (cid:96) 1 penalty and MCP.
Revised: w 1 and (cid:102) w 2 for (cid:96) 1 penalty and MCP.

Original: Then for every problem ( Z 1: n , β ∗ ) ∈
Revised: Then for every problem ( Z 1: n , β ∗ ) ∈

Original: D m : 
Revised: D m : (cid:107) β ∞

Original: [Equation]
Revised: [Equation]

Original: β
Revised: Z 1: n ; [Equation]

Original: [Equation]
Revised: [Equation]

Original: where b > 1 is a hyperparameter in MCP , and the index sets S ∗ 1 and S ∗ 2 are deﬁned as S ∗ 1 :=
Revised: where b > 1 is a hyperparameter in MCP , and the index sets S ∗ 1 and S ∗ 2 are deﬁned as S ∗ 1 :=

Original: { j ∈
Revised: { j ∈

Original: S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) ≤ bλ ∗ j } and S ∗ 2 :=
Revised: S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) ≤ bλ ∗ j } and S ∗ 2 :=

Original: { j ∈ S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) > bλ ∗ j } .
Revised: { j ∈ S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) > bλ ∗ j } .

Original: See Appendix D for the proof.
Revised: See Appendix E for the proof.

Original: This reﬁned bound reveals the beneﬁt of learning the penalty function because:
Revised: This reﬁned bound reveals the beneﬁt of learning the penalty function because:

Original:  is a more
Revised: According to Lemma 3.1, the optimal penalty function is problem-dependent . For example, if (8( bρ − + 1) κ m + 1) (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ 1 (cid:107) 2 > (8( bρ − − 1)

Original: of
Revised: κ m − 1) (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ 2 (cid:107) 2 , choosing (cid:102) w 2 = 0 can induce a smaller error bound. Otherwise, (cid:102) w 2 = 1 is better.

Revised: Therefore, learning is a more suitable way of choosing the penalty function.

Original: The convergence speed C θ in Eq.
Revised: The convergence speed C θ in Eq.

Original: 9 isalso
Revised: 10 is

Original:  affected by the weights, monotonely decreasing in (cid:102) w 2 .
Revised: also affected by the weights , monotonely decreasing in (cid:102) w 2 .

Original: Through gradient-based training, we can automatically ﬁnd the optimal combination of penalty functions to strike a nice balance between the statistical error and convergence speed.
Revised: Through gradient-based training, we can automatically ﬁnd the optimal combination of penalty functions to strike a nice balance between the statistical error and convergence speed.

Original:  G ENERALIZATION A NALYSIS
Revised:  G ENERALIZATION A NALYSIS

Original: How well can the learned PLISA θ solve new problems outside the training set?
Revised: How well can the learned PLISA θ solve new problems outside the training set?

Original: In this section, we conduct the generalization analysis in a novel way to focus on answering the questions:
Revised: In this section, we conduct the generalization analysis in a novel way to focus on answering the questions:

Original: How is the generalization bound of PLISA θ related to its algorithmic properties ?
Revised: How is the generalization bound of PLISA θ related to its algorithmic properties ?

Original: And how is it different from conventional neural networks?
Revised: And how is it different from conventional neural networks?

Original: 4. S ECOND M AIN R ESULT : G ENERALIZATION B OUND
Revised: 4. S ECOND M AIN R ESULT : G ENERALIZATION B OUND

Original: To analyze the generalization properties of neural networks, many works have adopted the analysis framework of Bartlett & Mendelson (2002) to bound the Rademacher complexity via Dudley’s integral (Bartlett et al., 2017;
Revised: To analyze the generalization properties of neural networks, many works have adopted the analysis framework of Bartlett & Mendelson (2002) to bound the Rademacher complexity via Dudley’s integral (Bartlett et al., 2017;

Original: Chen et al., 2019; Garg et al., 2020;
Revised: Chen et al., 2019; Garg et al., 2020;

Original: Joukovsky et al., 2021).
Revised: Joukovsky et al., 2021).

Original: A key step in this analysis framework is deriving the robustness of the training loss to the small perturbation in model parameters θ .
Revised: A key step in this analysis framework is deriving the robustness of the training loss to the small perturbation in model parameters θ .

Original: Since we can view PLISA θ as an iterative algorithm, we borrow the analysis tools of classic optimization algorithms to derive its robustness in θ .
Revised: Since we can view PLISA θ as an iterative algorithm, we borrow the analysis tools of classic optimization algorithms to derive its robustness in θ .

Original: The following lemma states this key intermediate result, which connects the Lipschitz constant to algorithmic properties of PLISA θ .
Revised: The following lemma states this key intermediate result, which connects the Lipschitz constant to algorithmic properties of PLISA θ .

Original: Lemma 4.1 (Robustness to θ ) .
Revised: Lemma 4.1 (Robustness to θ ) .

Original: Assume P satisﬁes Assumption 3.1 and D m ∼ P( P ) m .
Revised: Assume P satisﬁes Assumption C.1 and D m ∼ P P ) m .

Original: Assume
Revised: Assume

Original: PLISA θ contains T > t 0 cells and K layers.
Revised: PLISA θ contains T > t 0 blocks and K layers.

Original: Consider a parameter space Θ in which the parameters satisfy (i)
Revised: Consider a parameter space Θ in which the parameters satisfy (i)

Original: α ∈
Revised: α ∈

Original: [ α min , 1 ρ + ] , (ii) η j ∈ [ σ − 1 (0 . 9) , η max ] , (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , and (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]
Revised: [ α min , 1 ρ + ] , (ii) η j ∈ [ σ − 1 (0 . 9) , η max ] , (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , and (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ∨ ε, λ max ] with some positive constants α min , η max , ξ max , and λ max .
Revised: j | ∨ ε, λ max ] with some positive constants α min , η max , ξ max , and λ max .

Original: Then for any θ = { η , λ ∗ , w , α } and θ (cid:48)
Revised: Then for any θ = { η , λ ∗ , w , α } and θ (cid:48)

Original: = { η (cid:48) , λ ∗(cid:48) , w (cid:48) , α (cid:48) } in Θ , and for any recovery problem ( Z 1: n , β ∗ ) ∈
Revised: = { η (cid:48) , λ ∗(cid:48) , w (cid:48) , α (cid:48) } in Θ , and for any recovery problem ( Z 1: n , β ∗ ) ∈

Original: D m , the following inequality holds,
Revised: D m , the following inequality holds,

Original: [Equation]
Revised: [Equation]

Revised: stability

Revised: [Table]

Revised: [Equation]

Original: where c 1 , c 2 , c 3 , c 4 and C Θ are some positive constants.
Revised: where c 1 , c 2 , c 3 , c 4 and C Θ are some positive constants.

Original: Note that similar to Theorem 3.1, K and tare required to be larger than certain small values.
Revised: Note that similar to Theorem 3.1, K and tare required to be larger than certain small values.

Original: See Appendix E.1 for the proof.
Revised: See Appendix F.1 for the proof.

Original: Convergence rate & step size perturbation.
Revised: Convergence rate & step size perturbation.

Original: In Eq.
Revised: In Eq.

Original: 13, the Lipschitz constant in the step size α scales at the same rate as the convergence rate of PLISA θ , decreasing exponentially in T and K (See Fig.
Revised: 14, the Lipschitz constant in the step size α scales at the same rate as the convergence rate of PLISA θ , decreasing exponentially in T and K (See Fig.

Original: 3 for a visualization).
Revised: 3 for a visualization).

Original: To understand this, consider when both step sizes α and α (cid:48) are within the convergence region (i.e., (0 , ρ − 1+ ] ).
Revised: To understand this, consider when both step sizes α and α (cid:48) are within the convergence region (i.e., (0 , ρ − 1+ ] ).

Original: After inﬁnitely many steps, their induced outputs will both converge to the same optimal point.
Revised: After inﬁnitely many steps, their induced outputs will both converge to the same optimal point.

Original: This intuitively explains why the output perturbation caused by α -perturbation has the same decrease rate as the optimization error. 
Revised: This intuitively explains why the output perturbation caused by α -perturbation has the same decrease rate as the optimization error. [Table] [Figure]

Original: Stability rate & regularization perturbation.
Revised: Stability rate & regularization perturbation.

Original: In the literature of optimization, stability of an algorithm expresses its robustness to small perturbation in the optimization objective.
Revised: In the literature of optimization, stability of an algorithm expresses its robustness to small perturbation in the optimization objective.

Original: This is clearly related to the robustness of PLISA θ to the perturbation in η , λ ∗ , w , because these parameters jointly determine the regularization P w ( λ t , β ) , which is a part of the optimization objective in Eq.
Revised: This is clearly related to the robustness of PLISA θ to the perturbation in η , λ ∗ , w , because these parameters jointly determine the regularization P w ( λ t , β ) , which is a part of the optimization objective.

Original: Therefore, we exploit the analysis techniques for algorithmic stability to derive the robustness in ( η , λ ∗ , w ) -perturbation and obtain the Lipschitz constant in Eq. 14, which is bounded but increasing in T and K (See Fig. 3 for a visualization).
Revised: Therefore, we exploit the analysis techniques for algorithmic stability to derive the robustness in ( η , λ ∗ , w ) -perturbation and obtain the Lipschitz constant in Eq. 15, which is bounded but increasing in T and K (See Fig.for a visualization).

Original: Based on the key result in Lemma 4.1, we can apply Dudley’s integral to measure the empirical
Revised: Based on the key result in Lemma 4.1, we can apply Dudley’s integral to measure the empirical

Original: Rademachar complexity which immediately yields the following generalization bound.
Revised: Rademachar complexity which immediately yields the following generalization bound.

Original: Theorem 4.1 (Generalization gap) .
Revised: Theorem 4.1 (Generalization gap) .

Original: Assume the assumptions in Lemma 4.1.
Revised: Assume the assumptions in Lemma 4.1.

Original: For any (cid:15) > 0 , with probability at least 1 − (cid:15) , the generalization gap is bounded by
Revised: For any (cid:15) > 0 , with probability at least 1 − (cid:15) , the generalization gap is bounded by

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 3 , C Θ are constants independent of d, m, K and T .
Revised: where c 1 , c 2 , c 3 , C Θ are constants independent of d, m, K and T .

Original: See Appendix E for the proof.
Revised: See Appendix F for the proof.

Original: KT

Original: Convergence Rate

Original: KT

Original: Stability

Original: KT

Original: Generalization Bound

Original: KT

Original: Convergence Rate

Original: KT

Original: Stability

Original: KT

Original: Generalization Bound

Original: Fig.
Revised: Fig.

Original: 3 visualizes how the generalization bound in Theorem 4.1 grows when KT increases.
Revised: 3 visualizes how the generalization bound in Theorem 4.1 grows when KT increases.

Original: The two sets of plots look slightly different by picking different constants C Θ .
Revised: The two sets of plots look slightly different by picking different constants C Θ .

Original: We have also tried varying the values of c 2 , c 3 , d, m in Theorem 4.1.
Revised: We have also tried varying the values of c 2 , c 3 , d, m in Theorem 4.1.

Original: Overall, they lead to the two types of behaviors in Fig.
Revised: Overall, they lead to the two types of behaviors in Fig.

Original: An important observation in Theorem 4.1 and Figure 3 is that the generalization gap could decrease in the number of layers , and we will see in Section 7 that this matches the empirical observations.
Revised: An important observation in Theorem 4.1 and Figure 3 is that the generalization gap could decrease in the number of layers , and we will see in Section 7 that this matches the empirical observations.

Original: It also distinguishes algorithm-unrolling based architectures from conventional neural networks, whose generalization gaps rarely decrease in the number of layers.
Revised: It also distinguishes algorithm-unrolling based architectures from conventional neural networks, whose generalization gaps rarely decrease in the number of layers.

Original: Remark.
Revised: Remark.

Original: The above generalization results are conducted on a constrained parameter space (as described in Lemma 4.1) so that we can utilize the algorithmic properties of PLISA θ .
Revised: The above generalization results are conducted on a constrained parameter space (as described in Lemma 4.1) so that we can utilize the algorithmic properties of PLISA θ .

Original: We focus on this space because the analysis contains more interesting and new ingredients.
Revised: We focus on this space because the analysis contains more interesting and new ingredients.

Original: For parameters outside this space, the analysis procedure is similar to other conventional recurrent networks.
Revised: For parameters outside this space, the analysis procedure is similar to other conventional recurrent networks.

Original: Since the bound in Theorem 4.1 has matched the empirical observations, it is reasonable to believe that after training, the learned parameters are likely to be in this ‘nice’ constrained space.
Revised: Since the bound in Theorem 4.1 has matched the empirical observations, it is reasonable to believe that after training, the learned parameters are likely to be in this ‘nice’ constrained space.

Original:  E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING
Revised:  E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING

Original: Real-world datasets may not contain the ground-truth parameters β ∗ , but only contain the samples from each task, D Um = { Z (1)1: n , · · · , Z ( m ) 1: n } .
Revised: Real-world datasets may not contain the ground-truth parameters β ∗ , but only contain the samples from each task, D Um = { Z (1)1: n , · · · , Z ( m ) 1: n } .

Original: In this setting, we can construct an unsupervised training loss to minimize the empirical loss function L n (e.g., the likelihood function) on the samples.Unsupervised training loss:
Revised: In this setting, we can construct an unsupervised training loss to minimize the empirical loss function L n (e.g., the likelihood function) on the samples.

Original: [Equation]
Revised: [Equation]

Original: In this loss, both Z ( i ) 1: n 1 and Z ( i ) 1: n 2 are subsets of Z ( i ) 1: n .
Revised: In this loss, both Z ( i ) 1: n 1 and Z ( i ) 1: n 2 are subsets of Z ( i ) 1: n .

Original: The samples Z ( i ) 1: n 1 are used as the input to
Revised: The samples Z ( i ) 1: n 1 are used as the input to

Original: PLISA θ
Revised: PLISA θ

Original: and the samples Z ( i ) 1: n 2 are used for evaluating the output β T from PLISA θ .
Revised: and the samples Z ( i ) 1: n 2 are used for evaluating the output β T from PLISA θ .

Original: Let θ ∗
Revised: Let θ ∗

Original: U ∈ arg min
Revised: U ∈ arg min

Original: L Utrain ( D Um ; θ ) be a minimizer to this unsupervised loss.
Revised: L Utrain ( D Um ; θ ) be a minimizer to this unsupervised loss.

Original: Theoretically , to bound the generalization error of θ ∗ U , we can show that
Revised: Theoretically , to bound the generalization error of θ ∗ U , we can show that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Compared to Eq.
Revised: Compared to Eq.

Original: 8, this upper bound contains an additional statistical error, which appears because of the gap between the unsupervised loss and the true error that we aim to optimize.
Revised: 9, this upper bound contains an additional statistical error, which appears because of the gap between the unsupervised loss and the true error that we aim to optimize.

Original: Clearly, in this upper bound, the generalization gap can be bounded by Theorem 4.1.
Revised: Clearly, in this upper bound, the generalization gap can be bounded by Theorem 4.1.

Original: Furthermore, the unsupervised training error in Eq.
Revised: Furthermore, the unsupervised training error in Eq.

Original: 17 can be bounded by combining Theorem 3.1 with the following in equality.
Revised: 18 can be bounded by combining Theorem 3.1 with the following in equality.

Original: [Equation]
Revised: [Equation]

Original: More details of the extension to unsupervised setting can be found in Appendix G.
Revised: More details of the extension to unsupervised setting can be found in Appendix H.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning-to-learn has become an active research direction in recent years (Bora et al., 2017;
Revised: Learning-to-learn has become an active research direction in recent years (Bora et al., 2017;

Original: Franceschi et al., 2017; Niculae et al., 2018; Denevi et al., 2018; Poganˇci´c et al., 2019; Liu et al., 2019b; Berthet et al., 2020).
Revised: Franceschi et al., 2017; Niculae et al., 2018; Denevi et al., 2018; Poganˇci´c et al., 2019; Liu et al., 2019b; Berthet et al., 2020).

Original: Many works share the idea of unrolling or differentiating through algorithms to design the architecture (Yang et al., 2017; Borgerding et al., 2017; Corbineau et al., 2019; Xie et al., 2019;
Revised: Many works share the idea of unrolling or differentiating through algorithms to design the architecture (Yang et al., 2017; Borgerding et al., 2017; Corbineau et al., 2019; Xie et al., 2019;

Original: Shrivastava et al., 2020; Chen et al., 2020a; Wei et al., 2020; Indyk et al., 2019; Grover et al., 2019; Wu et al., 2019).
Revised: Shrivastava et al., 2020; Chen et al., 2020a; Wei et al., 2020; Indyk et al., 2019; Grover et al., 2019; Wu et al., 2019).

Original: A well-known example of learning-based algorithm is
Revised: A well-known example of learning-based algorithm is

Original: LISTA (Gregor & LeCun, 2010) which interprets ISTA (Daubechies et al., 2004) as layers of neural networks and has been an active research topic (Zhang & Ghanem, 2018; Kamilov & Mansour, 2016; Chen et al., 2018; Liu et al., 2019a; Wu et al., 2020; Kim & Park, 2020).
Revised: LISTA (Gregor & LeCun, 2010) which interprets ISTA (Daubechies et al., 2004) as layers of neural networks and has been an active research topic (Zhang & Ghanem, 2018; Kamilov & Mansour, 2016; Chen et al., 2018; Liu et al., 2019a; Wu et al., 2020; Kim & Park, 2020).

Original: However, the generalization of algorithm learning has received less attention.
Revised: However, the generalization of algorithm learning has received less attention.

Original: The only exceptions are several works.
Revised: The only exceptions are several works.

Original: However, Chen et al.
Revised: However, Chen et al.

Original: (2020b)
Revised: (2020b)

Original: and Wang et al.
Revised: and Wang et al.

Original: (2021) only consider learning to optimize quadratic losses.
Revised: (2021) only consider learning to optimize quadratic losses.

Original: Behboodi et al.
Revised: Behboodi et al.

Original: Joukovsky et al.
Revised: Joukovsky et al.

Original: (2021) do not connect the generalization analysis with algorithmic properties to provide tighter bounds as in our work.
Revised: (2021) do not connect the generalization analysis with algorithmic properties to provide tighter bounds as in our work.

Original: Unlike our work that analyzes the Lipschitz continuity (Lemma 4.1), the work of Gupta & Roughgarden (2017); Balcan et al.
Revised: Unlike our work that analyzes the Lipschitz continuity (Lemma 4.1), the work of Gupta & Roughgarden (2017); Balcan et al.

Original: (2021) studied the generalization of learning-based algorithms with a focus on scenarios when the Lipschitz continuity is unavailable.
Revised: (2021) studied the generalization of learning-based algorithms with a focus on scenarios when the Lipschitz continuity is unavailable.

Original: We will refer the audience to Shlezinger et al.
Revised: We will refer the audience to Shlezinger et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: for a more comprehensive summary of related works.
Revised: for a more comprehensive summary of related works.

Original:  E XPERIMENTS 7. S YNTHETIC E XPERIMENTS
Revised:  E XPERIMENTS 7. S YNTHETIC E XPERIMENTS

Original: In synthetic datasets, we consider sparse linear regression problems and sparse precision matrix estimation problems for Gaussian graphical models.
Revised: In synthetic datasets, we consider sparse linear regression problems and sparse precision matrix estimation problems for Gaussian graphical models.

Original: Speciﬁcally, we recover target vectors β ∗ ∈ R d , where d = { 256 , 1024 } in SLR, and estimate precision matrices Θ ∗ ∈ R d × d , where d = { 50 , 100 } in SPE.
Revised: Speciﬁcally, we recover target vectors β ∗ ∈ R d , where d = { 256 , 1024 } in SLR, and estimate precision matrices Θ ∗ ∈ R d × d , where d = { 50 , 100 } in SPE.

Original: See Appendix H.2 for descriptions of the dataset and data preparation.
Revised: See Appendix I.2 for descriptions of the dataset and data preparation.

Original: 7.1. S PARSE L INEAR R EGRESSION (SLR)
Revised: 7.1. S PARSE L INEAR R EGRESSION (SLR)

Original: [Figure]
Revised: [Figure]

Original: In this experiment, we compare PLISA with several baselines and also verify the theorems.
Revised: In this experiment, we compare PLISA with several baselines and also verify the theorems.

Original: Performance comparison.
Revised: Performance comparison.

Original: We consider baselines including APF (Wang et al., 2014b), ALISTA (Liu et al., 2019a), RNN (Andrychowicz et al., 2016), and
Revised: We consider baselines including APF (Wang et al., 2014), ALISTA (Liu et al., 2019a), RNN (Andrychowicz et al., 2016), and

Original: RNN- (cid:96) 1 .
Revised: RNN- (cid:96) 1 .

Original: APF is the path-following algorithm that is used as the basis of our architecture.
Revised: APF is the path-following algorithm that is used as the basis of our architecture.

Original: ALISTA is a representative of algorithm unrolling based architectures, which is an advanced variant of LISTA.
Revised: ALISTA is a representative of algorithm unrolling based architectures, which is an advanced variant of LISTA.

Original: We have tried the vanilla LISTA, but it performs worse than ALISTA on our tasks so it is not reported.
Revised: We have tried the vanilla LISTA, but it performs worse than ALISTA on our tasks so it is not reported.

Original: RNN refers to the LSTM-based model in Andrychowicz et al.
Revised: RNN refers to the LSTM-based model in Andrychowicz et al.

Original: Besides, we add a soft-thresholding operator to this model to enforce sparsity, and include this variant as a baseline, called RNN- (cid:96) 1 .
Revised: Besides, we add a soft-thresholding operator to this model to enforce sparsity, and include this variant as a baseline, called RNN- (cid:96) 1 .

Original: Except for APF, all methods are trained on the same set of training problems and selected by the validation problems.
Revised: Except for APF, all methods are trained on the same set of training problems and selected by the validation problems.

Original: For APF, we perform grid-search to choose its hyperparameters, which is also selected by the validation set.
Revised: For APF, we perform grid-search to choose its hyperparameters, which is also selected by the validation set.

Original: The detailed speciﬁcation of each model can be found in Appendix H.3.
Revised: The detailed speciﬁcation of each model can be found in Appendix I.3.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 4 shows the convergence of (cid:107) β t − β ∗ (cid:107) 2 for problems in the test set.
Revised: 4 shows the convergence of (cid:107) β t − β ∗ (cid:107) 2 for problems in the test set.

Original: The x -axis indicates the wall-clock time.
Revised: The x -axis indicates the wall-clock time.

Original: In terms of the ﬁnal recovery accuracy, PLISA outperforms all baseline methods.
Revised: In terms of the ﬁnal recovery accuracy, PLISA outperforms all baseline methods.

Original: In the more difﬁcult setting (i [Figure] e, d = 1024 ), its advantage is obvious.
Revised: In the more difﬁcult setting (i.e, d = 1024 ), its advantage is obvious.

Original: Although
Revised: Although

Original: PLISA is slightly slower than other deep learning based models due to the computations of
Revised: PLISA is slightly slower than other deep learning based models due to the computations of

Original: MCP and SCAD , PLISA achieves a better accuracy and it has been converging much faster than the classic algorithm APF.
Revised: MCP and SCAD , PLISA achieves a better accuracy and it has been converging much faster than the classic algorithm APF.

Original: APF is very slow mainly due to the use of line-search for selecting step sizes.
Revised: APF is very slow mainly due to the use of line-search for selecting step sizes.

Original: Generalization gap.
Revised: Generalization gap.

Original: We are interested in the generalization behavior of PLISA .
Revised: We are interested in the generalization behavior of PLISA .

Original: As this experiment is conducted for theoretical interest, we do not use the validation set to select the model.
Revised: As this experiment is conducted for theoretical interest, we do not use the validation set to select the model.

Original: We vary the number of layers ( K ) and cells ( T ) in PLISA to create a set of models with different depths.
Revised: We vary the number of layers ( K ) and blocks ( T ) in PLISA to create a set of models with different depths.

Original: For each depth, we train the model with 2000 training problems, and then test it on a separate set of 100 problems to approximate the generalization gap.
Revised: For each depth, we train the model with 2000 training problems, and then test it on a separate set of 100 problems to approximate the generalization gap.

Original: In Fig.
Revised: In Fig.

Original: 5, we observe the interesting behavior of the generalization gap, where the left one increases in KT at the beginning and then decrease to a constant, and the right one increases fast and then decrease very slowly.
Revised: 5, we observe the interesting behavior of the generalization gap, where the left one increases in KT at the beginning and then decrease to a constant, and the right one increases fast and then decrease very slowly.

Original: This surprisingly matches the two different visualizations in Fig.
Revised: This surprisingly matches the two different visualizations in Fig.

Original: 3 of the predicted generalization gap given by Theorem 4.1.
Revised: 3 of the predicted generalization gap given by Theorem 4.1.

Original: 7.1. S PARSE P RECISION M ATRIX E STIMATION (SPE)
Revised: 7.1. S PARSE P RECISION M ATRIX E STIMATION (SPE)

Original: We compare PLISA with APF, GLASSO (Friedman et al., 2008), GISTA (Guillot et al., 2012), and
Revised: We compare PLISA with APF, GLASSO (Friedman et al., 2008), GISTA (Guillot et al., 2012) [Table]

Original:  GGM (Belilovsky et al., 2017) on sparse precision estimation tasks in Gaussian graphical models.
Revised: and GGM (Belilovsky et al., 2017) on sparse precision estimation tasks in Gaussian graphical models.

Original: GLASSO estimates the precision matrix by block-coordinate decent methods.
Revised: GLASSO estimates the precision matrix by block-coordinate decent methods.

Original: GISTA is a proximal gradient method for precision matrix estimation.
Revised: GISTA is a proximal gradient method for precision matrix estimation.

Original: GGM utilizes convolutional neural network to estimate the precision matrix.
Revised: GGM utilizes convolutional neural network to estimate the precision matrix.

Original: Details of each model and the training can be found in Appendix H.4.
Revised: Details of each model and the training can be found in Appendix I.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 1 reports the Frobenius error (cid:107)
Revised: Table 1 reports the Frobenius error (cid:107)

Original: Θ − Θ ∗ (cid:107) 2 F between the estimation Θ and the true precision matrix Θ ∗ , averaged over 100 test problems.
Revised: Θ − Θ ∗ (cid:107) 2 F between the estimation Θ and the true precision matrix Θ ∗ , averaged over 100 test problems.

Original: PLISA achieves consistent improvements.
Revised: PLISA achieves consistent improvements.

Original: Classic algorithm are slower because they perform linesearch.
Revised: Classic algorithm are slower because they perform linesearch.

Original: GLASSO is faster than other classic algorithm because we use the sklearn package (Pedregosa et al., 2011) in which the implementations are optimized.
Revised: GLASSO is faster than other classic algorithm because we use the sklearn package (Pedregosa et al., 2011) in which the implementations are optimized.

Original: 7.1. D ISCUSSION ON T RAINING -T ESTING T IME
Revised: 7.1. D ISCUSSION ON T RAINING -T ESTING T IME

Original: Test time.
Revised: Test time.

Original: Fig.
Revised: Fig.

Original: 4 and Table 1 shows the wall-clock time for solving test problems.
Revised: 4 and Table 1 shows the wall-clock time for solving test problems.

Original: Overall, classic algorithms are slower because they need to perform line-search.
Revised: Overall, classic algorithms are slower because they need to perform line-search.

Original: It is noteworthy that learning-based methods can solve a batch of problems parallelly but most classic algorithms cannot.
Revised: It is noteworthy that learning-based methods can solve a batch of problems parallelly but most classic algorithms cannot.

Original: To allow more advantages for classic algorithms, the test problems are solved without batching in all methods.
Revised: To allow more advantages for classic algorithms, the test problems are solved without batching in all methods.

Original: Train time.
Revised: Train time.

Original: As metioned earlier, wealso perform grid-search to select the hyperparameters in classic algorithms usingthe validation sets.
Revised: As metioned earlier, we perform grid-search to select the hyperparameters in classic algorithms using validation sets.

Original: The training time for both classic methods and learningbased methods are summarized in Table 3 and Table 4 in Appendix H.4.
Revised: The training time comparison is summarized in Table 3 and Tablein Appendix I.4.

Original: We can see that training time is not a bottleneck of this problem.
Revised: We can see that training time is not a bottleneck of this problem.

Original: Most methods can be trained efﬁciently.
Revised: Moreover, In

Original: In SPE, classic algorithms even require a longer training time than learning-based methods.
Revised: SPE, classic algorithms even require a longer training time than learning-based methods.

Original: 7. U NSUPERVISED L EARNING ON R EAL - WORLD D ATASETS
Revised: 7. U NSUPERVISED L EARNING ON R EAL - WORLD D ATASETS

Original: We conduct experiments of unsupervised algorithm learning on 3 datasets: Gene expression dataset (Kouno et al., 2013), Parkinsons patient dataset (Tsanas et al., 2009), and School exam score dataset (Zhou et al., 2011).
Revised: We conduct experiments of unsupervised algorithm learning on 3 datasets: Gene expression dataset (Kouno et al., 2013), Parkinsons patient dataset (Tsanas et al., 2009), and School exam score dataset (Zhou et al., 2011).

Original: See Appendix H.6 for details of datasets and the conﬁguration.
Revised: See Appendix I.6 for details of datasets and the conﬁguration.

Original: [Table]
Revised: [Table]

Original: The goal of the algorithm is to estimation the sparse linear regression parameters β ∗ for each problem.
Revised: The goal of the algorithm is to estimation the sparse linear regression parameters β ∗ for each problem.

Original: Recovery accuracy is estimated by the least-square error on a set of held-out samples for each problem.
Revised: Recovery accuracy is estimated by the least-square error on a set of held-out samples for each problem.

Original: Table 2 shows the effectiveness of PLISA .
Revised: Table 2 shows the effectiveness of PLISA .

Original: Note that these real-world datasets may not satisfy the assumptions in this paper.
Revised: Note that these real-world datasets may not satisfy the assumptions in this paper.

Original: This set of experiments are conducted only to demonstrate the robustness of the proposed method.
Revised: This set of experiments are conducted only to demonstrate the robustness of the proposed method.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposedan algorithm-unrolling based architecture called PLISA for learning to solve sparse parameter recovery problems.
Revised: We proposed PLISA for learning to solve sparse parameter recovery problems.

Original: We presented a theoretical analysis for both the capacity and generalization ability of PLISA .
Revised: We analyze its capacity and generalization ability.

Original: Theresults can shed new light on the relation between generalization and the algorithmic properties, and the techniques could be used to derive guarantees for other algorithmunrolling based architectures.
Revised: The techniques could be used to derive guarantees for other algorithm-unrolling based architectures.

Original: The model PLISA can be improved by using a more ﬂexible penalty function (e.g., a conventional neural network) as long as it satisﬁes Assumption B.1.
Revised: The model PLISA can be improved by using a more ﬂexible penalty function (e.g., a conventional neural network) as long as it satisﬁes Assumption B.1.

Original: A L IST OF DEFINITIONS AND NOTATIONS
Revised: A L IST OF DEFINITIONS AND NOTATIONS

Original: For the convenience of the reader
Revised: For the convenience of the reader

Original: [Equation]
Revised: [Equation]

Original: we summarize a list of notations blow.
Revised: we summarize a list of notations blow.

Original: (cid:107) β (cid:107) λ , 1 := (cid:80) dj
Revised: (cid:107) β (cid:107) λ , 1 := (cid:80) dj

Original: =

Revised: =

Original: λ j | β j | denotes element-wise (cid:96) 1 norm.
Revised: λ j | β j | denotes element-wise (cid:96) 1 norm.

Original: ( x ) S is the sub-vector of x with entries in the index set S .
Revised: ( x ) S is the sub-vector of x with entries in the index set S .

Original: x ∨ (cid:15) = max { x , (cid:15) } =
Revised: x ∨ (cid:15) = max { x , (cid:15) } =

Original: [max { x 1 , (cid:15) } , · · · , max { x d , (cid:15) } ] (cid:62) .
Revised: [max { x 1 , (cid:15) } , · · · , max { x d , (cid:15) } ] (cid:62) .

Original: Element-wise soft-threshold:
Revised: Element-wise soft-threshold:

Original: [Equation]
Revised: [Equation]

Original: Denote a single iteration of the modiﬁed proximal gradient step as
Revised: Denote a single iteration of the modiﬁed proximal gradient step as

Original: [Equation]
Revised: [Equation]

Original: B P ENALTY F UNCTIONS
Revised: B P ENALTY F UNCTIONS

Original: We provide detailed descriptions of the penalty functions used in PLISA .
Revised: We provide detailed descriptions of the penalty functions used in PLISA .

Original: We focus on learning the combination of three well-known penalty functions:
Revised: We focus on learning the combination of three well-known penalty functions:

Original: [Equation]
Revised: [Equation]

Original: where MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given below.
Revised: where MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given below.

Original: B. A NALYTICAL FORMS
Revised: B. A NALYTICAL FORMS

Original: The MCP penalty can be written as
Revised: The MCP penalty can be written as

Original: [Equation]
Revised: [Equation]

Original: where b > 0 is some hyperparameter.
Revised: where b > 0 is some hyperparameter.

Original: The SCAD penalty can be written as
Revised: The SCAD penalty can be written as

Original: [Equation]
Revised: [Equation]

Original: where a > 2 is some hyperparameter
Revised: where a > 2 is some hyperparameter

Original: [Equation]
Revised: [Equation]

Original: PLISA uses the concave components of these penalty functions.
Revised: PLISA uses the concave components of these penalty functions.

Original: Their analytical forms are as follows.
Revised: Their analytical forms are as follows.

Original: [Equation]
Revised: [Equation]

Original: B. I MPLICATIONS
Revised: B. I MPLICATIONS

Original: Based on the deﬁnitions of these penalty functions, it is easy to check that their convex combination satisﬁes a set of conditions as stated in the following Assumption B.1.
Revised: Based on the deﬁnitions of these penalty functions, it is easy to check that their convex combination satisﬁes a set of conditions as stated in the following Assumption B.1.

Original: Assumption B.1 (Regularization) .
Revised: Assumption B.1 (Regularization) .

Original: Let P ( λ , β ) =
Revised: Let P ( λ , β ) =

Original: Q ( λ , β )+ (cid:107) β (cid:107) λ , 1 where Q is concave in β , and that Q ( λ , β ) =
Revised: Q ( λ , β )+ (cid:107) β (cid:107) λ , 1 where Q is concave in β , and that Q ( λ , β ) =

Original: (cid:80) dj
Revised: (cid:80) dj

Original: =1 q ( λ j , β j ) .
Revised: =1 q ( λ j , β j ) .

Original: Denote the partial gradient of q as q (cid:48) λ ( β ) :
Revised: Denote the partial gradient of q as q (cid:48) λ ( β ) :

Original: =
Revised: =

Original: ∂q ( λ,β ) ∂β .
Revised: ∂q ( λ,β ) ∂β .

Original: Assume the following regularization conditions.
Revised: Assume the following regularization conditions.

Original: (a) There exists constants ξ ≥ 0 such that for any β (cid:48) >
Revised: (a) There exists constants ξ ≥ 0 such that for any β (cid:48) >

Original: β ,
Revised: β ,

Original: [Equation]
Revised: [Equation]

Original: (b) q ( λ, − β ) =
Revised: (b) q ( λ, − β ) =

Original: q ( λ, β ) .
Revised: q ( λ, β ) .

Original: (c) q (cid:48) λ
Revised: (c) q (cid:48) λ

Original: =
Revised: =

Original: q ( λ, 0) = 0 .
Revised: q ( λ, 0) = 0 .

Original: (d) | q (cid:48) λ
Revised: (d) | q (cid:48) λ

Original: ( β ) | ≤ λ .
Revised: ( β ) | ≤ λ .

Original: (e) | q (cid:48)
Revised: (e) | q (cid:48)

Original: λ 1 ( β ) − q (cid:48)
Revised: λ 1 ( β ) − q (cid:48)

Original: λ 2 ( β ) | ≤ | λ 1 − λ 2 | .
Revised: λ 2 ( β ) | ≤ | λ 1 − λ 2 | .

Original: Note that the penalty function P w ( λ , β ) in PLISA θ satisﬁes all these conditions, with the constant ξ in condition (a) being
Revised: Note that the penalty function P w ( λ , β ) in PLISA θ satisﬁes all these conditions, with the constant ξ in condition (a) being

Original: [Equation]
Revised: [Equation]

Original: where a and b are the hyperparameters in SCAD and MCP .
Revised: where a and b are the hyperparameters in SCAD and MCP .

Original: Next, we present an important lemma which states the restricted strongly convexity and smoothness of the modiﬁed loss
Revised: Next, we present an important lemma which states the restricted strongly convexity and smoothness of the modiﬁed loss

Original: [Equation]
Revised: [Equation]

Original: Lemma B.1 (Restricted Strongly Convex and Restricted Strongly Smooth) .
Revised: Lemma B.1 (Restricted Strongly Convex and Restricted Strongly Smooth) .

Original: Assume P satisﬁes Assumption 3.1 and ( Z n , β ∗ ) ∈ P .
Revised: Assume P satisﬁes Assumption C.1 and ( Z n , β ∗ ) ∈ P .

Original: If Q ( λ , β ) satisﬁes the regularization conditions in Assumption B. with a constant ξ < ρ − .
Revised: If Q ( λ , β ) satisﬁes the regularization conditions in Assumption B.1 with a constant ξ < ρ − .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: is strongly convex and smooth in the subspace {(cid:107) β (cid:48) − β [-(cid:107) 
-]
Revised: is strongly convex and smooth in the subspace {(cid:107) β [Equation] − β [-(cid:107) 
-][Equation]

Original:  ≤ s ∗ + 2˜ s } .
Revised: 0 ≤ s ∗ + 2˜ s } .

Original: That is, for any β , β (cid:48) such that (cid:107) β (cid:48) − β (cid:107) 

Revised: That is, for any β , β (cid:48) such that (cid:107) β (cid:48) − β (cid:107)

Original: ≤ s ∗ + 2˜ s , the following inequalities hold
Revised: ≤ s ∗ + 2˜ s , the following inequalities hold

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is straightforward by using the condition (a) in Assumption B.1 and the sparseeigenvalue condition in Assumption 3.1.
Revised: The proof is straightforward by using the condition (a) in Assumption B.1 and the sparseeigenvalue condition in Assumption C.1.

Original: CR ECOVERY G UARANTEE F OR P ARAMETERS I N C ONSTRAINED S PACE 
Revised: C P ROBLEM S PACE A SSUMPTIONS

Original: {+We follow the notations in Wang et al. ; Loh & Wainwright (2015) to describe some classic assumptions on the estimation problems. Assumption C.
+}
Revised: {+We follow the notations in Wang et al. ; Loh & Wainwright (2015) to describe some classic assumptions on the estimation problems. Assumption C.
+}

Revised: (Problem Space) .

Revised: Let s ∗ , ˜ s be positive integers and ρ − , ρ + be positive constants such that ˜ s >

Revised: (121( ρ + /ρ − )

Revised: + 144( ρ + /ρ − ) 2 ) s ∗ .

Revised: Assume for every estimation problem ( Z 1: n , β ∗ ) in the space P , the following conditions are satisﬁed.

Original: {+(a) (cid:107) β ∗ (cid:107) 
+}
Revised: {+(a) (cid:107) β ∗ (cid:107) 
+}

Revised: ≤ s ∗ and (cid:107) β ∗ (cid:107) ∞

Revised: ≤ B 1 ; (b) For any nonzero v ∈ R d with sparsity (cid:107) v (cid:107) 0 ≤ s ∗ + 2˜ s , it

Revised: [Table]

Revised: [Table]

Revised: (c) 8 | [ ∇ β L n ( Z 1: n , β ∗ )]

Revised: j | ≤ | [ ∇ β L n ( Z 1: n , 0 )]

Revised: j | ≤ B 2 , ∀ j = 1 , · · · , d .

Revised: Condition (a) assumes β ∗ is s ∗ -sparse and B 1 -bounded.

Revised: Condition (b) is commonly referred to as ‘sparse eigenvalue condition’ (Zhang, 2010b; Wang et al., 2014), which is weaker than the wellknown restricted isometry property (RIP) in compressed sensing (Candes & Tao, 2005).

Revised: Note that the class of functions satisfying conditions of this type is much larger than the class of convex losses.

Revised: In the special case when L n ( Z 1: n , β ) is strongly convex in β , condition (b) holds with ˜ s → ∞ .

Revised: The last condition bounds the gradient of the empirical loss L n at the true parameter β ∗ and 0 .

Revised: D R ECOVERY G UARANTEE F OR P ARAMETERS I N C ONSTRAINED S PACE

Original: Our generalization analysis is conducted on a constrained parameter space.
Revised: Our generalization analysis is conducted on a constrained parameter space.

Original: In this section, we provide theoretical guarantees for the estimation error of PLISA θ when θ is in the constrained space.
Revised: In this section, we provide theoretical guarantees for the estimation error of PLISA θ when θ is in the constrained space.

Original: We ﬁrst formally deﬁne the parameter space as follows.
Revised: We ﬁrst formally deﬁne the parameter space as follows.

Original: Deﬁnition C.1 (Constrained parameter space) .
Revised: Deﬁnition D.1 (Constrained parameter space) .

Original: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume P satisﬁes Assumption C.1 and D m ⊆ P .

Original: Given some positive values α min , η max , ξ max , and λ max .
Revised: Given some positive values α min , η max , ξ max , and λ max .

Original: We deﬁnite the constrained parameter space Θ
Revised: We deﬁnite the constrained parameter space Θ

Original: ( D m ) as Θ( D m ) :
Revised: ( D m ) as Θ( D m ) :

Original: = { θ = { η , λ ∗ , w , α } : conditions (i) (ii) (iii) (iv) are satisﬁed } , where the conditions are:
Revised: = { θ = { η , λ ∗ , w , α } : conditions (i) (ii) (iii) (iv) are satisﬁed } , where the conditions are:

Original: (i) α ∈ [ α min , 1 ρ + ] .
Revised: (i) α ∈ [ α min , 1 ρ + ] .

Original: (ii) η j ∈ [ η min :=
Revised: (ii) η j ∈ [ η min :=

Original: σ − 1 (0 . 9) , η max ] , for all j = 1 , · · · , d . (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , where b is the hyperparameter in MCP and a is the hyperparameter in SCAD .
Revised: σ − 1 (0 . 9) , η max ] , for all j = 1 , · · · , d . (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , where b is the hyperparameter in MCP and a is the hyperparameter in SCAD .

Original: (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]
Revised: (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ∨ ε, λ max ] , for all j = 1 , · · · , d .
Revised: j | ∨ ε, λ max ] , for all j = 1 , · · · , d .

Original: For parameters in this constrained space, we can show the performance guarantee for the outputs of
Revised: For parameters in this constrained space, we can show the performance guarantee for the outputs of

Original: PLISA θ , as stated in the following Theorem C.1.
Revised: PLISA θ , as stated in the following Theorem D.1.

Original: Theorem C.1 (Recovery guarantee) .
Revised: Theorem D.1 (Recovery guarantee) .

Original: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume P satisﬁes Assumption C.1 and D m ⊆ P .

Original: Let Θ( D m ) be the constrained parameter space deﬁned in Deﬁnition C.1 and assume θ = { η , λ ∗ , w ,
Revised: Let Θ( D m ) be the constrained parameter space deﬁned in Deﬁnition D.1 and assume θ = { η , λ ∗ , w ,

Original: α } ∈ Θ
Revised: α } ∈ Θ

Original: ( D m ) .
Revised: ( D m ) .

Original: Assume
Revised: Assume

Original: [Equation]
Revised: [Equation]

Original: where δ := 1 − α min ( ρ − − ξ max ) .
Revised: where δ := 1 − α min ( ρ − − ξ max ) .

Original: Then for any problem ( Z 1: n , β ∗ ) in the training set D m , all intermediate outputs of PLISA θ have the following error bounds for all t
Revised: Then for any problem ( Z 1: n , β ∗ ) in the training set D m , all intermediate outputs of PLISA θ have the following error bounds for all t

Original: = 1 , · · · , T .
Revised: = 1 , · · · , T .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this theorem is based on a series of key lemmas for the properties of the modiﬁed proximal gradient steps in each cell of PLISA .
Revised: The proof of this theorem is based on a series of key lemmas for the properties of the modiﬁed proximal gradient steps in each cell of PLISA .

Original: We state these key lemmas and their proofs in
Revised: We state these key lemmas and their proofs in

Original: Section F. With these lemmas, the proof of this theorem is straightforward, and we state the proof below.
Revised: Section G. With these lemmas, the proof of this theorem is straightforward, and we state the proof below.

Original: Given a ﬁxed problem ( Z 1: n , β ∗ ) ∈
Revised: Given a ﬁxed problem ( Z 1: n , β ∗ ) ∈

Original: D m and a ﬁxed set of parameters θ = { η , λ ∗ , w , α } ∈ Θ( D m ) , we simplify some notations in this proof by removing the dependency on Z 1: n and θ .
Revised: D m and a ﬁxed set of parameters θ = { η , λ ∗ , w , α } ∈ Θ( D m ) , we simplify some notations in this proof by removing the dependency on Z 1: n and θ .

Original: For example, we denote β t =
Revised: For example, we denote β t =

Original: β t
Revised: β t

Original: ( Z 1: n ; θ ) , (cid:101) β
Revised: ( Z 1: n ; θ ) , (cid:101) β

Original: kt = (cid:101) β kt ( Z 1: n ; θ ) , λ t = λ t ( Z 1: n ; θ ) , etc.
Revised: kt = (cid:101) β kt ( Z 1: n ; θ ) , λ t = λ t ( Z 1: n ; θ ) , etc.

Original: Notations.
Revised: Notations.

Original: Furthermore, we will use the following simpliﬁed notations and deﬁnitions.
Revised: Furthermore, we will use the following simpliﬁed notations and deﬁnitions.

Original: empirical loss L n ( β ) :
Revised: empirical loss L n ( β ) :

Original: = L n ( Z 1: n , β ) modiﬁed loss L λ
Revised: = L n ( Z 1: n , β ) modiﬁed loss L λ

Original: [Equation]
Revised: [Equation]

Original: β )
Revised: β )

Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Original: regularized loss φ λ ( β )
Revised: regularized loss φ λ ( β )

Original: [Equation]
Revised: [Equation]

Revised: =

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: : n 
Revised: : n ,

Original: [Equation]
Revised: [Equation]

Original: local optimal
Revised: local optimal

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to state the proof.
Revised: Now we are ready to state the proof.

Original: We ﬁrst show the following statement holds true for all t ≤ T by mathematical induction.
Revised: We ﬁrst show the following statement holds true for all t ≤ T by mathematical induction.

Original: Statement ( t) :
Revised: Statement [Equation] t

Original: ω λ t ( (cid:101) β 0 t ) ≤ 1 2 , and (cid:107) ( (cid:101) β kt ) S ∗ (cid:107) 

Revised: [Equation] [Equation] and (cid:107) ( (cid:101) β kt ) S ∗ (cid:107)

Original: ≤ ˜ s, ∀ k = 1 , · · · , K.
Revised: ≤ ˜ s, ∀ k = 1 , · · · , K.

Original: Statement (1) .
Revised: Statement (1) .

Original: First we verify that Statement (1) holds true.
Revised: First we verify that Statement (1) holds true.

Original: Recall that λ 0 = |∇
Revised: Recall that λ 0 = |∇

Original: L n ( 0 ) | .
Revised: L n ( 0 ) | .

Original: In the following, we prove that (cid:101) β 01 is a local solution of φ λ 0 ( β) : = L
Revised: In the following, we prove that (cid:101) β 01 is a local solution of φ λ [Equation] β

Original: [Equation]
Revised: [Equation]

Original: 0 ( β ) + (cid:107) β (cid:107) λ 0 , 1 .
Revised: := [Equation] + [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since − sign( ∇ L n ( 0 )) ∈
Revised: Since − sign( ∇ L n ( 0 )) ∈

Original: ∂ (cid:107) 

Revised: ∂ (cid:107)

Original: (cid:107) 1 , then we have 0 ∈ ∇ L λ 0 ( (cid:101) β 01 ) + λ 0 ◦
Revised: (cid:107) 1 , then we have 0 ∈ ∇ L λ 0 ( (cid:101) β 01 ) + λ 0 ◦

Original: ∂ (cid:107) (cid:101) β 01 (cid:107) 1 .
Revised: ∂ (cid:107) (cid:101) β 01 (cid:107) 1 .

Original: Therefore, ω λ 0 ( (cid:101) β 01 ) = 0 .
Revised: Therefore, ω λ 0 ( (cid:101) β 01 ) = 0 .

Original: Since λ 1 = λ 0 ◦ η 1 , by Lemma F.6, we have
Revised: Since λ 1 = λ 0 ◦ η 1 , by Lemma G.6, we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma F.5, we have φ λ 1 ( [Equation] β 01 )
Revised: By Lemma G.5, we have φ λ 1 ( (cid:101) β 01 )

Original: [Equation]
Revised: [Equation]

Original: max , which implies that the conditions in Lemma F.4 are satisﬁed.
Revised: max , which implies that the conditions in Lemma G.4 are satisﬁed.

Original: Therefore, we have proved that
Revised: Therefore, we have proved that

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof of Statement (1) .
Revised: This ﬁnishes the proof of Statement (1) .

Original: Statement ( t ) .
Revised: Statement ( t ) .

Original: Now we assume Statement ( t − 1) is true and prove Statement ( t ) .
Revised: Now we assume Statement ( t − 1) is true and prove Statement ( t ) .

Original: First, we prove that ω λ
Revised: First, we prove that ω λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: β Kt − 1 ) ≤ 1 / 4 .
Revised: β Kt − 1 ) ≤ 1 / 4 .

Original: By Lemma F.2 and Lemma F.5,
Revised: By Lemma G.2 and Lemma G.5,

Original: [Equation]
Revised: [Equation]

Original: Assume K ≥ log (cid:16) 4 (cid:113) 29 ρ + ρ − − ξ max (1 +
Revised: Assume K ≥ log (cid:16) 4 (cid:113) 29 ρ + ρ − − ξ max [Equation] +

Original: [Equation]
Revised: [Equation]

Original: + 1 , where
Revised: + 1 , where

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:101) β 0 t = (cid:101) β
Revised: Since (cid:101) β 0 t = (cid:101) β

Original: Kt − 1 , Lemma F.6 implies that
Revised: Kt − 1 , Lemma G.6 implies that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by Lemma F.5, ω λ t ( (cid:101) β 0 t ) ≤ 1 / 2 implies
Revised: Furthermore, by Lemma G.5, ω λ t ( (cid:101) β 0 t ) ≤ 1 / 2 implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which in turns implies that the conditions in Lemma F.4 are satisﬁed.
Revised: which in turns implies that the conditions in Lemma G.4 are satisﬁed.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of Statement ( t ) .
Revised: which completes the proof of Statement ( t ) .

Original: Now we derive the error bounds.
Revised: Now we derive the error bounds.

Original: Similar to how we derive Eq.
Revised: Similar to how we derive Eq.

Original: 25 and Eq. 26, it is easy to show that ω λ t
Revised: 26 and Eq. 27, it is easy to show that ω λ t

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , and φ λt
Revised: and φ λ

Original: [Equation]
Revised: [Equation]

Original:  β 
Revised: Combining φ λ t ( (cid:101) β 0 t ) − φ λ t ( β ∗ ) ≤ 21 / 2 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 22 ρ −
Revised: t ) S ∗

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T. Combining φ λ t ( (cid:101) β 0 t ) −φ λ t ( β ∗ ) ≤ 21 / 2 (cid:107) ( λ t ) S ∗ (cid:107) 22 ρ −− ξ max with Lemma F.3, we have
Revised: − −

Original: [Equation]
Revised: [Equation]

Original: Combining ω λ t ( (cid:101) β Kt ) ≤ 1 / 4 with Lemma F.5, we have(cid:107)
Revised: max with Lemma G.3, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Combining ω λ t ( (cid:101) β Kt ) ≤ 1 / 4 with Lemma G.5, we have

Original: [Equation]
Revised: [Equation]

Original: 19 / 8 (cid:107) ( λ t ) S ∗ (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]
Revised: [Equation]

Original: max .

Original: D C APACITY A NALYSIS : P ROOF OF T HEOREM 3. 
Revised: E C APACITY A NALYSIS : P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: This theorem is a direct result of Theorem C.1, by taking a suitable set of parameters θ .
Revised: This theorem is a direct result of Theorem D.1, by taking a suitable set of parameters θ .

Original: Similar to the proof of Theorem C.1, some notations are simpliﬁed.
Revised: Similar to the proof of Theorem D.1, some notations are simpliﬁed.

Original: Please refer to the proof of
Revised: Please refer to the proof of

Original: Theorem C.1 for detailed illustrations.
Revised: Theorem D.1 for detailed illustrations.

Original: Denote the union support set of the true parameters β ∗ by S ∗ m :=
Revised: Denote the union support set of the true parameters β ∗ by S ∗ m :=

Original: { j : j ∈ supp ( β ∗ ) , ( Z 1: n , β ∗ ) ∈
Revised: { j : j ∈ supp ( β ∗ ) , ( Z 1: n , β ∗ ) ∈

Original: D m } .
Revised: D m } .

Original: Then we specify a set of parameters θ = { η , λ ∗ , w , α } for PLISA θ as follows.
Revised: Then we specify a set of parameters θ = { η , λ ∗ , w , α } for PLISA θ as follows.

Original: • α = 1 ρ + .
Revised: • α = 1 ρ + .

Original: • w : The weights w can take any values as long as they satisfy (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 < ρ − .
Revised: • w : The weights w can take any values as long as they satisfy (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 < ρ − .

Original: Since (cid:102) w 2 and (cid:102) w 3 and be arbitrary close to 0, there must exists a set of weights w that satisfy this constraint.
Revised: Since (cid:102) w 2 and (cid:102) w 3 and be arbitrary close to 0, there must exists a set of weights w that satisfy this constraint.

Original: In particular, we denote this value by ξ w := (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 .
Revised: In particular, we denote this value by ξ w := (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 .

Original: • λ ∗ :
Revised: • λ ∗ :

Original: For each j ∈ S ∗ m , we take λ ∗ j := 8 max ( Z n , β ∗ ) ∈D m |∇ L n ( Z n , β ∗ )
Revised: For each j ∈ S ∗ m , we take λ ∗ j := 8 max ( Z n , β ∗ ) ∈D m |∇ L n ( Z n , β ∗ )

Original: j | ∨ ε as the target regularization parameter.
Revised: j | ∨ ε as the target regularization parameter.

Original: For each j / ∈ S ∗ m , we take λ ∗ j = B 2 , which is the upper bound of (cid:107)∇ β L n ( Z 1: n , 0 ) (cid:107) ∞ by Assumption 3.1.
Revised: For each j / ∈ S ∗ m , we take λ ∗ j = B 2 , which is the upper bound of (cid:107)∇ β L n ( Z 1: n , 0 ) (cid:107) ∞ by Assumption C.1.

Original:  η  For all j , take η j = log 9 so that the decrease ratio is σ ( η j ) = 0 .9 .
Revised: • η : For all j , take η j = log 9 so that the decrease ratio is σ ( η j ) = 0 .

Original: Clearly, this set of parameters satisfy the conditions in Theorem C.1, so we can apply its result in this proof.
Revised: Clearly, this set of parameters satisfy the conditions in Theorem D.1, so we can apply its result in this proof.

Original: In the following, we will show that, with this speciﬁcation of the parameters, PLISA θ can achieve the recovery accuracy stated in Theorem 3.1.
Revised: In the following, we will show that, with this speciﬁcation of the parameters, PLISA θ can achieve the recovery accuracy stated in Theorem 3.1.

Original: Let t 0 =
Revised: Let t 0 =

Original: min { t : t ≥ 0 , λ ∗ = max { 0 .
Revised: min { t : t ≥ 0 , λ ∗ = max { 0 .

Original: 9 t |∇ β L n ( Z 1: n , 0 ) | , λ ∗ } , Z 1: n ∈ D m , λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t ( Z 1: n ; θ )
Revised: 9 t |∇ β L n ( Z 1: n , 0 ) | , λ ∗ } , Z 1: n ∈ D m , λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t ( Z 1: n ; θ )

Original: = λ ∗ .
Revised: = λ ∗ .

Original: We know t 0 is a small number because the decrease rate is linear.
Revised: We know t 0 is a small number because the decrease rate is linear.

Original: To be more clear, it is easy to show that t 0 ≤ log (cid:0) B 2 ε (cid:1) / log(10 / 9) .
Revised: To be more clear, it is easy to show that t 0 ≤ log (cid:0) B 2 ε (cid:1) / log(10 / 9) .

Original: Therefore, after t 0 cells, the regularization parameters do not change anymore.
Revised: Therefore, after t 0 cells, the regularization parameters do not change anymore.

Original: That is,λ
Revised: That is,

Original: [Equation]

Original: ) =

Original: [Equation]
Revised: [Equation]

Original: ∀

Original: [Equation] ≥ [Equation] Since the speciﬁed parameters satisfy the conditions in Theorem C.1, we can follow the same way as how we derive Eq.
Revised: Since the speciﬁed parameters satisfy the conditions in Theorem D.1, we can follow the same way as how we derive Eq.

Original:  to obtain that
Revised: 26 to obtain that

Original: [Equation]
Revised: [Equation]

Original: Since we have λ t = λ ∗ for t ≥ t 0 .
Revised: Since we have λ t = λ ∗ for t ≥ t 0 .

Original: The last ( T − t 0 ) cells can be viewed as a single cell with
Revised: The last ( T − t 0 ) cells can be viewed as a single cell with

Original: K ( T − t 0 ) many steps.
Revised: K ( T − t 0 ) many steps.

Original: Therefore, we can apply Lemma F.2 and Lemma F.5 to obtain
Revised: Therefore, we can apply Lemma G.2 and Lemma G.5 to obtain

Original: [Equation]
Revised: [Equation]

Original: ( − C θ K ( t − t 0 )) , where C θ = − 12 log (1 − α ( ρ − − ξ w )) , c θ = (cid:113) 29 ρ + ρ − − ξ w (1 + 1 αρ + )
Revised: ( − C θ K ( t − t 0 )) , where C θ = − 12 log (1 − α ( ρ − − ξ w )) , c θ = (cid:113) 29 ρ + ρ − − ξ w (1 + 1 αρ + )

Original: B 2 1 − α ( ρ − − ξ w ) .
Revised: B 2 1 − α ( ρ − − ξ w ) .

Original: By Lemma F.5, (cid:107) [Equation] t 
Revised: By Lemma G.5, (cid:107) β t − β ∗

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: t (

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: ˜ c θ = √ s ∗ B 2 ρ − − ξ w c θ
Revised: ˜ c θ = √ s ∗ B 2 ρ − − ξ w c θ

Original: and c (cid:48) θ = 17 8( ρ − − ξ w ) .
Revised: and c (cid:48) θ = 17 8( ρ − − ξ w ) .

Original: What remains is bounding (cid:107) λ ∗ S ∗ (cid:107) 2 .
Revised: What remains is bounding (cid:107) λ ∗ S ∗ (cid:107) 2 .

Original: First, we deﬁne a condition number as
Revised: First, we deﬁne a condition number as

Original: [Equation]
Revised: [Equation]

Original: In fact, κ m ≤ B 2 ε .
Revised: In fact, κ m ≤ B 2 ε .

Original: Then recall the speciﬁcation of λ ∗ at the beginning of this proof.
Revised: Then recall the speciﬁcation of λ ∗ at the beginning of this proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Plugging this back to Eq.
Revised: Plugging this back to Eq.

Original: 28, we have
Revised: 29, we have

Original: [Equation]
Revised: [Equation]

Original: This is equivalent to the statement to be proved.
Revised: This is equivalent to the statement to be proved.

Original: E G ENERALIZATION A NALYSIS : P ROOF OF T HEOREM 4. 
Revised: F G ENERALIZATION A NALYSIS : P ROOF OF T HEOREM 4.

Original: We ﬁrst state a well-known inequality that bounds the generalization gap by empirical Rademacher complexity .
Revised: We ﬁrst state a well-known inequality that bounds the generalization gap by empirical Rademacher complexity .

Original: Theorem E.1 (Adapted from Theorem 26.5 (Shalev-Shwartz & Ben-David, 2014)) .
Revised: Theorem F.1 (Adapted from Theorem 26.5 (Shalev-Shwartz & Ben-David, 2014)) .

Original: Assume that for all ( Z 1: n , β ∗ ) ∈ P and all θ ∈ Θ we have that (cid:96) ( β T ( Z 1: n ; θ ) , β ∗ ) ≤ c .
Revised: Assume that for all ( Z 1: n , β ∗ ) ∈ P and all θ ∈ Θ we have that (cid:96) ( β T ( Z 1: n ; θ ) , β ∗ ) ≤ c .

Original: Then with probability at least 1 − (cid:15) , for all θ ∈ Θ ,
Revised: Then with probability at least 1 − (cid:15) , for all θ ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } is the space of loss functions of our model PLISA θ , and R m ( (cid:96) Θ ) is its empirical Rademacher complexity.
Revised: where (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } is the space of loss functions of our model PLISA θ , and R m ( (cid:96) Θ ) is its empirical Rademacher complexity.

Original: It is deﬁned as
Revised: It is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where { σ i } mi =1 are m independent Rademacher random variables.
Revised: where { σ i } mi =1 are m independent Rademacher random variables.

Original: Therefore, it resorts to bound the empirical Rademacher complexity, which can be bounded via
Revised: Therefore, it resorts to bound the empirical Rademacher complexity, which can be bounded via

Original: Dudley’s integral.
Revised: Dudley’s integral.

Original: The following theorem states the bound for the empirical Rademacher complexity and its proof follows.
Revised: The following theorem states the bound for the empirical Rademacher complexity and its proof follows.

Original: Theorem E.2 (Empirical Rademacher complexity bound) .
Revised: Theorem F.2 (Empirical Rademacher complexity bound) .

Original: Assume the assumptions in Theorem 4.1.
Revised: Assume the assumptions in Theorem 4.1.

Original: Let Θ = Θ
Revised: Let Θ = Θ

Original: ( D m ) be the constrained space deﬁned in Deﬁnition C.1.
Revised: ( D m ) be the constrained space deﬁned in Deﬁnition D.1.

Original: Let (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } be the space of loss functions of our model PLISA θ .
Revised: Let (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } be the space of loss functions of our model PLISA θ .

Original: Then the empirical Rademacher complexity is bounded by R m (cid:96) Θ ≤ c 1 √
Revised: Then the empirical Rademacher complexity is bounded by R m (cid:96) Θ ≤ c 1 √

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 3 , c 4 are some constants independent of d, m, K and T .
Revised: where c 1 , c 2 , c 3 , c 4 are some constants independent of d, m, K and T .

Original: Proof.
Revised: Proof.

Original: The classical Dudley’s entropy integral bound gives us an upper bound for the empirical
Revised: The classical Dudley’s entropy integral bound gives us an upper bound for the empirical

Original: Rademacher complexity in terms of the covering number.
Revised: Rademacher complexity in terms of the covering number.

Original: [Equation]
Revised: [Equation]

Original: The notation P m means the empirical measure deﬁned based on the samples in D m , and
Revised: The notation P m means the empirical measure deﬁned based on the samples in D m , and

Original: [Equation]
Revised: [Equation]

Original: by Theorem C.
Revised: by Theorem D.

Original: [Equation]
Revised: [Equation]

Original: supθ
Revised: sup

Original: [Equation]
Revised: [Equation]

Original: Here we use a constant C as the upper bound because we only care about the scales in d, m, K and
Revised: Here we use a constant C as the upper bound because we only care about the scales in d, m, K and

Original: T .
Revised: T .

Original: Next, we bound the covering number N (cid:16) 1 √ m , (cid:96) Θ , L 2 ( P m ) (cid:17) .
Revised: Next, we bound the covering number N (cid:16) 1 √ m , (cid:96) Θ , L 2 ( P m ) (cid:17) .

Original: By the key result in Lemma 4.1, it is easy to derive that
Revised: By the key result in Lemma 4.1, it is easy to derive that

Original: [Equation]
Revised: [Equation]

Original: From now on
Revised: From now on

Original: [Equation]
Revised: [Equation]

Original: we abuse the symbol c to generally represent some constant that does not depend on d, m, K, T
Revised: we abuse the symbol c to generally represent some constant that does not depend on d, m, K, T

Original: [Equation]
Revised: [Equation]

Original: For the covering number of a d -dimensional compact vector space T ⊆
Revised: For the covering number of a d -dimensional compact vector space T ⊆

Original: R d , we can bound it by the (cid:15)
Revised: R d , we can bound it by the (cid:15)

Original: -packing number.
Revised: -packing number.

Original: That is, N ( T, (cid:15), (cid:96) 2 ) ≤ M ( T, (cid:15), (cid:96) 2 ) ≤ c (cid:81) dj
Revised: That is, N ( T, (cid:15), (cid:96) 2 ) ≤ M ( T, (cid:15), (cid:96) 2 ) ≤ c (cid:81) dj

Original: =1 (cid:16) T j, max − T j, min (cid:15) + 1 (cid:17) for some constant c .
Revised: =1 (cid:16) T j, max − T j, min (cid:15) + 1 (cid:17) for some constant c .

Original: Applying this fact, we have
Revised: Applying this fact, we have

Original: [Equation]

Original: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: which implies log
Revised: which implies log

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: 1 − exp( − C Θ KT ))
Revised: − exp( − C Θ KT ))

Original: + 1 (cid:1) + 3 log (cid:16) c √ dm (1 − exp( − C Θ KT ))
Revised: + 1 (cid:1) + 3 log [Equation]

Original:  (cid:17) ≤ log (cid:0) c √ mK ( T − t 0 ) exp( − C Θ K ( T − t 0 ))
Revised: ( − C Θ KT )) (cid:17) ≤ log (cid:0) c √ mK ( T − t 0 ) exp( − C Θ K ( T − t 0 ))

Original: ∨ 1 (cid:1) + cd log (cid:0) c √ m (1 − exp( − C Θ KT ))
Revised: ∨ 1 (cid:1) + cd log (cid:0) c √ m (1 − exp( − C Θ KT ))

Original: + 1 (cid:1) .
Revised: + 1 (cid:1) .

Original: Therefore, R m (cid:96) Θ ≤c
Revised: Therefore, R m (cid:96) Θ ≤

Original: [Equation]
Revised: [Equation]

Original: E. R OBUSTNESS TO P ARAMETER P ERTURBATION : P ROOF OF L EMMA 4. 
Revised: F. R OBUSTNESS TO P ARAMETER P ERTURBATION : P ROOF OF L EMMA 4.

Original: We split the parameters θ = { λ ∗ , w , η , α } into two sets θ = θ 1 ∪ θ 2 , where θ 1 = { α } and θ 2 = { λ ∗ , w , η } .
Revised: We split the parameters θ = { λ ∗ , w , η , α } into two sets θ = θ 1 ∪ θ 2 , where θ 1 = { α } and θ 2 = { λ ∗ , w , η } .

Original: Clearly, the robustness can be bounded by
Revised: Clearly, the robustness can be bounded by

Original: [Equation]
Revised: [Equation]

Original: We derive the upper bounds of these two terms in Lemma E.1 and
Revised: We derive the upper bounds of these two terms in Lemma F.1 and

Original: Lemma E.
Revised: Lemma F.

Original: [Equation]
Revised: [Equation]

Original: which imply
Revised: which imply

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .

Original: Lemma 4.1 is equivalent to this inequality by taking
Revised: Lemma 4.1 is equivalent to this inequality by taking

Original: C Θ = − log( δ ) > 0 .
Revised: C Θ = − log( δ ) > 0 .

Original: Therefore, the key derivation steps are in the proof of Lemma E.1 and
Revised: Therefore, the key derivation steps are in the proof of Lemma F.1 and

Original: Lemma E.2, as stated below.
Revised: Lemma F.2, as stated below.

Original: Lemma E.1.
Revised: Lemma F.1.

Original: Assume the assumptions in Lemma 4.1 are satisﬁed.
Revised: Assume the assumptions in Lemma 4.1 are satisﬁed.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: , α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: } be parameters in the constrained space Θ .
Revised: } be parameters in the constrained space Θ .

Original: Let { β t
Revised: Let { β t

Original: Tt =1 = PLISA θ
Revised: Tt =1 = PLISA θ

Original: ( Z 1: n ) and { β t (cid:48) } Tt =

Revised: ( Z 1: n ) and { β t (cid:48) } Tt =

Original: = PLISA θ (cid:48)
Revised: = PLISA θ (cid:48)

Original: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .
Revised: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .

Original: Assume
Revised: Assume

Original: K ≥log
Revised: K ≥

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By triangular inequality,
Revised: By triangular inequality,

Original: [Equation]
Revised: [Equation]

Original: WLOG, assume α > α (cid:48) .
Revised: WLOG, assume α > α (cid:48) .

Original: Applying Lemma E.3 and Lemma E.4 to the above two terms on the right hand side
Revised: Applying Lemma F.3 and Lemma F.4 to the above two terms on the right hand side

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where c is the Lipschtiz constant.
Revised: where c is the Lipschtiz constant.

Original: c is a ﬁnite number because α is bounded and positive.
Revised: c is a ﬁnite number because α is bounded and positive.

Original: Applying it recursively
Revised: Applying it recursively

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we bound the term (cid:13)(cid:13)(cid:13) (cid:101) β kt − (cid:101)
Revised: Now we bound the term (cid:13)(cid:13)(cid:13) (cid:101) β kt − (cid:101)

Original: β k − 1 t (cid:13)(cid:13)(cid:13) 2 .
Revised: β k − 1 t (cid:13)(cid:13)(cid:13) 2 .

Original: By Lemma F.1 and Lemma F.2, (cid:13)(cid:13)(cid:13)(cid:101) [Equation] kt [Equation]
Revised: By Lemma G.1 and Lemma G.2, (cid:13)(cid:13)(cid:13)

Original: k − 1 t

Original: [Equation]
Revised: [Equation]

Original: min

Original: [Equation]

Original: [Equation]

Original: ≤

Original: [Equation]

Original: (By Lemma F.6)
Revised: (By Lemma G.6)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where c (cid:48) :
Revised: where c (cid:48) :

Original: = (cid:113) 2 ρ + (cid:113) 10 ρ − − ξ max B 2 δ − 1 .
Revised: = (cid:113) 2 ρ + (cid:113) 10 ρ − − ξ max B 2 δ − 1 .

Original: However, we can have a tighter bound for this term if t is larger than a certain integer.
Revised: However, we can have a tighter bound for this term if t is larger than a certain integer.

Original: More precisely, recall the deﬁnition that λ t = max { λ 0 ◦ σ ( η ) t , λ ∗ } .
Revised: More precisely, recall the deﬁnition that λ t = max { λ 0 ◦ σ ( η ) t , λ ∗ } .

Original: Since σ ( η max ) < 1 , the entries of λ 0 ◦ σ ( η ) t will decrease exponentially to reach the entry values of λ ∗ .
Revised: Since σ ( η max ) < 1 , the entries of λ 0 ◦ σ ( η ) t will decrease exponentially to reach the entry values of λ ∗ .

Original: Let t 0 =
Revised: Let t 0 =

Original: min { t : t ≥ 0 , λ ∗ = max { λ 0 ◦ σ ( η max ) t , λ ∗ } , ∀ λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t = λ ∗ .
Revised: min { t : t ≥ 0 , λ ∗ = max { λ 0 ◦ σ ( η max ) t , λ ∗ } , ∀ λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t = λ ∗ .

Original: We know t 0 is a small number because the decrease rate is linear.
Revised: We know t 0 is a small number because the decrease rate is linear.

Original: Now, the regularization parameters satisfy λ t = λ ∗ , ∀ t ≥ t 0 .
Revised: Now, the regularization parameters satisfy λ t = λ ∗ , ∀ t ≥ t 0 .

Original: Therefore, we can apply Lemma F. and Lemma F.2 again to obtain the following bound for all t ≥ t 0 :
Revised: Therefore, we can apply Lemma G. and Lemma G.2 again to obtain the following bound for all t ≥ t 0 :

Original: [Equation]
Revised: [Equation]

Original:  Lemma F.6)
Revised: By Lemma G.6)

Original: [Equation]
Revised: [Equation]

Original: Combining the two different bounds for (cid:13)(cid:13)(cid:13)
Revised: Combining the two different bounds for (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: β kt − (cid:101) β k 
Revised: β kt − (cid:101) β k −

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13) 2 with Eq. 30, we have
Revised: (cid:13)(cid:13)(cid:13) 2 with Eq. 31, we have

Original: [Equation]
Revised: [Equation]

Original: where C = cc (cid:48)
Revised: where C = cc (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We apply the above inequality recursively and obtain that
Revised: We apply the above inequality recursively and obtain that

Original: [Equation]
Revised: [Equation]

Original: Lemma E.2.
Revised: Lemma F.2.

Original: Assume the assumptions in Lemma 4.1 are satisﬁed.
Revised: Assume the assumptions in Lemma 4.1 are satisﬁed.

Original: Letθ
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: be parameters in the constrained space Θ .
Revised: be parameters in the constrained space Θ .

Original: Let { β t
Revised: Let { β t

Original: Tt =1 = PLISA θ
Revised: Tt =1 = PLISA θ

Original: ( Z 1: n ) and { β t (cid:48) } Tt =

Revised: ( Z 1: n ) and { β t (cid:48) } Tt =

Original: = PLISA θ (cid:48)
Revised: = PLISA θ (cid:48)

Original: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .
Revised: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 and C 1 , 2 , 4 are some constants.
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 and C 1 , 2 , 4 are some constants.

Original: Proof.
Revised: Proof.

Original: Let { β t
Revised: Let { β t

Original: Tt =1 ∪ { (cid:101) β kt } Tt =

Revised: Tt =1 ∪ { (cid:101) β kt } Tt =

Original: Kk =1 and { β t (cid:48) }
Revised: Kk =1 and { β t (cid:48) }

Original: Tt =1 ∪ { (cid:101) β kt (cid:48) }
Revised: Tt =1 ∪ { (cid:101) β kt (cid:48) }

Original: Tt =

Revised: Tt =

Original: Kk =1 be the intermediate outputs of PLISA θ
Revised: Kk =1 be the intermediate outputs of PLISA θ

Original: ( Z 1: n ) and PLISA θ (cid:48)
Revised: ( Z 1: n ) and PLISA θ (cid:48)

Original: ( Z 1: n ) respectively.
Revised: ( Z 1: n ) respectively.

Original: By triangle inequality,
Revised: By triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma E
Revised: Applying Lemma F.3, Lemma F.

Original: [Equation]
Revised: [Equation]

Original: [Equation] Lemma E.
Revised: and Lemma F.6 to the above 3 terms on the right hand side

Original: [Equation]
Revised: [Equation]

Original: and Lemma E.6 to the above 3 terms on the right hand side, we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where C is some absolute constant
Revised: where C is some absolute constant

Original: [Equation]
Revised: [Equation]

Original: Term ( i ) .
Revised: Term ( i ) .

Original: By deﬁnition and triangular inequality,
Revised: By deﬁnition and triangular inequality,

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds by Assumption 3.1 and the bounded ﬁrst derivative of σ ( · ) .
Revised: The last inequality holds by Assumption C.1 and the bounded ﬁrst derivative of σ ( · ) .

Original: Therefore, for some constants C 2 = 2 ρ + and ˜ C 1 = C 2 B 2 4 ,
Revised: Therefore, for some constants C 2 = 2 ρ + and ˜ C 1 = C 2 B 2 4 ,

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds because the value σ ( η max ) t t is bounded by some constant.
Revised: The last inequality holds because the value σ ( η max ) t t is bounded by some constant.

Original: Term ( ii ) .
Revised: Term ( ii ) .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Plugging the bounds for (i)-(iii) into Eq. 31, we have
Revised: Plugging the bounds for (i)-(iii) into Eq. 32, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  direct computation
Revised: By direct computation,

Original: [Equation]
Revised: [Equation]

Original: Recall that (cid:101)
Revised: Recall that (cid:101)

Original: β
Revised: β

Original: Kt = β t and (cid:101) β 0 t = β t − 1 .
Revised: Kt = β t and (cid:101) β 0 t = β t − 1 .

Original: We can apply the above inequality recursively and obtain that
Revised: We can apply the above inequality recursively and obtain that

Revised: [Equation]

Revised: T

Original: [Equation]
Revised: [Equation]

Original: Lemma E.3 (Robustness to β ) .
Revised: Lemma F.3 (Robustness to β ) .

Original: Assume ( Z 1: n , β ∗ ) ∈ P and P satisﬁes Assumption 3.1.
Revised: Assume ( Z 1: n , β ∗ ) ∈ P and P satisﬁes Assumption C.1.

Original: Assume β , β (cid:48) satisfy (cid:107) β − β (cid:48)
Revised: Assume β , β (cid:48) satisfy (cid:107) β − β (cid:48)

Original: (cid:107) 

Revised: (cid:107)

Original: ≤ s = s ∗ + 2˜ s and α ≤ 1 ρ + .
Revised: ≤ s = s ∗ + 2˜ s and α ≤ 1 ρ + .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: λ

Original: [Equation]
Revised: [Equation]

Original: , w ,
Revised: w ,

Original: [Equation]
Revised: [Equation]

Original: ; λ
Revised: ; λ

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: α )

Original: [Equation][Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since on the restricted subspace { β : (cid:107) β − β ∗ (cid:107) 

Revised: Since on the restricted subspace { β : (cid:107) β − β ∗ (cid:107)

Original: ≤ s ∗ + 2˜ s }
Revised: ≤ s ∗ + 2˜ s }

Original: [Equation]
Revised: [Equation]

Original: L λ is ( ρ − − ξ w ) -strongly convex and ρ + -smooth, by Lemma F
Revised: L λ is ( ρ − − ξ w ) -strongly convex and ρ + -smooth, by Lemma G

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq.
Revised: Combining this inequality with Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By the assumption that α ≤ 1 ρ + , the second term in the above inequality is non-positive.
Revised: By the assumption that α ≤ 1 ρ + , the second term in the above inequality is non-positive.

Original: Furthermore, this assumption also implies that 1 − 2 α ( ρ − − ξ w ) ρ
Revised: Furthermore, this assumption also implies that 1 − 2 α ( ρ − − ξ w ) ρ

Original: + ρ − − ξ w + ρ + ≥ 0 .
Revised: + ρ − − ξ w + ρ + ≥ 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma E.4 (Robustness to α ) .
Revised: Lemma F.4 (Robustness to α ) .

Original: With out loss of generalization, assume α ≥ α (cid:48) .
Revised: With out loss of generalization, assume α ≥ α (cid:48) .

Original: Denote β + := MPG ( β ; λ , w , α ) and β + (cid:48) :
Revised: Denote β + := MPG ( β ; λ , w , α ) and β + (cid:48) :

Original: =
Revised: =

Original: MPG ( β ; λ , w , α (cid:48) ) .
Revised: MPG ( β ; λ , w , α (cid:48) ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne the quadratic approximation function as
Revised: Deﬁne the quadratic approximation function as

Original: [Equation]
Revised: [Equation]

Original: Clearly,
Revised: Clearly,

Original: [Equation]
Revised: [Equation]

Original: Since ψ α, λ , w ( z , β ) is 1 α -strongly convex in z , and that β
Revised: Since ψ α, λ , w ( z , β ) is 1 α -strongly convex in z , and that β

Original: + :=
Revised: + :=

Original: MPG ( β ; λ , w , α ) is its optimal point,
Revised: MPG ( β ; λ , w , α ) is its optimal point,

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Taking z = β + (cid:48) in Eq.
Revised: Taking z = β + (cid:48) in Eq.

Original: 36 and z = β + in Eq.
Revised: 37 and z = β + in Eq.

Original: 37 yields
Revised: 38 yields

Original: [Equation]
Revised: [Equation]

Original: Summing the above two inequalities
Revised: Summing the above two inequalities

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Lemma E.5 (Robustness to λ ) .
Revised: Lemma F.5 (Robustness to λ ) .

Original: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)
Revised: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)

Original: =
Revised: =

Original: MPG ( β ; λ (cid:48) , w , α ) .
Revised: MPG ( β ; λ (cid:48) , w , α ) .

Original: Then
Revised: Then

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 38) where S :
Revised: where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) ) [Equation]
Revised: = supp( β + ) ∪ supp( β + (cid:48) ) .

Original: Proof.
Revised: Proof.

Original: Recall the quadratic approximation in Eq.
Revised: Recall the quadratic approximation in Eq.

Original: 50 and the deﬁnition that MPG ( β ; λ , w , α ) =
Revised: 51 and the deﬁnition that MPG ( β ; λ , w , α ) =

Original: arg min
Revised: arg min

Original: z ψ α, λ , w ( z , β ) .
Revised: z ψ α, λ , w ( z , β ) .

Original: By the 1 α -strongly convexity of ψ α, λ , w in z , it holds true for all z that
Revised: By the 1 α -strongly convexity of ψ α, λ , w in z , it holds true for all z that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking z = β + (cid:48) in Eq. 39, taking z = β + in Eq.
Revised: Taking z = β + (cid:48) in Eq. 40, taking z = β + in Eq.

Original: [Equation]
Revised: [Equation]

Original: and summing up the two inequalities, we have
Revised: and summing up the two inequalities, we have

Original: [Equation]
Revised: [Equation]

Original: Assumption B.1)

Original: [Equation]

Original: where S :
Revised: where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) ) .
Revised: = supp( β + ) ∪ supp( β + (cid:48) ) .

Original: Dividing both side by (cid:107) β + (cid:48)
Revised: Dividing both side by (cid:107) β + (cid:48)

Original: − β + (cid:107) 2 draws the conclusion.
Revised: − β + (cid:107) 2 draws the conclusion.

Original: Lemma E.6 (Robustness to w ) .
Revised: Lemma F.6 (Robustness to w ) .

Original: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)
Revised: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)

Original: =
Revised: =

Original: MPG ( β ; λ , w (cid:48) , α ) .
Revised: MPG ( β ; λ , w (cid:48) , α ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where S :
Revised: where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) ) and C is some absolute constant.
Revised: = supp( β + ) ∪ supp( β + (cid:48) ) and C is some absolute constant.

Original: Proof.
Revised: Proof.

Original: Recall the quadratic approximation in Eq.
Revised: Recall the quadratic approximation in Eq.

Original: 50 and the deﬁnition that MPG ( β ; λ , w , α ) =
Revised: 51 and the deﬁnition that MPG ( β ; λ , w , α ) =

Original: arg min
Revised: arg min

Original: z ψ α, λ , w ( z , β ) .
Revised: z ψ α, λ , w ( z , β ) .

Original: By the 1 α -strongly convexity of ψ α, λ , w in β , it holds true for all z in the restricted subspace that
Revised: By the 1 α -strongly convexity of ψ α, λ , w in β , it holds true for all z in the restricted subspace that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Taking z = β + (cid:48) in Eq.
Revised: Taking z = β + (cid:48) in Eq.

Original: [Equation]
Revised: [Equation]

Original: taking z = β + in Eq.
Revised: taking z = β + in Eq.

Original: [Equation]
Revised: [Equation]

Original: and summing up the two inequalities
Revised: and summing up the two inequalities

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that q (cid:48) w ( λ j , β j )
Revised: Note that q (cid:48) w ( λ j , β j )

Original: = (cid:80) 

Revised: = (cid:80)

Original: i =1 exp( w i )
Revised: i =1 exp( w i )

Original: Z ( w ) q
Revised: Z ( w ) q

Original: ( i ) (cid:48) ( λ j , β j ) where Z ( w ) =
Revised: ( i ) (cid:48) ( λ j , β j ) where Z ( w ) =

Original: (cid:80) 3 i =1 exp( w i ) .
Revised: (cid:80) 3 i =1 exp( w i ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: 3 (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )
Revised: 3 (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )

Original: Z ( w ) − exp( w (cid:48) i )
Revised: Z ( w ) − exp( w (cid:48) i )

Original: Z ( w (cid:48) ) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) q
Revised: Z ( w (cid:48) ) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) q

Original: ( i ) (cid:48) ( λ j , β j ) (cid:12)(cid:12)(cid:12) (by Assumption B.1) ≤ 3 (cid:88)
Revised: ( i ) (cid:48) ( λ j , β j ) (cid:12)(cid:12)(cid:12) (by Assumption B.1) ≤ 3 (cid:88)

Original: i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )
Revised: i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )

Original: Z ( w ) − exp( w (cid:48) i )
Revised: Z ( w ) − exp( w (cid:48) i )

Original: Z ( w (cid:48) ) (cid:12)(cid:12)(cid:12)(cid:12) λ j .
Revised: Z ( w (cid:48) ) (cid:12)(cid:12)(cid:12)(cid:12) λ j .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: for some constant C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This section supplies some fundamental results for the modiﬁed proximal gradient algorithm.
Revised: This section supplies some fundamental results for the modiﬁed proximal gradient algorithm.

Original: All the results in this section are based on the following assumption and notations.
Revised: All the results in this section are based on the following assumption and notations.

Original: Assumption F.1.
Revised: Assumption G.1.

Original: Assume P satisﬁes Assumption 3.1 and consider a problem ( Z n , β ∗ ) ∈ P .
Revised: Assume P satisﬁes Assumption C [Equation] 1 and consider a problem ( Z n , β ∗ ) ∈ P .

Original: Assume we have a penalty function P ( λ , β ) whose concave component Q ( λ , β )
Revised: Assume we have a penalty function P ( λ , β ) whose concave component Q ( λ , β )

Original: =
Revised: =

Original: P ( λ , β ) −(cid:107) λ ◦ β (cid:107)satisﬁes the regularization conditions in Assumption B
Revised: P ( λ , β ) −(cid:107) λ ◦ β (cid:107)satisﬁes the regularization conditions in Assumption B

Original: [Equation]
Revised: [Equation]

Original: 1 with a Lipschitz constant ξ for the condition (a).
Revised: 1 with a Lipschitz constant ξ for the condition (a).

Original: We adopt the following notations for the statements in this sections.
Revised: We adopt the following notations for the statements in this sections.

Original: [Equation]
Revised: [Equation]

Original: Lemma F.1 (Contraction of MPG) .
Revised: Lemma G.1 (Contraction of MPG) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G.1.

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:

Original: [Equation]
Revised: [Equation]

Original: Assume (cid:107) ( β k ) S ∗ (cid:107) 

Revised: Assume (cid:107) ( β k ) S ∗ (cid:107)

Original: ≤ ˜ s .
Revised: ≤ ˜ s .

Original: Then(cid:18)
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where δ α ( β ) :
Revised: where δ α ( β ) :

Original: =
Revised: =

Original: α − 1 ( β − T α λ ( β − α ∇ L λ ( β ))) .
Revised: α − 1 ( β − T α λ ( β − α ∇ L λ ( β ))) .

Original: It is easy to observe that δ α ( β ) ∈ ∇ L λ ( β [Equation] + λ ◦ ∂ [Equation]
Revised: It is easy to observe that δ α ( β ) ∈ ∇ L λ ( β ) + λ ◦ ∂ (cid:107) β − α δ α ( β ) (cid:107) 1 .

Original: β − α δ α

Original: [Equation]

Original: β )

Original: [Equation]

Original: By RSS in Lemma B.1, it holds that
Revised: By RSS in Lemma B.1, it holds that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by RSC in Lemma B.1, for any z in the space { (cid:13)(cid:13)
Revised: Furthermore, by RSC in Lemma B.1, for any z in the space { (cid:13)(cid:13)

Original: z − β k − 1 (cid:13)(cid:13) [-
-]
Revised: z − β k [Equation] 1 (cid:13)(cid:13) [-
-][Equation]

Original: ≤ s ∗ + 2˜ s } ,
Revised: ≤ s ∗ + 2˜ s } ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Denote v = δ α ( β k − 1 ) − ∇ L λ ( β k − 1 ) .
Revised: Denote v = δ α ( β k − 1 ) − ∇ L λ ( β k − 1 ) .

Original: We have v ∈ λ ◦ ∂ (cid:13)(cid:13)
Revised: We have v ∈ λ ◦ ∂ (cid:13)(cid:13)

Original: β k − 1 − α δ α ( β k − 1 ) (cid:13)(cid:13) 1 since δ α ( β k −
Revised: β k − 1 − α δ α ( β k − 1 ) (cid:13)(cid:13) 1 since δ α ( β k −

Original: [Equation]
Revised: [Equation]

Original: ∈ ∇ L λ ( 
Revised: ∈ ∇ L λ ( β

Original: [Equation]
Revised: [Equation]

Original: + λ
Revised: + λ

Original: [Equation]
Revised: [Equation]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: β k − 1 − α δ α
Revised: β k − 1 − α δ α

Original: ( β k − 1 ) (cid:13)(cid:13) 1 .
Revised: ( β k − [Equation] (cid:13)(cid:13) 1 .

Original: By the convexity of (cid:107)·(cid:107) λ , 1 ,(cid:13)(cid:13)
Revised: By the convexity of (cid:107)·(cid:107) λ , 1 ,

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq.
Revised: Combining this inequality with Eq.

Original: 47, we have
Revised: 48, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking z = β k − 1 in Eq.
Revised: Taking z = β k − 1 in Eq.

Original: 49 implies that
Revised: 50 implies that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.2 (Convergence of modiﬁed proximal gradient) .
Revised: Lemma G.2 (Convergence of modiﬁed proximal gradient) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G.

Revised: [Equation]

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:

Original: [Equation]
Revised: [Equation]

Original: with a step size α in (0 , 1 ρ + ] .
Revised: with a step size α in (0 , 1 ρ + ] .

Original: Assume (cid:107) ( β k ) S ∗ (cid:107) 0 ≤ ˜ s .
Revised: Assume (cid:107) ( β k ) S ∗ (cid:107) 0 ≤ ˜ s .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the quadratic approximation function
Revised: Consider the quadratic approximation function

Original: [Equation]
Revised: [Equation]

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Since ψ α, λ ( z , β k − 1 ) is 1 α -strongly convex with respect to z
Revised: Since ψ α, λ ( z , β k − 1 ) is 1 α -strongly convex with respect to z

Original: [Equation]
Revised: [Equation]

Original: and that β k is its optimal point
Revised: and that β k is its optimal point

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Note that by Lemma B.1,
Revised: Note that by Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Similarly, by Lemma B.1,
Revised: Similarly, by Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two results with Eq.
Revised: Combining the above two results with Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Taking z = c (cid:98) β λ + (1 − c ) β k − 1 for some c ∈ [0 , 1] , then φ λ ( β k ) ≤
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Taking c
Revised: Taking c

Original: = α ( ρ− − ξ ) , it implies φ [Equation] − [Equation]
Revised: = α ( ρ

Original: [Equation]
Revised: [Table]

Original: Assume α ≤ 1 ρ + .

Original: Then [Equation] Taking z = (cid:98) β λ in Eq. 52 impliesφ λ
Revised: − ξ ) , it implies

Original: [Equation]
Revised: [Table]

Original: β [Equation] ) ≤φ [Equation]
Revised: Assume α ≤

Original: +

Original: [Equation]
Revised: [Table]

Original: α − [Equation] ρ − − ξ ) (cid:107) (cid:98) β λ − β
Revised: Then

Original: [Equation]
Revised: [Table]

Original: − 1 (cid:107) 22 − 1 2 α (cid:107)

Original: [Equation]
Revised: [Equation]

Original: By the optimality of (cid:98) β λ , we have
Revised: By the optimality of (cid:98) β λ , we have

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we derive the upper bound for ω λ ( β k ) .
Revised: we derive the upper bound for ω λ ( β k ) .

Original: Since β k = arg min z ψ α, λ ( z , β k − 1 ) , by the optimality condition, for each j = 1 , · · · , d [Equation] there exists a subgradient ξ j [Equation]
Revised: Since β k = arg min z ψ α, λ ( z , β k − 1 ) , by the optimality condition, for each j = 1 , · · · , d , there exists a subgradient ξ j ∈ ∂ (cid:12)(cid:12)

Original: ∂ (cid:12)(cid:12) β kj (cid:12)(cid:12) such that
Revised: β kj (cid:12)(cid:12) such that

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of ω λ and combining it with the above equality,
Revised: By the deﬁnition of ω λ and combining it with the above equality,

Original: [Equation]
Revised: [Equation]

Original: by RSS)≤
Revised: by RSS)

Original: [Equation]
Revised: [Equation]

Original: By Lemma F.1,
Revised: By Lemma G.1,

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]

Original: − β k )

Original: [Equation]
Revised: [Equation]

Original: k 
Revised: k )

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.3 (Statistical (cid:96) 2 error) .
Revised: Lemma G.3 (Statistical (cid:96) 2 error) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G

Revised: [Equation]

Original: If the following conditions are satisﬁed
Revised: If the following conditions are satisﬁed

Original: [Equation]
Revised: [Equation]

Original: then 
Revised: then (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the restricted strong convexity of L λ in Lemma B.1,
Revised: By the restricted strong convexity of L λ in Lemma B.1,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the term (i),
Revised: For the term (i),

Original: [Equation]
Revised: [Equation]

Original: S ∗ , 1 −
Revised: ∗ , 1 −

Original: [Equation]
Revised: [Equation]

Original: S [Equation] (cid:107) λ S ∗ ,= (cid:107) ( β − β ∗ )
Revised: S ∗ (cid:107) λ S ∗ ,= (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 − (cid:107) ( β − β ∗ )
Revised: S ∗ , 1 − (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 .
Revised: S ∗ , 1 .

Original: For the term (ii), −∇
Revised: For the term (ii), −∇

Original: [Equation]
Revised: [Equation]

Original: −∇
Revised: β

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Revised: [Equation]

Revised: −∇

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: ( β − β ∗ )
Revised: β − β ∗ )

Original: (By Eq. 55) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ ) (cid:62) ( β − β ∗ ) (since q (cid:48)
Revised: [Equation]

Original: λ j (0) = 0 )
Revised: λ ( β ∗

Original:  = 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ S ∗ ) (cid:62) ( β − β ∗ )
Revised: [Equation] ( β − β ∗ ) (By Eq. 56) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ ) (cid:62) ( β − β ∗ ) (since q (cid:48) λ j (0) = 0 ) = 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ S ∗ ) (cid:62) ( β − β ∗ )

Original: S ∗ (since (cid:12)(cid:12)(cid:12) q (cid:48)
Revised: S ∗ (since (cid:12)(cid:12)(cid:12) q (cid:48)

Original: λ j ( β ) (cid:12)(cid:12)(cid:12) ≤ λ j ) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 + (cid:107) ( β − β ∗ )
Revised: λ j ( β ) (cid:12)(cid:12)(cid:12) ≤ λ j ) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 + (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ S ∗ ,=8 (cid:107) ( β − β ∗ )
Revised: S ∗ (cid:107) λ S ∗ ,=8 (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 + 1 8 (cid:107) ( β − β ∗ )
Revised: S ∗ , 1 + 1 8 (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 .
Revised: S ∗ , 1 .

Original: Combining the upper bounds of term (i) and term (ii) with Eq.
Revised: Combining the upper bounds of term (i) and term (ii) with Eq.

Original: 56, we have
Revised: 57, we have

Original: [Equation]
Revised: [Equation]

Original: Let a ≥ 0 be a constant.
Revised: Let a ≥ 0 be a constant.

Original: If (cid:107) ( β − β ∗ ) S ∗(cid:107) λ S ∗ , 1 ≤
Revised: If (cid:107) ( β − β ∗ ) S ∗

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ∗(cid:107) 22 , then
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: 1 ≤ [Equation] 22 , then

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( [Equation] [Equation] ∗ (cid:107) λ

Original: S ∗

Original: [Equation]
Revised: [Equation]

Original: 1 >
Revised: 1 >

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:107) β − β ∗ (cid:107) 2 ≤
Revised: which implies (cid:107) β − β ∗ (cid:107) 2 ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: + 174 (cid:1)
Revised: + 174 (cid:1)

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: ξ .
Revised: ξ .

Original: Combining the two cases, we know that
Revised: Combining the two cases, we know that

Original: [Equation]
Revised: [Equation]

Original: Taking a = 7 it is easy to derive that
Revised: Taking a = 7 it is easy to derive that

Original: [Equation]
Revised: [Equation]

Original: Lemma F.4 (Retain in the restricted space) .
Revised: Lemma G.4 (Retain in the restricted space) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G.1.

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the following update:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the following update:

Original: [Equation]
Revised: [Equation]

Original:  with a step sizes α in [ α min , 1 ρ + ] .
Revised: ) with a step sizes α in [ α min , 1 ρ + ] .

Original: Assume the following conditions:
Revised: Assume the following conditions:

Original: [Equation]
Revised: [Equation]

Original: where κ :=
Revised: where κ :=

Original: ρ + ρ − − ξ .
Revised: ρ + ρ − − ξ .

Original: Then for all k = 1 , · · · , K ,
Revised: Then for all k = 1 , · · · , K ,

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original:  S
Revised: ) S

Original: [Equation]
Revised: [Equation]

Original:  ≤ ˜ s, and
Revised: 0 ≤ ˜ s, and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that β k = T α λ (cid:0) ¯ β k (cid:1) where
Revised: Recall that β k = T α λ (cid:0) ¯ β k (cid:1) where

Original: [Equation]
Revised: [Equation]

Revised: To show (cid:13)(cid:13)

Original: To show (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13) 0 ≤ ˜ s , we need to prove that, for j ∈ S ∗ , the number of j ’s such that | ¯ β kj | > αλ j is no more than ˜ s .
Revised: [Equation] β k ) S [Equation] (cid:13)(cid:13) 0 ≤ ˜ s , we need to prove that, for j ∈ S ∗ , the number of j ’s such that | ¯ β kj | > αλ j is no more than ˜ s .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, ∀ c ∈ [0 , 1] ,
Revised: Therefore, ∀ c ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: 59) where Card () represents the size of the set.
Revised: + [Equation] [Equation] where Card () represents the size of the set.

Original: For the term in Eq.
Revised: For the term in Eq.

Original: Card
Revised: Card

Original: [Equation]
Revised: [Equation]

Original: Bounding (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) 1 .
Revised: Bounding (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) 1 .

Original: Following the derivation of Eq.
Revised: Following the derivation of Eq.

Original: 57, we can obtain
Revised: 58, we can obtain

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Let a ≥ 0 be a constant.
Revised: Let a ≥ 0 be a constant.

Original: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ

Original: S ∗
Revised: S ∗

Original: [Equation]
Revised: [Equation]

Original: ≤ a ρ − − ξ (cid:107) λ
Revised: ≤ a ρ − − ξ (cid:107) λ

Original: S ∗ (cid:107) 22 , then
Revised: S ∗ (cid:107) 22 , then

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ

Original: S ∗ , 1 > a ρ − − ξ (cid:107) λ
Revised: S ∗ , 1 > a ρ − − ξ (cid:107) λ

Original: S ∗ (cid:107) 22 , then
Revised: S ∗ (cid:107) 22 , then

Revised: [Equation]

Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases yields
Revised: Combining the above two cases yields

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is obtained by taking a = 25534 .
Revised: where the last inequality is obtained by taking a = 25534 .

Original: Therefore, the term in Eq.
Revised: Therefore, the term in Eq.

Original: 58 can be bounded by
Revised: 59 can be bounded by

Original: [Equation]
Revised: [Equation]

Original: Now we bound the term in Eq.
Revised: Now we bound the term in Eq.

Original: 59 as follows
Revised: 60 as follows

Original: [Equation]
Revised: [Equation]

Original: Denote the set as S (cid:48) :
Revised: Denote the set as S (cid:48) :

Original: = { j ∈ S ∗ :
Revised: = { j ∈ S ∗ :

Original: [Equation]
Revised: [Equation]

Original: > (1 − c ) · 78 λ j } and its size as s (cid:48) := | S (cid:48) | .
Revised: > (1 − c ) · 78 λ j } and its size as s (cid:48) := | S (cid:48) | .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds because L λ is ( ρ + ) -smooth in the restricted subspace by Lemma B.1.
Revised: The last inequality holds because L λ is ( ρ + ) -smooth in the restricted subspace by Lemma B.1.

Original: Applying Lemma F. [Equation] we have
Revised: Applying Lemma G.3, we have

Original: [Equation]
Revised: [Equation]

Original: Combining the above bounds for the terms in Eq.
Revised: Combining the above bounds for the terms in Eq.

Original: 58 and Eq.
Revised: 59 and Eq.

Revised: 60, we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: 49 cα

Original: [Equation]
Revised: [Equation]

Original: 49 cα (cid:19) ρ − − ξ + 49(1 − c )(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ (cid:32) α min ( ρ − − ξ ) +(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ ˜ s
Revised: [Equation] [Equation] ≤ ˜ s

Original: The second last inequality is obtained by taking c = 57 .
Revised: The second last inequality is obtained by taking c = 57 .

Original: Finally, if we can show φ λ ( β k ) − φ λ ( β ∗ ) ≤ 21 / 2 ρ − − ξ (cid:107) λ S ∗ (cid:107) 22 , then we can show by math induction that (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13) 

Revised: Finally, if we can show φ λ ( β k ) − φ λ ( β ∗ ) ≤ 21 / 2 ρ − − ξ (cid:107) λ S ∗ (cid:107) 22 , then we can show by math induction that (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13)

Original: ≤ ˜ s for all k .
Revised: ≤ ˜ s for all k .

Original: Taking z = β k − 1 in Eq.
Revised: Taking z = β k − 1 in Eq.

Original: 52, we have
Revised: 53, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.5 (Statistical error) .
Revised: Lemma G.5 (Statistical error) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G.1.

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then the following relations hold true.
Revised: then the following relations hold true.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since (cid:107) β − β ∗ (cid:107) 

Revised: Since (cid:107) β − β ∗ (cid:107)

Original: ≤ s ∗ + 2˜ s , then by the RSC and RSS in Lemma B.1,
Revised: ≤ s ∗ + 2˜ s , then by the RSC and RSS in Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ ( β ).
Revised: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ ( β )

Original: Adding and

Original: ( β − β ∗ ) (cid:62) ( ξ ) to the right-hand side of Eq.

Original: 60 yields

Original: [Equation]
Revised: [Equation]

Original: Now we bound the terms (i)-(iii).
Revised: Now we bound the terms (i)-(iii).

Original: Bound (i) using ω λ ( β ) .
Revised: Bound (i) using ω λ ( β ) .

Original: Since ξ attains the minimum in ω λ ( β ) , by deﬁnition,
Revised: Since ξ attains the minimum in ω λ ( β ) , by deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Bound (ii).
Revised: Bound (ii).

Original: Note that L λ =
Revised: Note that L λ =

Original: L n + Q λ .
Revised: L n + Q λ .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Since λ j ≥ 8 |
Revised: Since λ j ≥ 8 |

Original: [ L n ( β ∗ )]
Revised: [ L n ( β ∗ )]

Original: j | , we have
Revised: j | , we have

Original: [Equation]
Revised: [Equation]

Original: Since q (cid:48) λ j (0) = 0 and q (cid:48) λ j ( β ) ≤ λ j (Assumption B.1), it holds true that
Revised: Since q (cid:48) λ j (0) = 0 and q (cid:48) λ j ( β ) ≤ λ j (Assumption B.1), it holds true that

Original: [Equation]
Revised: [Equation]

Original: Therefore, |
Revised: Therefore, |

Original: [Equation]
Revised: [Equation]

Original: ii
Revised: ii

Original: [Equation]
Revised: [Equation]

Original: Bound (iii) by the deﬁnition of subgradient.
Revised: Bound (iii) by the deﬁnition of subgradient.

Original: [Equation]
Revised: [Equation]

Original: By Holder’s inequality and the fact
Revised: By Holder’s inequality and the fact

Original: [Equation]
Revised: [Equation]

Original: Since ξ ∈ ∂ (cid:107) β (cid:107) 1 and that β ∗ S ∗ = 0 ,
Revised: Since ξ ∈ ∂ (cid:107) β (cid:107) 1 and that β ∗ S ∗ = 0 ,

Original: [Equation]
Revised: [Equation]

Original: Combining the above three equations, it holds true that
Revised: Combining the above three equations, it holds true that

Original: [Equation]
Revised: [Equation]

Original: Combine (i)-(iii).
Revised: Combine (i)-(iii).

Original: Combining the bounds for (i)-(iii), it holds true that
Revised: Combining the bounds for (i)-(iii), it holds true that

Original: [Equation]
Revised: [Equation]

Original: Using the fact that (cid:107) ( β − β [Equation] S ∗ (cid:107) λ S ∗ , 1 ≤ (cid:107) λ
Revised: Using the fact that (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ S ∗ , 1 ≤ (cid:107) λ

Original: S ∗ (cid:107) 2 (cid:107) ( β − β ∗ ) S ∗ (cid:107) 2 , it implies the ﬁrst conclusion:
Revised: S ∗ (cid:107) 2 (cid:107) ( β − β ∗ ) S ∗ (cid:107) 2 , it implies the ﬁrst conclusion:

Original: [Equation]
Revised: [Equation]

Original: Now we prove the objective value.
Revised: Now we prove the objective value.

Original: Since L λ is convex on β , β ∗ and (cid:107)·(cid:107) 1 is convex,
Revised: Since L λ is convex on β , β ∗ and (cid:107)·(cid:107) 1 is convex,

Original: [Equation]
Revised: [Equation]

Original: , which implies
Revised: , which implies

Original: [Equation]
Revised: [Equation]

Original: Now we bound the norm (cid:107) β − β ∗ (cid:107) λ , 1 as follows.
Revised: Now we bound the norm (cid:107) β − β ∗ (cid:107) λ , 1 as follows.

Original: By triangle inequality,
Revised: By triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: To bound (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ S ∗ , 1 , moving the term (cid:107) ( β [Equation] ) S 
Revised: To bound (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ S ∗ , 1 , moving the term (cid:107) ( β − β ∗ ) S ∗

Original: [Equation]
Revised: [Equation]

Original: S [Equation] , 1 in Eq.
Revised: S ∗ , 1 in Eq.

Original: 62 to the left hand side yields
Revised: 63 to the left hand side yields

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging it into Eq. 65, we have
Revised: Plugging it into Eq. 66, we have

Original: [Equation]
Revised: [Equation]

Original: (by Eq. 63) ≤
Revised: (by Eq. 64) ≤

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: Combining it with Eq.
Revised: Combining it with Eq.

Original: 64, we have
Revised: 65, we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we derive the bound for the term
Revised: Finally, we derive the bound for the term

Original: φ λ ( β ) − φ λ ( (cid:98) β λ ) .
Revised: φ λ ( β ) − φ λ ( (cid:98) β λ ) .

Original: Since (cid:107) β − (cid:98) β λ (cid:107)[-
-]
Revised: Since (cid:107) β − (cid:98) β λ (cid:107)[-
-]

Original:  ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then[Equation]
Revised: [Equation] ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then

Original: [Equation]
Revised: [Equation]

Original: Now we only need to bound the term (cid:107) β ∗ − (cid:98)
Revised: Now we only need to bound the term (cid:107) β ∗ − (cid:98)

Original: β λ (cid:107) λ , 1 .
Revised: β λ (cid:107) λ , 1 .

Original: We can derive its bound following the same steps as we derive Eq.
Revised: We can derive its bound following the same steps as we derive Eq.

Original: 67 and using the fact that ω λ ( (cid:98) β λ ) = 0 , which will imply that
Revised: and using the fact that ω λ ( (cid:98) β λ ) = 0 , which will imply that

Original: [Equation]
Revised: [Equation]

Original: Plugging it into Eq.
Revised: Plugging it into Eq.

Original: 68, we have
Revised: 69, we have

Original: [Equation]
Revised: [Equation]

Original: Assuming (cid:15) ≤ 1 / 2 , then
Revised: Assuming (cid:15) ≤ 1 / 2 , then

Original: [Equation]
Revised: [Equation]

Original: Lemma F.6 (Optimality when λ decreases) .
Revised: Lemma G.6 (Optimality when λ decreases) .

Original: Assume Assumption F.1.
Revised: Assume Assumption G.1.

Original: If
Revised: If

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with η j

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: with η j ∈ [0 . 9 , 1] , (71) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 71) then
Revised: (72) then

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ρ − − ξ .
Revised: ( λ t ) S ∗ (cid:107)ρ − − ξ .

Original: If (cid:15) t − 1 ≤ 1 / 
Revised: If (cid:15) t − 1 ≤ 1 / 4 , then

Original: [Equation]
Revised: [Equation]

Original: then
Revised: − −

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the deﬁnition of ω λ t − 1 ( β ) .
Revised: Recall the deﬁnition of ω λ t − 1 ( β ) .

Original: [Equation]
Revised: [Equation]

Original: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ t − 1 ( β ) .
Revised: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ t − 1 ( β ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Now we consider λ t =
Revised: Now we consider λ t =

Original: η ◦ λ t − 1 .
Revised: η ◦ λ t − 1 .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Using the fact that (cid:107) β − β (cid:48) (cid:107) λ t , 1 ≥ 0 .
Revised: Using the fact that (cid:107) β − β (cid:48) (cid:107) λ t , 1 ≥ 0 .

Original: 9 (cid:107) β − β (cid:48)
Revised: 9 (cid:107) β − β (cid:48)

Original: (cid:107) λ t − 1 , 1 , term (i) can be bounded as follows.
Revised: (cid:107) λ t − 1 , 1 , term (i) can be bounded as follows.

Original: [Equation]
Revised: [Equation]

Original: Using the fact | ξ j | ≤ 1 and the fact that [ λ t − 
Revised: Using the fact | ξ j | ≤ 1 and the fact that [ λ t − 1 − λ t ] j

Original: [Equation]
Revised: [Equation]

Original: [ λ
Revised: η j

Original: [Equation]
Revised: [Equation]

Revised: 9 −

Original: [Equation]
Revised: [Equation]

Original:  , term (ii) can be bounded as follows.
Revised: [ λ t ] j , term (ii) can be bounded as follows.

Original: [Equation]
Revised: [Equation]

Original: (cid:18)0 . 9 −(cid:19)
Revised: (cid:18)0 . 9 −(cid:19)

Original: =0 .
Revised: =0 .

Original: 9 − 1 .
Revised: 9 − 1 .

Original: Term (iii) is bounded similarly as term (ii).
Revised: Term (iii) is bounded similarly as term (ii).

Original: Since (cid:12)(cid:12)(cid:12) q λ j ( β ) − q λ (cid:48)
Revised: Since (cid:12)(cid:12)(cid:12) q λ j ( β ) − q λ (cid:48)

Original: j ( β ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12) λ
Revised: j ( β ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12) λ

Original: j − λ (cid:48)
Revised: j − λ (cid:48)

Original: j (cid:12)(cid:12) , we have (iii) ≤
Revised: j (cid:12)(cid:12) , we have (iii) ≤

Original: [Equation]
Revised: [Equation]

Original: j =

Original: [Equation]

Original: [ λ
Revised: [ λ

Original: [Equation][Equation] [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining the bounds for (i)-(iii), we have
Revised: Combining the bounds for (i)-(iii), we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst part of the proof is ﬁnished.
Revised: The ﬁrst part of the proof is ﬁnished.

Original: Now we bound the term φ λ t ( β ) − φ λ t ( (cid:98) β λ t ) .
Revised: Now we bound the term φ λ t ( β ) − φ λ t ( (cid:98) β λ t ) .

Original: Since (cid:107) β − (cid:98) β λ (cid:107) 

Revised: Since (cid:107) β − (cid:98) β λ (cid:107)

Original: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then
Revised: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then

Original: [Equation]
Revised: [Equation]

Original: ) ≤ ω λ [Equation] β ) [Equation]
Revised: Following the same way we derive Eq.

Original: β − [Equation] Following the same way we derive Eq. 66, we can obtain that
Revised: 67, we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using the same arguments for the term (cid:107) β ∗ − (cid:98) β λ t (cid:107) λ t , 1 and using the fact that ω λ t ( (cid:98) β λ t ) ≤ 0 · λ t , we obtain that
Revised: Using the same arguments for the term (cid:107) β ∗ − (cid:98) β λ t (cid:107) λ t , 1 and using the fact that ω λ t ( (cid:98) β λ t ) ≤ 0 · λ t , we obtain that

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq.
Revised: Combining this inequality with Eq.

Original: 72 and Eq.
Revised: 73 and Eq.

Original: 74, we have
Revised: 75, we have

Original: [Equation]
Revised: [Equation]

Original: Assuming (cid:15) t − 1 ≤ 1 / 4 , then
Revised: Assuming (cid:15) t − 1 ≤ 1 / 4 , then

Original: [Equation]
Revised: [Equation]

Original: Lemma F.7 (Coercivity of the gradient) .
Revised: Lemma G.7 (Coercivity of the gradient) .

Original: [Lemma 3.11 in Bubeck (2014)] Let f be a -strongly convex b -smooth and on B ⊆ R d .
Revised: [Lemma 3.11 in Bubeck (2014)] Let f be a -strongly convex b -smooth and on B ⊆ R d .

Original: Then for all β , β (cid:48) ∈ B , one has
Revised: Then for all β , β (cid:48) ∈ B , one has

Original: [Equation]
Revised: [Equation]

Original: β

Original: [Equation]

Original: ) , β − β
Revised: ) , β − β

Original: [Equation]
Revised: [Equation]

Original: G D ETAILS OF S ECTION 4.1: E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING
Revised: H D ETAILS OF S ECTION 5: E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING

Original: In addition to the assumptions in the supervised setting, in this unsupervised setting, we assume for each estimation problem, both L n 1 ( Z 1: n 1 ) and L n 2 ( Z 1: n 2 ) satisfy condition (b) and (c) in Assumption 3.1.
Revised: In addition to the assumptions in the supervised setting, in this unsupervised setting, we assume for each estimation problem, both L n 1 ( Z 1: n 1 ) and L n 2 ( Z 1: n 2 ) satisfy condition (b) and (c) in Assumption C.1.

Original: Similar to the supervised setting, the generalization error of the optimizer θ ∗ U can be bounded by
Revised: Similar to the supervised setting, the generalization error of the optimizer θ ∗ U can be bounded by

Original: [Equation]
Revised: [Equation]

Original: However, in the unsupervised setting, the minimizer θ ∗ U only guarantees that unsupervised training loss function 1 m(cid:80) mi =
Revised: However, in the unsupervised setting, the minimizer θ ∗ U only guarantees that unsupervised training loss function 1 m

Original: [Equation]
Revised: [Equation]

Original: L n 2 (cid:16) Z ( i ) 1: n 2 , β T ( Z ( i ) 1: n 1 ; θ ∗ U ) (cid:17) is small.
Revised: [Equation] Z ( i ) 1: n 2 , β T ( Z ( i ) 1: n 1 ; θ ∗ U ) (cid:17) is small.

Original: It requires further derivations to show that the corresponding supervised training loss in Eq.
Revised: It requires further derivations to show that the corresponding supervised training loss in Eq.

Original: 76 is small, too.
Revised: 77 is small, too.

Original: Therefore, in the following, we bridge the gap between the supervised training loss and the unsupervised training loss.
Revised: Therefore, in the following, we bridge the gap between the supervised training loss and the unsupervised training loss.

Original: Based on the proof of Theorem C.1, we know that (cid:107) β T ( Z ( i ) 1: n 1 ; θ ∗ U ) | S ∗ (cid:107) 

Revised: Based on the proof of Theorem D.1, we know that (cid:107) β T ( Z ( i ) 1: n 1 ; θ ∗ U ) | S ∗ (cid:107)

Original: ≤ ˜ s .
Revised: ≤ ˜ s .

Original: Therefore, we can apply the restricted strong convexity of L n 2 to obtain the following inequality for β T = β T ( Z ( i ) 1: n 1 ; θ ∗ U ) , β ∗ = β ∗ ( i ) , and Z =
Revised: Therefore, we can apply the restricted strong convexity of L n 2 to obtain the following inequality for β T = β T ( Z ( i ) 1: n 1 ; θ ∗ U ) , β ∗ = β ∗ ( i ) , and Z =

Original: Z ( i ) 1: n 2 for each i = 1 , · · · , m .
Revised: Z ( i ) 1: n 2 for each i = 1 , · · · , m .

Original: [Equation]
Revised: [Equation]

Original: Aggregating this inequality for i = 1 , · · · , m , we can obtain the following inequality that bounds the supervised training loss using the unsupervised loss.
Revised: Aggregating this inequality for i = 1 , · · · , m , we can obtain the following inequality that bounds the supervised training loss using the unsupervised loss.

Original: [Equation]
Revised: [Equation]

Original: The right-hand-side of this inequality consists of a statistical error and the unsupervised training error.
Revised: The right-hand-side of this inequality consists of a statistical error and the unsupervised training error.

Original: Finally, to characterize how small the unsupervised training error can be [Equation] we can combine the following inequality (by restricted strongly smooth) with the result of Theorem 3.1.
Revised: Finally, to characterize how small the unsupervised training error can be, we can combine the following inequality (by restricted strongly smooth) with the result of Theorem 3.1.

Original: [Equation]
Revised: [Equation]

Original: Overall,the generalization error can be bounded by
Revised: Overall,

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation]
Revised: can

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: H D ETAILS OF S YNTHETIC E XPERIMENTS H. A G ENTLE R EVIEW OF L INEAR R EGRESSION AND P RECISION E STIMATION H.1. S PARSE L INEAR R EGRESSION
Revised: I D ETAILS OF S YNTHETIC E XPERIMENTS I. A G ENTLE R EVIEW OF L INEAR R EGRESSION AND P RECISION E STIMATION I.1. S PARSE L INEAR R EGRESSION

Original: A sparse linear regression model reads
Revised: A sparse linear regression model reads

Original: [Equation]
Revised: [Equation]

Original: where β ∗ is a sparse vector and (cid:15) is Gaussian noise.
Revised: where β ∗ is a sparse vector and (cid:15) is Gaussian noise.

Original: Given n observations Z 1: n := { ( x 1 , y 1 ) , · · · , ( x n , y n ) } , the goal is to estimate the vector β ∗ .
Revised: Given n observations Z 1: n := { ( x 1 , y 1 ) , · · · , ( x n , y n ) } , the goal is to estimate the vector β ∗ .

Original: In our method, we use the the least-square error to deﬁne the empirical loss function.
Revised: In our method, we use the the least-square error to deﬁne the empirical loss function.

Original: [Equation]
Revised: [Equation]

Revised: I.

Revised: [Equation]

Revised: [Equation]

Original: H.1. S PARSE P RECISION E STIMATION
Revised: S PARSE P RECISION E STIMATION

Original: The sparse precision matrix estimation problem in Gaussian graphical models assumes the observation of n samples from a distribution N (0 , Σ) .
Revised: The sparse precision matrix estimation problem in Gaussian graphical models assumes the observation of n samples from a distribution N (0 , Σ) .

Original: Given the n samples, X 1: n := { x 1 , · · · , x n } from
Revised: Given the n samples, X 1: n := { x 1 , · · · , x n } from

Original: N (0 , Σ) , the goal is to estimate the precision matrix Θ ∗ :
Revised: N (0 , Σ) , the goal is to estimate the precision matrix Θ ∗ :

Original: = Σ − 1 which is the inverse covariance matrix.
Revised: = Σ − 1 which is the inverse covariance matrix.

Original: This precision matrix represents the conditional independency among random variables.
Revised: This precision matrix represents the conditional independency among random variables.

Original: A commonly used objective for estimating the precision matrix is the (cid:96) 1 -penalized log determinant divergence:
Revised: A commonly used objective for estimating the precision matrix is the (cid:96) 1 -penalized log determinant divergence:

Original: [Equation]
Revised: [Equation]

Original: where ˆΣ n is the sample covariance matrix.
Revised: where ˆΣ n is the sample covariance matrix.

Original: In our method, we use the likelihood term as the empirical loss.
Revised: In our method, we use the likelihood term as the empirical loss.

Original: [Equation]
Revised: [Equation]

Original: H. D ATA P REPARATION
Revised: I. D ATA P REPARATION

Original: For the sparse linear recovery problem, we follow the setting in Wang et al.
Revised: For the sparse linear recovery problem, we follow the setting in Wang et al.

Original: (2014b).

Original: We create the synthetic data by sampling a set of estimation problems { (( X ( i ) , Y ( i ) ) , β ∗ ( i ) ) } .
Revised: We create the synthetic data by sampling a set of estimation problems { (( X ( i ) , Y ( i ) ) , β ∗ ( i ) ) } .

Original: In each problem, the design matrix X ( i ) ∈ R n × d contains n = 64 independent realizations of a random vector x ∈ R p with p = 256 , 1024 in easy or difﬁcult setting, respectively.
Revised: In each problem, the design matrix X ( i ) ∈ R n × d contains n = 64 independent realizations of a random vector x ∈ R p with p = 256 , 1024 in easy or difﬁcult setting, respectively.

Original: x follows a zero mean Gaussian distribution with covariance matrix (Σ) i,j = 0 .
Revised: x follows a zero mean Gaussian distribution with covariance matrix (Σ) i,j = 0 .

Original: 9 · 1 { i (cid:54) =
Revised: 9 · 1 { i (cid:54) =

Original: j } + 1 · 1 { i = j } .
Revised: j } + 1 · 1 { i = j } .

Original: The true parameter vector β ∗ ( i ) has a sparsity s ∗ =
Revised: The true parameter vector β ∗ ( i ) has a sparsity s ∗ =

Original: (cid:107) β ∗ ( i ) (cid:107) 

Revised: (cid:107) β ∗ ( i ) (cid:107)

Original: = 16 and its nonzero entries take values uniformly sampled from ( − 2 , − 12 ) ∪ ( 12 , 2) .
Revised: = 16 and its nonzero entries take values uniformly sampled from ( − 2 , − 12 ) ∪ ( 12 , 2) .

Original: The support set of each β ∗ ( i ) is independently sampled from a union support set S , with | S | = 128 to allow some similarity among the problems.
Revised: The support set of each β ∗ ( i ) is independently sampled from a union support set S , with | S | = 128 to allow some similarity among the problems.

Original: The observation Y ( i ) is sampled such that Y ( i ) − X ( i ) β ∗ ( i ) is a n -dimensional Gaussian random vector with zero mean and covariance matrix I n .
Revised: The observation Y ( i ) is sampled such that Y ( i ) − X ( i ) β ∗ ( i ) is a n -dimensional Gaussian random vector with zero mean and covariance matrix I n .

Original: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.
Revised: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.

Original: In sparse precision matrix estimation problem, we follow the setting in Guillot et al.
Revised: In sparse precision matrix estimation problem, we follow the setting in Guillot et al.

Original: We create the synthetic data by sampling a set of estimation problems { (ˆΣ ( i ) n , Θ ( i ) ) } .
Revised: We create the synthetic data by sampling a set of estimation problems { (ˆΣ ( i ) n , Θ ( i ) ) } .

Original: In each problem, the ground truth precision matrix is generated in the following ways: the lower diagonal part (lower triangular parting, exluding diagonal) of true precision matrix Θ has a sparsity s ∗ , that are uniformly selected from a union support with size S .
Revised: In each problem, the ground truth precision matrix is generated in the following ways: the lower diagonal part (lower triangular parting, exluding diagonal) of true precision matrix Θ has a sparsity s ∗ , that are uniformly selected from a union support with size S .

Original: After selecting the nonzero entries, we assign them values uniformly from ( − 2 , − 12 ) ∪ ( 12 , 2) and let the upper diagonal part has the same value as lower diagonal part.
Revised: After selecting the nonzero entries, we assign them values uniformly from ( − 2 , − 12 ) ∪ ( 12 , 2) and let the upper diagonal part has the same value as lower diagonal part.

Original: Finally, a multiple of the identity was added to the resulting matrix so that the smallest eigenvalue was equal to 1.
Revised: Finally, a multiple of the identity was added to the resulting matrix so that the smallest eigenvalue was equal to 1.

Original: In this way, Θ was insured to be sparse, positive deﬁnite, and wellconditioned.
Revised: In this way, Θ was insured to be sparse, positive deﬁnite, and wellconditioned.

Original: After the precision matrix is generated, we generate the observational samples for each precision matrix, by generating n independent samples of the random vector x ∼ N (0 , (Θ ( i ) ) − 1 ) .
Revised: After the precision matrix is generated, we generate the observational samples for each precision matrix, by generating n independent samples of the random vector x ∼ N (0 , (Θ ( i ) ) − 1 ) .

Original: The samples are denoted by X ( i ) ∈ R n × d .
Revised: The samples are denoted by X ( i ) ∈ R n × d .

Original: Using the samples, we can compute the sample covariance matrix ˆ Σ ( i ) n = 1 n (cid:80) ni
Revised: Using the samples, we can compute the sample covariance matrix ˆ Σ ( i ) n = 1 n (cid:80) ni

Original: =1 ( X ( i ) )
Revised: =1 ( X ( i ) )

Original: T X ( i ) .
Revised: T X ( i ) .

Original: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.
Revised: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.

Original: We use ( n, S, s ∗ ) =
Revised: We use ( n, S, s ∗ ) =

Original: (100 , 375 , 75) , (200 , 600 , 150) for d = 50 , 200 , respectively.
Revised: (100 , 375 , 75) , (200 , 600 , 150) for d = 50 , 200 , respectively.

Original: H. B ASELINE I MPLEMENTATION
Revised: I. B ASELINE I MPLEMENTATION

Original: ALISTA : We follow the implementation in Liu et al. (2019a).
Revised: ALISTA : We follow the implementation in Liu et al. (2019a).

Original: RNN : The RNN is designed following Andrychowicz et al.
Revised: RNN : The RNN is designed following Andrychowicz et al.

Original: Initialize the β 0 as 0 .
Revised: Initialize the β 0 as 0 .

Original: At every step, we compute the gradient ∇ β L ( β t ) w.r.t.
Revised: At every step, we compute the gradient ∇ β L ( β t ) w.r.t.

Original: to objective as the input to a LSTM with hidden dimension equals to 128.
Revised: to objective as the input to a LSTM with hidden dimension equals to 128.

Original: Then we use the output of the LSTM as the increment:
Revised: Then we use the output of the LSTM as the increment:

Original: [Equation]
Revised: [Equation]

Original: We repeat this iteration 20 steps and use β 20 as our ﬁnal output.
Revised: We repeat this iteration 20 steps and use β 20 as our ﬁnal output.

Original: RNN- (cid:96) 1 : RNN- (cid:96) 1 use the same architecture as RNN.
Revised: RNN- (cid:96) 1 : RNN- (cid:96) 1 use the same architecture as RNN.

Original: The only difference is we add an extra learnable parameter λ t for soft thresholding at each step:
Revised: The only difference is we add an extra learnable parameter λ t for soft thresholding at each step:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where the softthresholding η λ is an elementwise operator maps η λ ( x ) = 

Revised: 81) where the softthresholding η λ is an elementwise operator maps η λ ( x ) =

Original: is | x | < λ and η λ ( x ) = sign ( x )( | x
Revised: is | x | < λ and η λ ( x ) = sign ( x )( | x

Original: | − λ ) .
Revised: | − λ ) .

Original: GLASSO : we use the implementation in the sklearn package.
Revised: GLASSO : we use the implementation in the sklearn package.

Original: GGM : We follow the implementation in Belilovsky et al.
Revised: GGM : We follow the implementation in Belilovsky et al.

Original: APF : We follow the implementation in Wang et al.
Revised: APF : We follow the implementation in Wang et al.

Original: (2014b).

Original: The speciﬁc algorithm steps are summarized in Algorithm 2.
Revised: The speciﬁc algorithm steps are summarized in Algorithm 3.

Original: Algorithm 2: The Approximate Path Following Method
Revised: Algorithm 3: The Approximate Path Following Method

Original: Input: λ tgt > 0 , (cid:15) opt > 

Revised: Input: λ tgt > 0 , (cid:15) opt >

Original: (Here we assume (cid:15) opt (cid:28) λ tgt / 4 ), η ∈ [0 . 9 , 

Revised: (Here we assume (cid:15) opt (cid:28) λ tgt / 4 ), η ∈ [0 . 9 ,

Original: ) 1 Initialize ˜ β 0 ← 0 , L 0 ← L min , λ 

Revised: ) 1 Initialize ˜ β 0 ← 0 , L 0 ← L min , λ

Original: = (cid:107)∇L ( 0 ) (cid:107) ∞ , N ← ( λ 0 /λ tgt ) / log( η − 1 ) .
Revised: = (cid:107)∇L ( 0 ) (cid:107) ∞ , N ← ( λ 0 /λ tgt ) / log( η − 1 ) .

Original: 2 for t = 1 , ..., N − 1 do λ t ← η t λ (cid:15) t ← λ t / { ˜ β t , L t } ← Proximal-Gradient ( λ t , (cid:15) t , ˜ β t − 1 , L t − 1 , R ) as in Algorithm 3.
Revised: 2 for t = 1 , ..., N − 1 do λ t ← η t λ (cid:15) t ← λ t / { ˜ β t , L t } ← Proximal-Gradient ( λ t , (cid:15) t , ˜ β t − 1 , L t − 1 , R ) as in Algorithm 4.

Original: 6 end 7 λ N ← λ tgt 8 (cid:15) N ← (cid:15) opt 9 { ˜ β N , L N } ← Proximal-Gradient ( λ N , (cid:15) N , ˜ β
Revised: 6 end 7 λ N ← λ tgt 8 (cid:15) N ← (cid:15) opt 9 { ˜ β N , L N } ← Proximal-Gradient ( λ N , (cid:15) N , ˜ β

Original: N − 1 , L N − 1 , R ) .
Revised: N − 1 , L N − 1 , R ) .

Original: 10 return { ˜ β t } Nt =1 .
Revised: 10 return { ˜ β t } Nt =1 .

Original: Input: λ t > 0 , (cid:15) t > 0 , β 0 t ∈ R p , L 0 t >1 Initialize k ← 0 .
Revised: Input: λ t > 0 , (cid:15) t > 0 , β 0 t ∈ R p , L 0 t >1 Initialize k ← 0 .

Original: 2 repeat k ← k +
Revised: 2 repeat k ← k +

Original: L init ← max { L min , L k − 1 t / 2 } β kt , L kt ← Line-Search ( λ t , β k − 1 t , L init )
Revised: L init ← max { L min , L k − 1 t / 2 } β kt , L kt ← Line-Search ( λ t , β k − 1 t , L init )

Original: as in Algorithm 4.
Revised: as in Algorithm 5.

Original: 6 until ω λ t ( β kt ) ≤ (cid:15) t as deﬁned in Equation 81 ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .
Revised: 6 until ω λ t ( β kt ) ≤ (cid:15) t as deﬁned in Equation 82 ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .

Original: Input:
Revised: Input:

Original: [Equation]
Revised: [Equation]

Original: 1 repeat
Revised: 1 repeat

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:1) as deﬁned in Equation if φ λ t (cid:0) β kt (cid:1) > 
Revised: (cid:1) as deﬁned in Equation if φ λ t (cid:0) β kt (cid:1) > ψ L kt

Original: ψ L kt ,λ t (cid:0) β kt ; β k − 1 t (cid:1) then
Revised: [Equation] λ t (cid:0) β kt ; β k − 1 t (cid:1) then

Original: L kt ← L kt .
Revised: L kt ← L kt .

Original: end 6 until φ λ t ( β kt ) ≤ ψ L kt 
Revised: end 6 until φ λ t ( β kt ) ≤ ψ L kt ,λ t ( β kt

Original: [Equation]
Revised: [Equation]

Original: λ t ( β kt
Revised: β k

Original: [Equation]
Revised: [Equation]

Original: β k − 1 t ) as deﬁned in Equations83 and 84  ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .[Equation] (cid:0) [Equation]
Revised: 1 t ) as deﬁned in Equations 84 and 85 ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .

Original: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: H. T RAINING AND E VALUATION
Revised: I. T RAINING AND E VALUATION

Original: We trained the learning-based methods to minimize the weighted loss.
Revised: We trained the learning-based methods to minimize the weighted loss.

Original: That’s to say, for each sparse linear recovery problem, given the sequence of outputs ( β 1 , ..., β T ) , the loss is computed as:
Revised: That’s to say, for each sparse linear recovery problem, given the sequence of outputs ( β 1 , ..., β T ) , the loss is computed as:

Original: [Equation]
Revised: [Equation]

Original: where γ decreasing from 0 .
Revised: where γ decreasing from 0 .

Original: 9 to 0 .
Revised: 9 to 0 .

Original: 1 during the training process.
Revised: 1 during the training process.

Original: The procedure is similar in the sparse precision matrix estimation problem.
Revised: The procedure is similar in the sparse precision matrix estimation problem.

Original: We use optimizer Adam (Kingma & Ba, 2014).
Revised: We use optimizer Adam (Kingma & Ba, 2014).

Original: For sparse linear recovery problem, we use batch size 10, we train 500 epochs with learning rate 1e4 and select the model based on the l2 loss on valid data.
Revised: For sparse linear recovery problem, we use batch size 10, we train 500 epochs with learning rate 1e4 and select the model based on the l2 loss on valid data.

Original: For sparse precision matrix estimation problem, we use batch size 40, we train 200 epochs with learning rate 1e-3 and select the model based on Frobenius loss on valid data.
Revised: For sparse precision matrix estimation problem, we use batch size 40, we train 200 epochs with learning rate 1e-3 and select the model based on Frobenius loss on valid data.

Original: For classical algorithms, we select their parameters, λ for APF and ρ for GISTA, based on their performance on the validation set.
Revised: For classical algorithms, we select their parameters, λ for APF and ρ for GISTA, based on their performance on the validation set.

Original: Speciﬁcally, we evaluate APF or GISTA with ρ or λ from { 0 . 01 , 0 .
Revised: Speciﬁcally, we evaluate APF or GISTA with ρ or λ from { 0 . 01 , 0 .

Original: 05 , 0 . 1 , 0 . 2 } and use one with the best recovery error in test.
Revised: 05 , 0 . 1 , 0 . 2 } and use one with the best recovery error in test.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We report the total training time for learning based methods as well as the parameter tuning time for classical algorithms
Revised: We report the total training time for learning based methods as well as the parameter tuning time for classical algorithms.

Original: [Table]

Original: From Table 3 and Table
Revised: From Table 3 and Table

Original: [Table]
Revised: [Table]

Original: we can see that training a learning-based method is cheap in our experiments, as 1)
Revised: we can see that training a learning-based method is cheap in our experiments, as 1)

Original: A single forward for PLISA or GGM is very fast as stated in Table 1.
Revised: A single forward for PLISA or GGM is very fast as stated in Table 1.

Original: We can easily parallel the computations to handle a batch of problems during the training time, as the learning-based methods do not require line-search.
Revised: We can easily parallel the computations to handle a batch of problems during the training time, as the learning-based methods do not require line-search.

Original: The evaluation is performed on a server with CPU: Intel(R)
Revised: The evaluation is performed on a server with CPU: Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Silver 4116 CPU @ 2.10GHz,
Revised: Silver 4116 CPU @ 2.10GHz,

Original: GPU: Nvidia GTX 2080TI, Memory 264G, in single thread. 
Revised: GPU: Nvidia GTX 2080TI, Memory 264G, in single thread. I

Revised: [Figure]

Original: H.  A BLATION S TUDY
Revised:  A BLATION S TUDY

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We consider two variants of PLISA to perform the ablation study.
Revised: We consider two variants of PLISA to perform the ablation study.

Original: One is PLISA -single which employs a single regularization parameter across different entries, i.e., η 1 = · · · = η d and λ ∗ 1 = · · · = λ ∗ d .
Revised: One is PLISA -single which employs a single regularization parameter across different entries, i e., η 1 = · · · = η d and λ ∗ 1 = · · · = λ ∗ d .

Original: The other is PLISA - (cid:96) 1 which does not learn the penalty function but uses the (cid:96) 1 norm, i.e., P w ( λ , β ) =
Revised: The other is PLISA - (cid:96) 1 which does not learn the penalty function but uses the (cid:96) 1 norm, i.e., P w ( λ , β ) =

Original: (cid:107) λ ◦ β (cid:107) 1 .
Revised: (cid:107) λ ◦ β (cid:107) 1 .

Original: Fig.
Revised: Fig.

Original: 6 and Table 5 show the vanilla PLISA performs better than alternatives.
Revised: 6 and Table 5 show the vanilla PLISA performs better than alternatives.

Original: Especially, it has a much better accuracy than PLISA -single.
Revised: Especially, it has a much better accuracy than PLISA -single.

Original: Therefore, this ablation study has validated the effectiveness of using entry-wise regularization parameters and learning the penalty function.
Revised: Therefore, this ablation study has validated the effectiveness of using entry-wise regularization parameters and learning the penalty function.

Original: H. R EAL - WORLD E XPERIMENTS
Revised: I. R EAL - WORLD E XPERIMENTS

Original: We use the following 3 real-world datasets.
Revised: We use the following 3 real-world datasets.

Original: (1) Gene - a single-cell gene expression dataset that contains expression levels of 45 transcription factors measured at different time-points.
Revised: (1) Gene - a single-cell gene expression dataset that contains expression levels of 45 transcription factors measured at different time-points.

Original: We follow Ollier & Viallon (2017) to pick the transcription factor, EGR2, as the response variable and the other 44 factors as the covariates.
Revised: We follow Ollier & Viallon (2017) to pick the transcription factor, EGR2, as the response variable and the other 44 factors as the covariates.

Original: In this dataset, each time-point is considered as an estimation problem.
Revised: In this dataset, each time-point is considered as an estimation problem.

Original: In each estimation problem, the goal is learning the regression weights β ∗ on the 44 factors to predict the expression level of the target transcription factor, EGR2.
Revised: In each estimation problem, the goal is learning the regression weights β ∗ on the 44 factors to predict the expression level of the target transcription factor, EGR2.

Original: Therefore, in this problem, p = 44 .
Revised: Therefore, in this problem, p = 44 .

Original: This dataset contains the gene expression data for 120 single cells at 8 different time points.
Revised: This dataset contains the gene expression data for 120 single cells at 8 different time points.

Original: For each time point, we randomly split the 120 samples into 6 sets, so that each set contains 20 samples.
Revised: For each time point, we randomly split the 120 samples into 6 sets, so that each set contains 20 samples.

Original: By doing this, we construct 48 estimation problems each of which contains 20 samples.
Revised: By doing this, we construct 48 estimation problems each of which contains 20 samples.

Original: 10 samples are used for recovering the parameters β ∗ and the other 10 samples are used for evaluate it by computing the least-square error.
Revised: 10 samples are used for recovering the parameters β ∗ and the other 10 samples are used for evaluate it by computing the least-square error.

Original: We use 36 problems for training, 6 problems for validation, and 6 problems for testing.
Revised: We use 36 problems for training, 6 problems for validation and 6 problems for testing.

Original: (2) Parkinsons - a disease dataset that contains symptom scores of Parkinson for different patients.
Revised: (2) Parkinsons - a disease dataset that contains symptom scores of Parkinson for different patients.

Original: Each patient is considered as an estimation problem.
Revised: Each patient is considered as an estimation problem.

Original: In each estimation problem, the goal is learning the regression weights β ∗ on 19 bio-medical features to predict the symptom score.
Revised: In each estimation problem, the goal is learning the regression weights β ∗ on 19 bio-medical features to predict the symptom score.

Original: This dataset contains 42 patients, so there are 42 estimation problems in total.
Revised: This dataset contains 42 patients, so there are 42 estimation problems in total.

Original: Each patient is examined at different time-point and each time a sample is generated.
Revised: Each patient is examined at different time-point and each time a sample is generated.

Original: For each patient we randomly selectsamples, so that eventually our dataset contains 42 estimation problems each of which containssamples.
Revised: For each patient we randomly selectsamples, so that eventually our dataset contains 42 estimation problems each of which containssamples.

Original: 50 are used for recovering the parameter β ∗ and 50 are used for evaluation.
Revised: 50 are used for recovering the parameter β ∗ and 50 are used for evaluation.

Original: We useproblems for training, 7 problems for validation, and 7 problems for testing.
Revised: We useproblems for training, 7 problems for validation, and 7 problems for testing.

Original: (3) School - an examination score dataset of students from 139 secondary schools in London.
Revised: (3) School - an examination score dataset of students from 139 secondary schools in London.

Original: Each 
Revised: Each school

Original: school is considered as an estimation problem.
Revised: is considered as an estimation problem.

Original: In each estimation problem, the goal is learning the regression weights β ∗ on 28-dimensional school and student features to present the exam scores for all students.
Revised: In each estimation problem, the goal is learning the regression weights β ∗ on 28-dimensional school and student features to present the exam scores for all students.

Original: We use the dataset from Malsar package (Zhou et al., 2011).
Revised: We use the dataset from Malsar package (Zhou et al., 2011).

Original: For each school, we randomly select 40 students as the samples.
Revised: For each school, we randomly select 40 students as the samples.

Original: Since some schools contain less than 40 students, we ﬁnally obtain 125 estimation problems (schools) each of wich contains 40 samples.
Revised: Since some schools contain less than 40 students, we ﬁnally obtain 125 estimation problems (schools) each of wich contains 40 samples.

Original: 20 are used for recovering
Revised: 20 are used for recovering

Original:  We 
Revised: 20 are We for training,problems for validation,

Original: On each dataset, we train each learning-based algorithm for 200 epochs using Adam with learning rata 1e-3.
Revised: On each dataset, we train each learning-based algorithm for 200 epochs using Adam with learning rata 1e-3.

Original: The batch size is set to be 6.
Revised: The batch size is set to be 6.