Original: P ROVABLE L EARNING - BASED A LGORITHM F OR S PARSE R ECOVERY
Revised: P ROVABLE L EARNING - BASED A LGORITHM F OR S PARSE R ECOVERY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recovering sparse parameters from observational data is a fundamental problem in machine learning with wide applications.
Revised: Recovering sparse parameters from observational data is a fundamental problem in machine learning with wide applications.

Original: Many classic algorithms can solve this problem with theoretical guarantees, but the theoretical performances depend on choosing the correct hyperparameters.
Revised: Many classic algorithms can solve this problem with theoretical guarantees, but the theoretical performances depend on choosing the correct hyperparameters.

Original: Besides, they do not fully exploit the particular problem distribution of interest.
Revised: Besides, they do not fully exploit the particular problem distribution of interest.

Original: In this work, we propose
Revised: In this work, we propose

Original: PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm) to learn algorithms automatically from data.
Revised: PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm) to learn algorithms automatically from data.

Original: PLISA is designed by unrolling a classic path-following algorithm, with some components being more ﬂexible and learnable.
Revised: PLISA is designed by unrolling a classic path-following algorithm, with some components being more ﬂexible and learnable.

Original: With this structure, we theoretically show the improved recovery accuracy achievable by PLISA .
Revised: With this structure, we theoretically show the improved recovery accuracy achievable by PLISA .

Original: Furthermore, we analyze the empirical Rademacher complexity of PLISA to characterize its generalization ability to solve new problems outside the training set.
Revised: Furthermore, we analyze the empirical Rademacher complexity of PLISA to characterize its generalization ability to solve new problems outside the training set.

Original: This paper contains novel theoretical contributions to the area of learning-based algorithms in the sense that (i) PLISA is generically applicable to a broad class of sparse estimation problems, (ii) generalization analysis has received less attention so far, and (iii) our analysis makes novel connections between the generalization ability and algorithmic properties such as stability and convergence, which leads to a tighter bound that can explain the empirical observations.
Revised: This paper contains novel theoretical contributions to the area of learning-based algorithms in the sense that (i) PLISA is generically applicable to a broad class of sparse estimation problems, (ii) generalization analysis has received less attention so far, and (iii) our analysis makes novel connections between the generalization ability and algorithmic properties such as stability and convergence of the unrolled algorithm, which leads to a tighter bound that can explain the empirical observations.

Original: The techniques could potentially be applied to analyze other learningbased algorithms in the literature.
Revised: The techniques could potentially be applied to analyze other learning-based algorithms in the literature.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: The problem of recovering a sparse vector β ∗ from ﬁnite observations Z 1: n ∼ ( P β ∗ ) n is fundamental in machine learning, covering a broad family of problems including compressed sensing, sparse regression analysis, graphical model estimation, etc.
Revised: The problem of recovering a sparse vector β ∗ from ﬁnite observations Z 1: n ∼ ( P β ∗ ) n is fundamental in machine learning, covering a broad family of problems including compressed sensing, sparse regression analysis, graphical model estimation, etc.

Original: It has also found applications in various domains.
Revised: It has also found applications in various domains.

Original: For example, in magnetic resonance imaging, sparse signals need to be reconstructed from measurements taken by a scanner.
Revised: For example, in magnetic resonance imaging, sparse signals need to be reconstructed from measurements taken by a scanner.

Original: In computational biology, estimating a sparse graph structure from gene expression data is important for understanding gene regulatory networks.
Revised: In computational biology, estimating a sparse graph struc - ture from gene expression data is important for understanding gene regulatory networks

Original: Various classic algorithms are available for solving sparse recovery problems, and many of them come with theoretical guarantees for the recovery accuracy.
Revised: Various classic algorithms are available for solving sparse recovery problems. Many of them come with theoretical guarantees for the recovery accuracy.

Original: However, the theoretical performance often relies on choosing the correct hyperparameters, such as regularization parameters and the learning rate, which may depend on unknown constants.
Revised: However, the theoretical performance often relies on choosing the correct hyperparameters, such as regularization parameters and the learning rate, which may depend on unknown constants.

Original: Furthermore, in practice, similar problems may need to be solved repeatedly, but it is hard for classic algorithms to fully utilize the information of the particular problem distribution of interest.
Revised: Furthermore, in practice, similar problems may need to be solved repeatedly, but it is hard for classic algorithms to fully utilize this information.

Original: To alleviate these limitations, we consider the approach of learning-to-learn and propose a neural algorithm, called PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm).
Revised: To alleviate these limitations, we consider the approach of learning-to-learn and propose a neural algorithm, called PLISA ( P rovable L earning-based I terative S parse recovery A lgorithm).

Original: PLISA is a deep learning model that takes the observations Z 1: n as the input and outputs an estimation for β ∗ .
Revised: PLISA is a deep learning model that takes the observations Z 1: n as the input and outputs an estimation for β ∗ .

Original: To make use of classic techniques developed by domain experts, we design the architecture of PLISA by unrolling and modifying a classic path-following algorithm proposed by Wang et al.
Revised: To make use of classic techniques developed by domain experts, we design the architecture of PLISA by unrolling and modifying a classic path-following algorithm proposed by Wang et al.

Revised: (2014b).

Original: To beneﬁt from learning, some components in this classic algorithm are made more ﬂexible with careful design and treated as learnable parameters in PLISA .
Revised: To beneﬁt from learning, some components in this classic algorithm are made more ﬂexible with careful design and treated as learnable parameters in PLISA .

Original: These parameters can be learned by optimizing the performances on a set of training problems.
Revised: These parameters can be learned by optimizing the performances on a set of training problems.

Original: The learned PLISA can then be used for solving other problems in the target distribution.
Revised: The learned PLISA can then be used for solving other problems in the target distribution.

Original: With the algorithm design problem converted to a deep learning problem, we naturally ask the two important questions:
Revised: With the algorithm design problem converted to a deep learning problem, we naturally ask the two important questions:

Original: Capacity: What’s the recovery accuracy achievable by PLISA ?
Revised: Capacity: What’s the recovery accuracy achievable by PLISA ?

Original: Can the ﬂexible components in PLISA lead to an algorithm which effectively improves the recovery performance?
Revised: Can the ﬂexible components in PLISA lead to an algorithm which effectively improves the recovery performance?

Original: Generalization: How well can the learned PLISA solve new problems outside the training set?
Revised: Generalization: How well can the learned PLISA solve new problems outside the training set?

Original: Is the generalization behavior related to the algorithmic properties of PLISA ?
Revised: Is the generalization behavior related to the algorithmic properties of PLISA ?

Original: Aiming at supplying rigorous answers to these questions, we conduct theoretical analysis of PLISA to provide guarantees for its recovery performance, measure its empirical Rademachar complexity, and characterize its generalization ability.
Revised: Aiming at supplying rigorous answers to these questions, we conduct theoretical analysis of PLISA to provide guarantees for its recovery performance, measure its empirical Rademachar complexity, and characterize its generalization ability.

Original: The results and the techniques in our analysis can distinguish our work from existing studies on algorithm learning and deep unrolling, which has become an active research direction recently.
Revised: The results and the techniques in our analysis can distinguish our work from existing studies on algorithm learning, which has become an active research direction recently.

Original: To be more speciﬁc, we summarize our new and novel contributions into the following three aspects.
Revised: We summarize our new and novel contributions into the following three aspects.

Original: Theoretical understanding.
Revised: Theoretical understanding.

Original: In contrast to the plethora of empirical studies on algorithm learning, there have been relatively few studies devoted to the theoretical understanding.
Revised: In contrast to the plethora of empirical studies on algorithm learning, there have been relatively few studies devoted to the theoretical understanding.

Original: Existing theoretical efforts primarily focus on analyzing the convergence rate achievable by the neural algorithm
Revised: Existing theoretical efforts primarily focus on analyzing the convergence rate achievable by the neural algorithm

Original: (Chen et al., 2018; Liu et al., 2019a; Zhang & Ghanem, 2018; Wu et al., 2020), but the generalization error bound has received less attention so far.
Revised: (Chen et al., 2018; Liu et al., 2019a; Zhang & Ghanem, 2018; Wu et al., 2020), but the generalization error bound has received less attention so far.

Original: A substantial body of works only argue intuitively that algorithm unrolling architectures can generalize well because they contain a small number of parameters.
Revised: A substantial body of works only argue intuitively that algorithm unrolling architectures can generalize well because they contain a small number of parameters.

Original: In comparison, we provide theoretical guarantees for both the capacity and the generalization ability of PLISA , which are more solid arguments.
Revised: In comparison, we provide theoretical guarantees for both the capacity and the generalization ability of PLISA , which are more solid arguments.

Original: Novel connection.
Revised: Novel connection.

Original: The algorithmic structure inthe architecture of PLISA can make it behaves differently from conventional neural networks.
Revised: The algorithmic structure in PLISA can make it behaves differently from conventional neural networks.

Original: Therefore, we largely utilize the analysis techniques in classic algorithms to derive its generalization bound.
Revised: Therefore, we largely utilize the analysis techniques in classic algorithms to derive its generalization bound.

Original: By combining the analysis tools of deep learning theory and optimization algorithms, our result reveals a novel connection between the generalization ability of PLISA and its algorithmic properties including the convergence rate and stability.
Revised: By combining the analysis tools of deep learning theory and optimization algorithms, our result reveals a novel connection between the generalization ability of PLISA and the algorithmic properties including the convergence rate and stability of the unrolled algorithm.

Original: Beneﬁt from this connection, our generalization bound is tight in the sense that it matches the interesting behavior of PLISA observed in experiments - the generalization gap could decrease in the number of layers, which is rarely observed in conventional neural networks.
Revised: Beneﬁt from this connection, our generalization bound is tight in the sense that it matches the interesting behavior of PLISA observed in experiments - the generalization gap could decrease in the number of layers, which is rarely observed in conventional neural networks.

Original: General setting.
Revised: General setting.

Original: The problem setting in this paper is new and more challenging.
Revised: The problem setting in this paper is new and more challenging.

Original: Existing works mainly focus on a speciﬁc problem.
Revised: Existing works mainly focus on a speciﬁc problem.

Original: For example, the compressed sensing problem with a ﬁxed design matrix is the mostly investigated one.
Revised: For example, the compressed sensing problem with a ﬁxed design matrix is the mostly investigated one.

Original: PLISA , however, is generic and is applicable to various sparse recovery problems as long as they satisfy certain conditions in Assumption 3.1.  
Revised: PLISA , however, is generic and is applicable to various sparse recovery problems as long as they satisfy certain conditions in Assumption 3.1.  PLISA : L EARNING T O S OLVE S PARSE E STIMATION P ROBLEMS

Original: This is complementary to the literature of algorithm learning.
Revised: [Equation] input [Equation]

Original: The remainder of the paper is organized as follows. In Section 2, we elaborate on the proposed architecture of PLISA , and present the learning-to-learn setting under which we learn the parameters. Section 3 states our ﬁrst theoretical result on the achievable recovery performance of PLISA . Section 4 is devoted to the generalization analysis of PLISA .
Revised: [Equation]

Original: In Section 5, we summarize the related works. Section 6 supplies results of numerical simulations. Proofs are contained in the Appendix.  PLISA : L EARNING T O S OLVE S PARSE E STIMATION P ROBLEMS
Revised: [Figure]

Original: A sparse estimation problem is to recover β ∗ from ﬁnite observations Z 1: n sampled from P β ∗ .
Revised: [Equation] ∗ … … [Figure] [Equation]

Original:  As a concrete example, in a sparse linear regression problem, n observations { Z i = ( x i , y i ) } ni =1 are sampled from a linear model y
Revised: [Figure] A sparse estimation problem is to recover β ∗ from ﬁnite observations Z 1: n sampled from P β ∗ . As a concrete example, in a sparse linear regression problem, n observations { Z i = ( x i , y i ) } ni =1 are sampled from a linear model y

Original: = x (cid:62) β ∗ + (cid:15) , and an algorithm needs to estimatethe vector β ∗in this linear model fromthe n observations.
Revised: = x (cid:62) β ∗ + (cid:15) , and an algorithm needs to estimate β ∗ from n observations.

Original: Therefore, an oracle algorithmthat perfectly recovers the true parameter can be view as a function that mapsthe observations Z 1: n to β ∗ :
Revised: Therefore, an oracle algorithm can be viewed as a function that maps Z 1: n to β ∗ :

Original: [Equation]
Revised: Oracle algorithm (function): Z 1: n (cid:55)→ β ∗ .

Original: Based on the equivalent view of algorithms and functions, we propose to approximate the oracle algorithm by a deep learning model, called PLISA θ .
Revised: Based on the equivalent view of algorithms and functions, we propose to approximate the oracle algorithm by a deep learning model, called PLISA θ .

Original: In this section, wewill ﬁrst describe the architecture of PLISA θ and explain its design principles.
Revised: In this section, we ﬁrst describe the architecture of PLISA θ and explain its design principles.

Original: After that, wewill describe how to optimize the parameters in 
Revised: After that, we describe how to optimize the parameters in PLISA θ under the learning-to-learn setting.

Original: 2. A RCHITECTURE OF PLISA θ
Revised: 2. A RCHITECTURE OF PLISA θ

Original: 𝜷 !

Original: =

Original: 𝑍 !: # input 𝝀 $ ∇𝐿 ! (𝑍 ":! ,𝟎) 𝝀 ! 𝜷 " 𝝀 , 𝜷 # 𝜼,𝝀 ∗ … … 𝜷 $%" 𝝀𝜷 $ cell 𝒘,𝜶0 algorithm steps
Revised: A naive architecture could be a recurrent neural network that takes

Original: [Figure] 𝜼,𝝀 ∗ 𝜼,𝝀 ∗ A naive architecture could be a feed-forward or recurrent neural network that takes Z 1: n as the input and outputs a d -dimensional vector.
Revised: Z 1: n as the input and outputs a d -dimensional vector.

Original: However, amore favorable design choice in the literature of algorithm learning is algorithm unrolling , which unrolls and truncates a classic iterative algorithm to design the architecture.
Revised: However, a favorable design choice in the literature of algorithm learning is algorithm unrolling , which unrolls and truncates a classic iterative algorithm to design the architecture.

Original: Based on this idea, we use a classic path-following algorithm proposed by Wang et al.
Revised: Based on this idea, we unroll a classic path-following algorithm proposed by Wang et al.

Original: (2014) as the basis to design PLISA θ .
Revised: (2014b) to design PLISA θ .

Original:  The key idea of path-following algorithms is creating a sequence of surrogate objective functions to gradually approach the target objective function, which is supposed to be more difﬁcult to solve.
Revised: This classic algorithm is presented in Algorithm 2 in Appendix H. The key idea of path-following algorithms is creating a sequence of objectives to gradually approach the target objective that is supposed to be more difﬁcult to solve.

Original: Following the similar design-logic of the path-following algorithm in Wang et al.

Original: (2014), PLISA θ is designed tosequentially approximate the local minimizers of a sequence of objectives, for t = 1 , · · · , T :
Revised: Following this idea, PLISA θ is designed to approximate the local minimizers of a sequence of objectives for t = 1 , · · · , T :

Original: [Equation]
Revised: [Equation]

Original: with regularization parameters λ 1 , · · · , λ T .

Original: In this objective, L n is an empirical loss that measures the “ﬁt” between a parameter β and the observations Z 1: n .

Original: As an example, it can be the least squared loss for solving linear regression problems.

Original: The second term in the objective, P w , is a sparsity penalty function, which is set to be λ (cid:107) β (cid:107) 1 in many classic algorithms.

Original: #cells: T , #layers per cell: K
Revised: #cells: T , #layers per cell: K

Original: Parameters: θ = { η , λ ∗ , w , α }
Revised: Parameters: θ = { η , λ ∗ , w , α }

Original: Input: samples Z 1: n β 0 ← 0 , λ 0 ← ∇ β L n ( Z 1: n , 0 )
Revised: Input: samples Z 1: n β 0 ← 0 , λ 0 ← ∇ β L n ( Z 1: n , 0 )

Original: For t = 1 , . . .
Revised: For t = 1 , . . .

Original: , T do λ t ← max { σ ( η ) ◦ λ t − 1 , λ ∗ } β t ← Cell K w ,α ( Z 1: n , β t − 1 , λ t ) by K
Revised: , T do λ t ← max { σ ( η ) ◦ λ t − 1 , λ ∗ } β t ← Cell K w ,α ( Z 1: n , β t − 1 , λ t ) by K

Original: many update steps in Eq.return β T 
Revised: many update steps in Eq.return β T with regularization parameters λ 1 , · · · , λ T and empirical loss L n which measures the “ﬁt” between a parameter β and observations Z 1: n . As an example, L n can be the least squared loss in linear regression problems. The second term in the objective, P w , is a sparsity penalty function, which is set to be λ (cid:107) β (cid:107) 1 in many classic algorithms.

Original: Consequently,as illustrated by Fig. 2 and Algorithm 1, the architecture of PLISA θ contains T recurrent cells.{+cells (Fig. 
 and Algorithm 1).+}
Revised: Consequently, the architecture of PLISA θ contains T recurrent {+cells (Fig. 
 and Algorithm 1).+}

Original: The t -th cellcorresponds to the t -th objective in Eq. 1, and it contains a K -step algorithm that iteratively minimize the cell-speciﬁc objective.
Revised: Each cell contains a K -step algorithm that minimizes one objective in Eq. 1.

Original: While maintaining the overall structure of the pathfollowing algorithm, PLISA θ is more ﬂexible, as it contains learnable parameters θ
Revised: PLISA θ is different from the classic path-following algorithm in Wang et al.

Original:  = { η , λ ∗ , w , α }  that consist of (i) learnable decrease ratios η and learnable target regularization parameters λ ∗ that jointly determine the sequence of regularization parameters λ  1 , · · · , λ T ; (ii) a learnable sparsity penalty function P w with parameters w ; and (iii) a learnable step size α for the optimization steps in the cells.
Revised: (2014a) in the sense that it contains learnable parameters θ = { η , λ ∗ , w , α } so that the regularization parameters, the sparsity penalty function, and the step size are learned from data. Furthermore, classic algorithms employ a uniform regularization parameter λ across all entries of β , but PLISA θ uses a d -dimensional vector λ t = [ λ t, 1 , · · · , λ t,d ] (cid:62) to enforce different levels of sparsity to different entries. Moreover, the classic algorithm determines the step sizes by line-search.

Revised: Experimentally, we ﬁnd it much slower than PLISA θ that contains learned step sizes.

Original: We explain the details in the following.
Revised: We explain the details in the following.

Original: Red-colored symbols are used to indicate learnable parameters in PLISA θ .
Revised: Red-colored symbols are used to indicate learnable parameters in PLISA θ .

Original: The concrete computations in PLISA θ are presented in Algorithm 1.
Revised: The concrete computations in PLISA θ are presented in Algorithm 1.

Original: (i) Entry-wise regularzation parameters.
Revised: (i) Entry-wise regularzation parameters.

Original: Given input samples Z 1: n , the regularization parameters are initialized by a vector λ 0 := ∇ β L n ( Z 1: n , 0 ) .
Revised: Given input samples Z 1: n , the regularization parameters are initialized by a vector λ 0 := ∇ β L n ( Z 1: n , 0 ) .

Original: Then PLISA θ will update it sequentially by
Revised: Then PLISA θ will update it sequentially by

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) is the element-wise sigmoid function, ◦ is element-wise multiplication, max {· , ·} is element-wise maximization, and  { η , λ ∗ } are learnable parameters.
Revised: where σ ( · ) , ◦ , and max {· , ·} are element-wise sigmoid function, multiplication, and maximization, respectively. { η , λ ∗ } are learnable parameters.

Original: This operation will create
Revised: Eq.

Original:  a decreasing sequence λ 1 , · · · , λ T through theentry-wise decrease ratio σ ( η ) , until they reach the entry values of the learnable target vector λ ∗ .
Revised: 2 will create a decreasing sequence λ 1 , · · · , λ T through the decrease ratio σ ( η ) , until they reach the entry values of the learnable target vector λ ∗ .

Original: It is worth mentioning that different from classic algorithms which employ a uniform regularization parameter λ across all entries of β , PLISA θ always maintains a d -dimensional vector λ t =

Original: [ λ t, 1 , · · · , λ t,d ] (cid:62) to enforce different levels of sparse penalty to different entries.

Original: (ii) Penalty function.
Revised: (ii) Penalty function.

Original: Unlike classic algorithms that use a pre-deﬁned penalty function such as the (cid:96) 1 regularizer

Original: PLISA θ parameterizes the penalty function as follows,
Revised: PLISA θ parameterizes the penalty function as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:102)w i
Revised: where (cid:102)

Original: [Equation]
Revised: [Equation]

Original: In other words, P w is a  convex combination of q penalty functions ( P (1) , · · · , P ( q ) ) , in which the weights of these functions are determined bya learnable vector w =
Revised: In other words, P w is a learnable convex combination of q penalty functions ( P (1) , · · · , P ( q ) ) . The weights of these functions are determined by learnable parameters w =

Original: [ w 1 , · · · , w q ] .
Revised: [ w 1 , · · · , w q ] .

Original: In this paper, we focus on learning the combination of three well-known penalty functions:
Revised: In this paper, we focus on learning the combination of three well-known penalty functions:

Original: [Equation]
Revised: [Equation]

Original:  where P (1) is convex, and MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given in Appendix B.
Revised: ) , where P (1) is convex, and MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given in Appendix B.

Original: In fact, one can include any other penalty functions as long as they satisfy a set of conditions speciﬁed in Appendix B.
Revised: In fact, one can include any other penalty functions as long as they satisfy a set of conditions speciﬁed in Appendix B.

Original: (iii) Cell K w ,α and the step size.
Revised: (iii) Cell K w ,α and the step size.

Original: As mentioned earlier, each cell in PLISA θ contains a K -step algorithm that iteratively minimizes the cell-speciﬁc objective inEq. 1.
Revised: Each cell in PLISA θ contains a K -step algorithm that iteratively minimizes the cell-speciﬁc objective in

Original: Mathematically, given the inputs Z 1: n , β t − 1 , and λ t , the t -th cell Cell K w ,α initialize (cid:101) β 0 t by the output of last cell β t −1 , and computes the following modiﬁed proximal gradient steps: for
Revised: Eq. The t -th cell Cell K w ,α initialize (cid:101) β 0 t by the output of last cell β t −

Original: [Equation]
Revised: and computes the following modiﬁed proximal

Original:  · , K ,
Revised: gradient steps: for k = 1 , · · · , K ,

Original: [Equation]
Revised: [Equation]

Original: cell output:
Revised: cell output:

Original: [Equation]
Revised: [Equation]

Original: In the classic algorithm, the step size α in Eq. 4 is obtained by performing line-search. In PLISA θ , α is a learnable parameter. Experimentally, we ﬁnd this is much more efﬁcient and effective.
Revised: [ T δ ( β )]

Original: For the notations in Eq. 4, T α · λ t is the entry-wise soft-thresholding function, deﬁned as [ T δ ( β )] j := sign( β j ) max {| β j | − δ j , 0 } .
Revised: j := sign( β j ) max {| β j | − δ j , 0 } is the entry-wise soft-thresholding function.

Original: The function Q wrepresents the concave component of the penalty function P w , deﬁned as Q w ( λ , β ) : = P w ( λ , β ) − (cid:107) λ ◦ β (cid:107) 1  .
Revised: The function Q w ( λ , β ) : = P w ( λ , β ) − (cid:107) λ ◦ β (cid:107) 1 represents the concave component of P w .

Original: The analytical form ofits gradient ∇ β Q w ( λ t , β ) can be found in Appendix B. 
Revised: The analytical form of ∇ β Q w ( λ t , β ) can be found in Appendix B. The step size α is a learnable parameter in θ .

Original: Discussion.
Revised: Discussion.

Original: We choose the path-following algorithm in (Wang et al., 2014) as the basis to design PLISA θ because this algorithm is applicable to nonconvex losses L n and nonconvex penalty functions P w .
Revised: We choose the path-following algorithm in (Wang et al., 2014b) as the basis to design PLISA θ because this algorithm is applicable to nonconvex losses L n and nonconvex penalty functions P w .

Original: This allows PLISA θ to be applicable to a broader class of problems (e.g., nonlinear sparse regression problems may involve nonconvex loss).
Revised: This allows PLISA θ to be applicable to a broader class of problems (e.g., nonlinear sparse regression problems may involve nonconvex loss).

Original: Furthermore, employing nonconvex L n and P w can potentially lead to better statistical properties (Fan & Li, 2001; Fan et al., 2009; Loh &
Revised: Furthermore, employing nonconvex L n and P w can potentially lead to better statistical properties (Fan & Li, 2001; Fan et al., 2009; Loh &

Original: Wainwright, 2015), for which we will explain more in Section 3 and in experiments.
Revised: Wainwright, 2015), for which we will explain more in Section 3 and in experiments.

Original: 2. L EARNING - TO - LEARN S ETTING
Revised: 2. L EARNING - TO - LEARN S ETTING

Original: Now we describe how to train the parameters θ in PLISA θ under the learning-to-learn setting.
Revised: Now we describe how to train the parameters θ in PLISA θ under the learning-to-learn setting.

Original: Training set.
Revised: Training set.

Original: Similar to other works in this domain, we assume the access to m problems from the target problem-space P , and use them as the training set:
Revised: Similar to other works in this domain, we assume the access to m problems from the target problem-space P , and use them as the training set:

Original: [Equation]
Revised: [Equation]

Original: with ( Z ( i ) 1:

Original: [Equation]

Original: i ,

Original: [Equation]

Original: ( i [Equation] ∈ P . Here each estimation problem is represented by a pair of observations and the corresponding true parameter to be recovered.
Revised: Here each estimation problem is represented by a pair of observations and the corresponding true parameter to be recovered.

Original: A different problem i can contain a different number n i of observations.
Revised: A different problem i can contain a different number n i of observations.

Original: Training loss.
Revised: Training loss.

Original: Since the intermediate outputs β t ( Z 1: n ; θ ) of PLISA θ are also estimates of β ∗ , a common design of the training loss is the weighted sum of the intermediate estimation errors (Chen et al., 2021).
Revised: Since the intermediate outputs β t ( Z 1: n ; θ ) of PLISA θ are also estimates of β ∗ , a common design of the training loss is the weighted sum of the intermediate estimation errors (Chen et al., 2021).

Original: More speciﬁcally, we employ the following training loss:
Revised: More speciﬁcally, we employ the following training loss:

Original: [Equation]
Revised: [Equation]

Original: where γ < 1 is a discounting factor If γ = 0 then the loss isonly estimatedat the last layer.
Revised: where γ < 1 is a discounting factor If γ = 0 then the loss is estimated

Original: Generalization error.
Revised: Generalization error.

Original: The ultimate goal of algorithm learning is to minimize the estimation error on expectation over all problems in the target problem distribution:
Revised: The ultimate goal of algorithm learning is to minimize the estimation error on expectation over all problems in the target problem distribution:

Original: [Equation]
Revised: [Equation]

Original: where P ( P ) is some distribution in the target problem-space P .
Revised: where P ( P ) is a distribution in the target problem-space P .

Original: Similar to other machine learning problems, this generalization error depends on both the(i) empirical error (i.e., training loss) and the(ii) generalization gap .
Revised: Let θ ∗ ∈ arg min L γ =0 train ( D m ; θ ) be a minimizer of the training loss. It is well-known that the generalization error can be bounded by: [Equation]

Original: We will theoretically characterize them in Section 3 and Section 4.
Revised: We will theoretically characterize these two terms in Theorem 3.1 and Theorem 4.1.

Original:  C APACITY OF PLISA
Revised:  C APACITY OF PLISA

Original: Can PLISA θ recover the true parameters accurately within a small number of layers (cells)?
Revised: Can PLISA θ achieve a small training error without using too many layers (cells)?

Original: In this section, we answer this question by theoretically showing that PLISA θ can achieve a smaller training error compared tothe classic algorithm. Moreover, PLISA θ enjoys a fast convergence rate, and therefore the small training error can be obtained without using too many layers.
Revised: How can designs of element-wise regularization and learnable penalty functions help PLISA θ to achieve a smaller training error compared to classic algorithms? We answer this question theorectically in this section.

Original: 3. P ROBLEM S PACE A SSUMPTIONS
Revised: 3. P ROBLEM S PACE A SSUMPTIONS

Original: Before stating the theorem, we follow the notations in Wang et al.
Revised: Before stating the theorem, we follow the notations in Wang et al.

Original:  ; Loh & Wainwright (2015) to describe some classic assumptions on the estimation problems.
Revised: (2014b) ; Loh & Wainwright (2015) to describe some classic assumptions on the estimation problems.

Original: Assumption 3.1 (Problem Space) .
Revised: Assumption 3.1 (Problem Space) .

Original: Let s ∗ , ˜ s be positive integers and ρ − , ρ + be positive constants such that ˜ s >
Revised: Let s ∗ , ˜ s be positive integers and ρ − , ρ + be positive constants such that ˜ s >

Original: (121( ρ + /ρ − )
Revised: (121( ρ + /ρ − )

Original: + 144( ρ + /ρ − ) 2 ) s ∗ .
Revised: + 144( ρ + /ρ − ) 2 ) s ∗ .

Original: Assume for every estimation problem ( Z 1: n , β ∗ ) in the space P , the following conditions are satisﬁed.
Revised: Assume for every estimation problem ( Z 1: n , β ∗ ) in the space P , the following conditions are satisﬁed.

Original: [-(a) (cid:107) β ∗ (cid:107) 
-]
Revised: [-(a) (cid:107) β ∗ (cid:107) 
-]

Original: ≤ s ∗ and (cid:107) β ∗ (cid:107) ∞

Original: ≤ B 1 ; (b) For any nonzero v ∈ R d with sparsity (cid:107) v (cid:107) 0 ≤ s ∗ + 2˜ s , it holds v

Original: [Table]
Revised: [Equation]

Original: [Table] [Table] [Table] (cid:107) 
Revised: ) (cid:107) β ∗ (cid:107) 0 ≤ s ∗ and (cid:107) β ∗ (cid:107) ∞

Original: [Table]

Original:  ∈  [ ρ [Table] , ρ + ] ; (c) 8 | [ ∇ β L n ( Z 1: n , β ∗ )]
Revised: ≤ B 1 ; (b) For any nonzero v ∈ R d with sparsity (cid:107) v (cid:107) 0 ≤ s ∗ + 2˜ s , it holds v (cid:62) ∇ 2 β L n ( Z 1: n , β ) v (cid:107) v (cid:107)∈ [ ρ − , ρ + ] ; (c) 8 | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ≤ | [ ∇ β L n ( Z 1: n , 0 )]
Revised: j | ≤ | [ ∇ β L n ( Z 1: n , 0 )]

Original: j | ≤ B 2 , ∀ j = 1 , · · · , d .
Revised: j | ≤ B 2 , ∀ j = 1 , · · · , d .

Original: Condition (a) assumes β ∗ is s ∗ -sparse and B 1 -bounded.
Revised: Condition (a) assumes β ∗ is s ∗ -sparse and B 1 -bounded.

Original: Condition (b) is commonly referred to as ‘sparse eigenvalue condition’ (Zhang, 2010b; Wang et al., 2014), which is weaker than the wellknown restricted isometry property (RIP) in compressed sensing (Candes & Tao, 2005).
Revised: Condition (b) is commonly referred to as ‘sparse eigenvalue condition’ (Zhang, 2010b; Wang et al., 2014b), which is weaker than the wellknown restricted isometry property (RIP) in compressed sensing (Candes & Tao, 2005).

Original: Note that the class of functions satisfying conditions of this type is much larger than the class of convex losses.
Revised: Note that the class of functions satisfying conditions of this type is much larger than the class of convex losses.

Original: In the special case when L n ( Z 1: n , β ) is strongly convex in β , condition (b) holds with ˜ s → ∞ .
Revised: In the special case when L n ( Z 1: n , β ) is strongly convex in β , condition (b) holds with ˜ s → ∞ .

Original: The last condition bounds the gradient of the empirical loss L n at the true parameter β ∗ and 0 .
Revised: The last condition bounds the gradient of the empirical loss L n at the true parameter β ∗ and 0 .

Original: 3. F IRST M AIN R ESULT : C APACITY
Revised: F IRST M AIN R ESULT : C APACITY

Original: Let β t ( Z 1: n ; θ ) be the output of the t -th cell in PLISA θ .
Revised: Let β t ( Z 1: n ; θ ) be the output of the t -th cell in PLISA θ .

Original: Let x ∨ a denote entry-wise maximal value max { x , a } .
Revised: Let x ∨ a denote entry-wise maximal value max { x , a } .

Original: Let ( x ) S denote the sub-vector of x with entries indexed by the set S .
Revised: Let ( x ) S denote the sub-vector of x with entries indexed by the set S .

Original: Theorem 3.1 (Capacity) .
Revised: Theorem 3.1 (Capacity) .

Original: Assume the problem space P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume the problem space P satisﬁes Assumption 3.1 and D m ⊆ P .

Original: Let
Revised: Let

Original: T be the number of cells in PLISA θ and let K be the number of layers in each cell.
Revised: T be the number of cells in PLISA θ and let K be the number of layers in each cell.

Original: For any ε > 0 , there exists a set of parameters θ = { η , λ ∗ , w , α } such that the estimation error of every problem ( Z 1: n , β ∗ )
Revised: For any ε > 0 , there exists a set of parameters θ = { η , λ ∗ , w , α } such that the estimation error of every problem ( Z 1: n , β ∗ )

Original: ∈
Revised: ∈

Original: D m is bounded as follows, ∀ T > t 0 ,
Revised: D m is bounded as follows, ∀ T > t 0 ,

Original: [Equation]
Revised: [Equation]

Original: optimization error
Revised: optimization error

Original: [Equation]
Revised: [Equation]

Original: statistical error

Original: [Equation]
Revised: [Equation]

Original:  where S ∗ : 
Revised: statistical error (10) where S ∗ : =

Original: = supp( β ∗ ) is the support indices of β ∗ , c θ , c (cid:48) θ , and C θ are some positive values depending on the chosen θ , and κ m is a condition number which reveals the similarity of the problems in D m .
Revised: supp( β ∗ ) is the support indices of β ∗ , c θ , c (cid:48) θ , and C θ are some positive values depending on the chosen θ , and κ m is a condition number which reveals the similarity of the problems in D m .

Original: Note that K and t 0 are required to be larger than certain values, but we will elaborate in
Revised: Note that K and t 0 are required to be larger than certain values, but we will elaborate in

Original: Appendix D that the required lower bounds are small.
Revised: Appendix D that the required lower bounds are small.

Original: See Appendix D for the proof of this theorem.
Revised: See Appendix D for the proof of this theorem.

Original: This estimation error can be interpreted as a combination of the optimization error (in Eq. 8) and the statistical error (in Eq. 9).
Revised: This estimation error can be interpreted as a combination of the optimization error (in Eq. 9) and the statistical error (in Eq. 10).

Original: Eq. 8 shows the optimization error decreases linearly in both K and T .
Revised: The optimization error decreases linearly in both K and T .

Original: This fast convergence rate allows us to use fewer layers and cells in PLISA θ , which is important because a deeper architecture could be more difﬁcult and less efﬁcient to train.

Original: The statistical error in Eq. 9 occurs because of the randomness in Z 1: n .
Revised: The statistical erroroccurs because of the randomness in Z 1: n .

Original: The gradient at the true parameter ∇ β L n ( Z 1: n , β ∗ ) characterizes how well the ﬁnite samples Z 1: n can represent the distribution P β ∗ .
Revised: The gradient at the true parameter ∇ β L n ( Z 1: n , β ∗ ) characterizes how well the ﬁnite samples Z 1: n can represent the distribution P β ∗ .

Revised: A direct consequence of Theorem 3.1 is that the training error can be small without using too many layers and cells in PLISA θ .

Original: In the following, we will elaborate on how thedesign of entry-wise regularization and learnable penalty function have improved the statistical error.
Revised: Furthermore, based on Theorem 3.1, we will elaborate on how the entry-wise regularization and learnable penalty function can effectively reduce the statistical error and therefore the training error in the following.

Original: (i) Impact of entry-wise regularization.
Revised: (i Impact of entry wise regularization.

Original: In classic algorithms that use a uniform regularization parameter, λ ∗ is usually taken to be proportional to (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞ in order to achieve the optimal statistical rate (Wang et al., 2014; Loh & Wainwright, 2015).

Original: In our case, restricting the regularization to be uniform across entries will lead to an error bound that replaces the norm (cid:107) ( ∇ β L n ( Z 1: n , β ∗ )
Revised: Restricting the regularization to be uniform across entries will lead to an error bound that replaces the statistical error (cid:107) ( ∇ β L n ( Z 1: n , β ∗ )

Original: ∨ ε ) S ∗ (cid:107) 2 in Eq. 9 by √ s ∗ ( (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞ ∨ ε ) .
Revised: ∨ ε ) S ∗ (cid:107) 2 in Eq.by √ s ∗ ( (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞ ∨ ε ) .

Original: To understand how the

Original: former has improved

Original: the

Original: latter , we can consider the sparse linear regression problem introduced
Revised: To understand how the former has improved the latter, we can consider the sparse linear regression problem.

Original: in Section 2 with L n being the least square loss.

Original: If the design matrix is normalized such that max 1 ≤ j ≤ d (cid:107) ([ x 1 ] j , · · · , [ x n ] j ) (cid:107) 2 ≤ √ n , then (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ (cid:107) 2 ≤ C (cid:112) s ∗ /n with high probability.
Revised: If the design matrix is normalized such that max 1 ≤ j ≤ d (cid:107) ([ x 1 ] j , · · · , [ x n ] j ) (cid:107) 2 ≤ √ n , then (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ (cid:107) 2 ≤ C (cid:112) s ∗ /n with high probability.

Original: In comparison, √ s ∗ (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞
Revised: In comparison, √ s ∗ (cid:107)∇ β L n ( Z 1: n , β ∗ ) (cid:107) ∞

Original: ≤ C (cid:112) s ∗ log d/n with high probability is a slower statistical rate due to the term log d .
Revised: ≤ C (cid:112) s ∗ log d/n with high probability is a slower statistical rate due to the term log d .

Original: (ii) Impact of learnable penalty function.
Revised: (ii) Impact of learnable penalty function.

Original: To explain the the beneﬁt of using learnable penalty function, we give a more reﬁned bound for the statistical error in Eq.
Revised: To explain the the beneﬁt of using learnable penalty function, we give a more reﬁned bound for the statistical error in Eq.

Original: 9 in the following lemma.
Revised: 10 in the following lemma.

Original: Lemma 3.1 (Reﬁned bound) .
Revised: Lemma 3.1 (Reﬁned bound) .

Original: Assume the same conditions and parameters θ in Theorem 3.1.
Revised: Assume the same conditions and parameters θ in Theorem 3.1.

Original: Assume T → ∞ so that the optimization error can be ignored.
Revised: Assume T → ∞ so that the optimization error can be ignored.

Original: For simplicity, assume (cid:102) w 3 = 0 and only consider the weights (cid:102) w 1 and (cid:102)
Revised: For simplicity, assume (cid:102) w 3 = 0 and only consider the weights (cid:102) w 1 and (cid:102)

Original: w 2 for (cid:96) 1 penalty and MCP.
Revised: w 2 for (cid:96) 1 penalty and MCP.

Original: Then for every problem ( Z 1: n , β ∗ ) ∈
Revised: Then for every problem ( Z 1: n , β ∗ ) ∈

Original: D m :(cid:107)
Revised: D m :

Original: [Equation]
Revised: [Equation]

Original: ( Z [Equation] [Equation] [Equation]
Revised: β

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where b > 1 is a hyperparameter in MCP , and the index sets S ∗ 1 and S ∗ 2 are deﬁned as S ∗ 1 :=
Revised: where b > 1 is a hyperparameter in MCP , and the index sets S ∗ 1 and S ∗ 2 are deﬁned as S ∗ 1 :=

Original: { j ∈
Revised: { j ∈

Original: S ∗ : (cid:12) (cid:12) β ∗ j (cid:12) (cid:12) ≤ bλ ∗ j } and S ∗ 2 := { j ∈ S ∗ : (cid:12) (cid:12) β ∗ j (cid:12) (cid:12) > bλ ∗ j } .
Revised: S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) ≤ bλ ∗ j } and S ∗ 2 := { j ∈ S ∗ : (cid:12)(cid:12) β ∗ j (cid:12)(cid:12) > bλ ∗ j } .

Original: See Appendix D for the proof.
Revised: See Appendix D for the proof.

Original: This reﬁned bound reveals the beneﬁt of learning the penalty function because:
Revised: This reﬁned bound reveals the beneﬁt of learning the penalty function because:

Original: According to Lemma 3.1, which penalty can minimize the error bound is problem-dependent.

Original: More speciﬁcally, if (8( bρ − + 1) κ m + 1) (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ 1 (cid:107) 2 is larger than (8( bρ − − 1) κ m − 1) (cid:107) ( ∇ β L n ( Z 1: n , β ∗ ) ∨ ε ) S ∗ 2 (cid:107) 2 , choosing (cid:102)
Revised: is a more

Original: w 2 = 0 can induce a smaller error bound. Otherwise, (cid:102) w 2 = 1 is better. Therefore, learning is a more suitable way ofchoosing the penalty function.
Revised: of

Original: Apart from the statistical error, the convergence speed C θ in Eq.
Revised: The convergence speed C θ in Eq.

Original: 8 is also affected by the weights, monotonely decreasing in (cid:102) w 2 .
Revised: 9 is also affected by the weights, monotonely decreasing in (cid:102) w 2 .

Original: Through gradient-based training, we can automatically ﬁnd the optimal combination of penalty functions to strike a nice balance between the statistical error and convergence speed.
Revised: Through gradient-based training, we can automatically ﬁnd the optimal combination of penalty functions to strike a nice balance between the statistical error and convergence speed.

Original:  G ENERALIZATION A NALYSIS
Revised:  G ENERALIZATION A NALYSIS

Original: How well can the learned PLISA θ solve new problems outside the training set?
Revised: How well can the learned PLISA θ solve new problems outside the training set?

Original: In this section, we analyze the generalization gap between the expected loss L gen in Eq.

Original: 7 and the training loss L γ =0 train in Eq.

Original: In particular, our analysisis conducted in a novel way to focus on answering the questions
Revised: In this section, we conduct the generalization analysis in a novel way to focus on answering the questions:

Original: : How is the generalization bound of PLISA θ related to its algorithmic properties ?
Revised: How is the generalization bound of PLISA θ related to its algorithmic properties ?

Original: And how is it different from conventional neural networks?
Revised: And how is it different from conventional neural networks?

Original: 4. S ECOND M AIN R ESULT : G ENERALIZATION B OUND
Revised: 4. S ECOND M AIN R ESULT : G ENERALIZATION B OUND

Original: To analyze the generalization properties of neural networks, many works have adopted the analysis framework of Bartlett & Mendelson (2002) to bound the Rademacher complexity via Dudley’s integral (Bartlett et al., 2017;
Revised: To analyze the generalization properties of neural networks, many works have adopted the analysis framework of Bartlett & Mendelson (2002) to bound the Rademacher complexity via Dudley’s integral (Bartlett et al., 2017;

Original: Chen et al., 2019; Garg et al., 2020;
Revised: Chen et al., 2019; Garg et al., 2020;

Original: Joukovsky et al., 2021).
Revised: Joukovsky et al., 2021).

Original: A key step in this analysis framework is deriving the robustness of the training loss to the small perturbation in model parameters θ .
Revised: A key step in this analysis framework is deriving the robustness of the training loss to the small perturbation in model parameters θ .

Original: In our case, we can view PLISA θ as an iterative algorithm and therefore borrow the analysis tools of classic optimization algorithms to derive its robustness in θ .
Revised: Since we can view PLISA θ as an iterative algorithm, we borrow the analysis tools of classic optimization algorithms to derive its robustness in θ .

Original: The following lemma states this key intermediate result, whichclearly connects the Lipschitz constant to algorithmic properties of PLISA θ .
Revised: The following lemma states this key intermediate result, which connects the Lipschitz constant to algorithmic properties of PLISA θ .

Original: Lemma 4.1 (Robustness to θ ) .
Revised: Lemma 4.1 (Robustness to θ ) .

Original: Assume P satisﬁes Assumption 3.1 and D m ∼ P ( P ) m .
Revised: Assume P satisﬁes Assumption 3.1 and D m ∼ P ( P ) m .

Original: Assume
Revised: Assume

Original: PLISA θ contains T > t 0 cells and K layers.
Revised: PLISA θ contains T > t 0 cells and K layers.

Original: Consider a parameter space Θ in which the parameters satisfy (i)
Revised: Consider a parameter space Θ in which the parameters satisfy (i)

Original: α ∈
Revised: α ∈

Original: [ α min , 1 ρ + ] , (ii) η j ∈ [ σ − 1 (0 . 9) , η max ] , (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , and (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]
Revised: [ α min , 1 ρ + ] , (ii) η j ∈ [ σ − 1 (0 . 9) , η max ] , (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , and (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ∨ ε, λ max ] with some positive constants α min , η max , ξ max , and λ max .
Revised: j | ∨ ε, λ max ] with some positive constants α min , η max , ξ max , and λ max .

Original: Then for any θ = { η , λ ∗ , w , α } and θ (cid:48)
Revised: Then for any θ = { η , λ ∗ , w , α } and θ (cid:48)

Original: = { η (cid:48) , λ ∗(cid:48) , w (cid:48) , α (cid:48) } in Θ , and for any recovery problem ( Z 1: n , β ∗ ) ∈
Revised: = { η (cid:48) , λ ∗(cid:48) , w (cid:48) , α (cid:48) } in Θ , and for any recovery problem ( Z 1: n , β ∗ ) ∈

Original: D m , the following inequality holds,
Revised: D m , the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: rate

Original: [Equation]

Original: where c 1 , c 2 , c 3 , c 4 and C Θ are some positive constants.
Revised: where c 1 , c 2 , c 3 , c 4 and C Θ are some positive constants.

Original: Note that similar to Theorem 3.1, K and tare required to be larger than certain small values.
Revised: Note that similar to Theorem 3.1, K and tare required to be larger than certain small values.

Original: See Appendix E.1 for the proof.
Revised: See Appendix E.1 for the proof.

Original: Convergence rate & step size perturbation.
Revised: Convergence rate & step size perturbation.

Original: As indicated in Eq.
Revised: In Eq.

Original: 12, the Lipschitz constant in the step size α scales at the same rate as the convergence rate of PLISA θ , decreasing exponentially in T and K (See Fig.
Revised: 13, the Lipschitz constant in the step size α scales at the same rate as the convergence rate of PLISA θ , decreasing exponentially in T and K (See Fig.

Original: 3 for a visualization).
Revised: 3 for a visualization).

Original: To understand this, consider when both step sizes α and α (cid:48) are within the convergence region (i.e., (0 , ρ − 1+ ] ).
Revised: To understand this, consider when both step sizes α and α (cid:48) are within the convergence region (i.e., (0 , ρ − 1+ ] ).

Original: After inﬁnitely many steps, their induced outputs will both converge to the same optimal point.
Revised: After inﬁnitely many steps, their induced outputs will both converge to the same optimal point.

Original: This intuitively explains why the output perturbation caused by α -perturbation has the same decrease rate as the optimization error.
Revised: This intuitively explains why the output perturbation caused by α -perturbation has the same decrease rate as the optimization error.

Original: In the proof, we exploit the techniques for analyzing algorithm convergence to obtain this result.

Original: [Table]

Original: KT

Original: [Table]

Original: [Figure]

Original: Stability rate & regularization perturbation.
Revised: Stability rate & regularization perturbation.

Original: In the literature of optimization, stability of an algorithm expresses its robustness to small perturbation in the optimization objective.
Revised: In the literature of optimization, stability of an algorithm expresses its robustness to small perturbation in the optimization objective.

Original: This is clearly related to the robustness of PLISA θ to the perturbation in η , λ ∗ , w , because these parameters jointly play the role of learning the regularization P w ( λ t , β ) , which is a part of the optimization objective in Eq.
Revised: This is clearly related to the robustness of PLISA θ to the perturbation in η , λ ∗ , w , because these parameters jointly determine the regularization P w ( λ t , β ) , which is a part of the optimization objective in Eq.

Original: Therefore, we exploit the analysis techniques for algorithmic stability to derive the robustness in ( η , λ ∗ , w ) -perturbation and obtain the Lipschitz constant in Eq. 13, which is bounded but increasing in T and K (See Fig.for a visualization).
Revised: Therefore, we exploit the analysis techniques for algorithmic stability to derive the robustness in ( η , λ ∗ , w ) -perturbation and obtain the Lipschitz constant in Eq. 14, which is bounded but increasing in T and K (See Fig. 3 for a visualization).

Original: Based on the key result in Lemma 4.1, we can apply Dudley’s integral to measure the empirical
Revised: Based on the key result in Lemma 4.1, we can apply Dudley’s integral to measure the empirical

Original: Rademachar complexity which immediately yields the following generalization bound.
Revised: Rademachar complexity which immediately yields the following generalization bound.

Original: Theorem 4.1 (Generalization gap) .
Revised: Theorem 4.1 (Generalization gap) .

Original: Assume the assumptions in Lemma 4.1.
Revised: Assume the assumptions in Lemma 4.1.

Original: For any (cid:15) > 0 , with probability at least 1 − (cid:15) , the generalization gap is bounded by[Equation] [Equation] stability
Revised: For any (cid:15) > 0 , with probability at least 1 − (cid:15) , the generalization gap is bounded by

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 3 , C Θ are constants independent of d, m, K and T .
Revised: where c 1 , c 2 , c 3 , C Θ are constants independent of d, m, K and T .

Original: See Appendix E for the proof. 
Revised: See Appendix E for the proof. KT

Original: Fig.
Revised: Convergence Rate KT Stability KT

Revised: Generalization Bound

Revised: KT

Revised: Convergence Rate

Revised: KT

Revised: Stability

Revised: KT

Original:  3 visualizes how the generalization bound in Theorem 4.1 grows when KT increases.
Revised: Generalization Bound Fig. 3 visualizes how the generalization bound in Theorem 4.1 grows when KT increases.

Original: The two sets of plots look slightly different by picking different constants C Θ .
Revised: The two sets of plots look slightly different by picking different constants C Θ .

Original: We have also tried varying the values of c 2 , c 3 , d, m in Theorem 4.1.
Revised: We have also tried varying the values of c 2 , c 3 , d, m in Theorem 4.1.

Original: Overall, theywill lead to the two types of behaviors in Fig.
Revised: Overall, they lead to the two types of behaviors in Fig.

Original: It will increase ﬁrst and then decrease to a constant, but the speed can vary.

Original: An important observation in Theorem 4.1 and Figure 3 is that the generalization gap could decrease in the number of layers , andthis matches the empirical observations of PLISA as we reported in
Revised: An important observation in Theorem 4.1 and Figure 3 is that the generalization gap could decrease in the number of layers , and we will see in

Original: Section 6.
Revised: Section 7 that this matches the empirical observations.

Original: It also distinguishes algorithm-unrolling based architectures from conventional neural networks, whose generalization gaps rarely decrease in the number of layers.
Revised: It also distinguishes algorithm-unrolling based architectures from conventional neural networks, whose generalization gaps rarely decrease in the number of layers.

Original: Remark.
Revised: Remark.

Original: The above generalization results are conducted on a constrained parameter space (as described in Lemma 4.1) so that we can utilize the algorithmic properties of PLISA θ .
Revised: The above generalization results are conducted on a constrained parameter space (as described in Lemma 4.1) so that we can utilize the algorithmic properties of PLISA θ .

Original: We focus on this space because the analysis contains more interesting and new ingredients.
Revised: We focus on this space because the analysis contains more interesting and new ingredients.

Original: For parameters outside this space, the analysis procedure is similar to other conventional recurrent networks.
Revised: For parameters outside this space, the analysis procedure is similar to other conventional recurrent networks.

Original: Since the bound in Theorem 4.1 has matched the empirical observations, it is reasonable to believe that after training, the learned parameters are likely to be in this ‘nice’ constrained space.  
Revised: Since the bound in Theorem 4.1 has matched the empirical observations, it is reasonable to believe that after training, the learned parameters are likely to be in this ‘nice’ constrained space.  E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING

Revised: Real-world datasets may not contain the ground-truth parameters β ∗ , but only contain the samples from each task, D Um = { Z (1)1: n , · · · , Z ( m ) 1: n } .

Original: R ELATED W ORK
Revised: In this setting, we can construct an unsupervised training loss to minimize the empirical loss function L n (e.g., the likelihood function) on the samples. Unsupervised training loss: [Equation] In this loss, both Z ( i ) 1: n 1 and Z ( i ) 1: n 2 are subsets of Z ( i ) 1: n .

Revised: The samples Z ( i ) 1: n 1 are used as the input to

Revised: PLISA θ

Revised: and the samples Z ( i ) 1: n 2 are used for evaluating the output β T from PLISA θ .

Revised: Let θ ∗

Revised: U ∈ arg min

Revised: L Utrain ( D Um ; θ ) be a minimizer to this unsupervised loss.

Revised: Theoretically , to bound the generalization error of θ ∗ U , we can show that

Revised: [Equation]

Revised: Compared to Eq.

Revised: 8, this upper bound contains an additional statistical error, which appears because of the gap between the unsupervised loss and the true error that we aim to optimize.

Revised: Clearly, in this upper bound, the generalization gap can be bounded by Theorem 4.1.

Revised: Furthermore, the unsupervised training error in Eq.

Revised: 17 can be bounded by combining Theorem 3.1 with the following in equality.

Original:  Learning-to-learnand deep unrolling has become an active research direction in recent years (Franceschi et al., 2017;Niculae et al., 2018;
Revised: [Equation] More details of the extension to unsupervised setting can be found in Appendix G.  R ELATED W ORK Learning-to-learn has become an active research direction in recent years (Bora et al., 2017;

Original:  Denevi et al., 2018; Poganˇci´c et al., 2019; Liu et al., 2019b; Berthet et al., 2020).
Revised: Franceschi et al., 2017; Niculae et al., 2018; Denevi et al., 2018; Poganˇci´c et al., 2019; Liu et al., 2019b; Berthet et al., 2020).

Original: Many works share the idea of unrolling an iterative algorithm to design the architecture (Yang et al., 2017; Borgerding et al., 2017; Corbineau et al., 2019; Xie et al., 2019;
Revised: Many works share the idea of unrolling or differentiating through algorithms to design the architecture (Yang et al., 2017; Borgerding et al., 2017; Corbineau et al., 2019; Xie et al., 2019;

Original: Shrivastava et al., 2020; Chen et al., 2020a; Wei et al., 2020).
Revised: Shrivastava et al., 2020; Chen et al., 2020a; Wei et al., 2020; Indyk et al., 2019; Grover et al., 2019; Wu et al., 2019).

Original: A well-known example  is
Revised: A well-known example of learning-based algorithm is

Original: LISTA (Gregor & LeCun, 2010) which interprets ISTA (Daubechies et al., 2004) as layers of neural networks and hassince then been an active research topic (Zhang & Ghanem, 2018; Kamilov & Mansour, 2016; Chen et al., 2018; Liu et al., 2019a; Wu et al., 2020; Kim & Park, 2020).
Revised: LISTA (Gregor & LeCun, 2010) which interprets ISTA (Daubechies et al., 2004) as layers of neural networks and has been an active research topic (Zhang & Ghanem, 2018; Kamilov & Mansour, 2016; Chen et al., 2018; Liu et al., 2019a; Wu et al., 2020; Kim & Park, 2020).

Original: However,not many works have been conducted on the theoretical understanding of algorithm learning, especially the generalization ability.
Revised: However, the generalization of algorithm learning has received less attention.

Original: The only exceptions are several recent works of Chen et al. (2020b); Wang et al. ; Behboodi et al.
Revised: The only exceptions are several works.

Original: Joukovsky et al.

Original: However, Chen et al.
Revised: However, Chen et al.

Original: (2020b)
Revised: (2020b)

Original: and Wang et al.
Revised: and Wang et al.

Original: (2021) only consider learning to optimize quadratic losses.
Revised: (2021) only consider learning to optimize quadratic losses.

Original: Behboodi et al.
Revised: Behboodi et al.

Original: Joukovsky et al.
Revised: Joukovsky et al.

Original: (2021) do not connect the generalization analysis with algorithmic properties to provide tighter bounds as in our work.
Revised: (2021) do not connect the generalization analysis with algorithmic properties to provide tighter bounds as in our work.

Revised: Unlike our work that analyzes the Lipschitz continuity (Lemma 4.1), the work of Gupta & Roughgarden (2017); Balcan et al.

Revised: (2021) studied the generalization of learning-based algorithms with a focus on scenarios when the Lipschitz continuity is unavailable.

Original: We will refer the audience to Shlezinger et al.
Revised: We will refer the audience to Shlezinger et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: for a more comprehensive summary of related works in this area.
Revised: for a more comprehensive summary of related works.

Original:  E XPERIMENTS 
Revised:  E XPERIMENTS 7. S YNTHETIC E XPERIMENTS

Original: Inthis section, we report the experimental results to validate the effectiveness of PLISA and our theoretical results. We create the synthetic data by sampling a set of linear sparse recovery problems { (( X ( i ) , Y ( i ) ) , β ∗ ( i ) , ) } . In each problem, the design matrix X ( i ) ∈ R n × d contains n = 64 independent realizations of a random vector x ∈ R d with d = 256 , 1024 in easy or difﬁcult setting, respectively. x follows a zero mean Gaussian distribution with covariance matrix (Σ) i,j = 0 .
Revised: In synthetic datasets, we consider sparse linear regression problems and sparse precision matrix estimation problems for Gaussian graphical models.

Original: 9 · 1 { i (cid:54) = j } + 1 · 1 { i = j } . The true parameter vector β ∗ ( i ) has a sparsity (cid:107) β ∗ [-( i ) (cid:107) 
-] = 16 and its nonzero entries take values uniformly sampled from ( − 2 , − 12 ) ∪ ( 12 , 2) .
Revised: Speciﬁcally, we recover target vectors β ∗ [-( i ) (cid:107) 
-]∈ R d , where d = 256 , 1024 , in SLR, and estimate precision matrices Θ ∗ ∈ R d × d , where d = 50 , 100 , in SPE.

Original: The support set ofeach β ∗ ( i ) is independently sampled from a union support set D , with | D | = 128 to allow some similarity among the problems.
Revised: See Appendix H.2 for descriptions of the dataset and data preparation.

Original: The observation Y ( i ) is sampled such that Y ( i ) − X ( i ) β ∗ ( i ) is a n -dimensional Gaussian random vector with zero mean and covariance matrix I n . In all the experiments, m =such problems are used for training.
Revised: 7.1. S PARSE L INEAR R EGRESSION (SLR)

Original: [Figure]
Revised: [Figure]

Revised: In this experiment, we compare PLISA with several baselines and also verify the theorems.

Original: Recovery performance.
Revised: Performance comparison.

Original: In the ﬁrst experiment, we compare PLISA with alternative methods including APF (Wang et al., 2014), ALISTA (Liu et al., 2019a), RNN (Andrychowicz et al., 2016), and RNN- (cid:96) 1 .
Revised: We consider baselines including APF (Wang et al., 2014b), ALISTA (Liu et al., 2019a), RNN (Andrychowicz et al., 2016), and RNN- (cid:96) 1 .

Original: APF is the classic approximate pathfollowing algorithm that is used as the basis of our architecture.
Revised: APF is the path-following algorithm that is used as the basis of our architecture.

Original: ALISTA is a representative ofthe algorithm unrolling based deep architecture, which is an advanced variant of LISTA.
Revised: ALISTA is a representative of algorithm unrolling based architectures, which is an advanced variant of LISTA.

Original: We have tried the vanilla LISTA, but it performs worse than ALISTA on our tasks so it is not reported.
Revised: We have tried the vanilla LISTA, but it performs worse than ALISTA on our tasks so it is not reported.

Original: RNN refers to the LSTMbased model in Andrychowicz et al.
Revised: RNN refers to the LSTM-based model in Andrychowicz et al.

Original: (2016) which is applied to learning-to-learn.

Original: Besides, we add a soft-thresholding operator to this model to enforce sparsity, and include this variant as a baseline, called RNN- (cid:96) 1 , which reveals to perform better than RNN.
Revised: Besides, we add a soft-thresholding operator to this model to enforce sparsity, and include this variant as a baseline, called RNN- (cid:96) 1 .

Original: Except for APF, all methods are trained on the same set of training problems and selected by the validation problems.
Revised: Except for APF, all methods are trained on the same set of training problems and selected by the validation problems.

Original: For APF, we perform grid-search to choose its hyperparameters, which is also selected by the validation set.
Revised: For APF, we perform grid-search to choose its hyperparameters, which is also selected by the validation set.

Original: The detailed speciﬁcation of each model can be found in Appendix G.
Revised: The detailed speciﬁcation of each model can be found in Appendix H.3.

Original: The comparison results are reported in Fig.
Revised: [Figure] Fig.

Original: 4, which shows the convergence ofthe recovery (cid:96) 2 error, (cid:107) β t − β ∗ (cid:107) 2 , inwhich the  x -axis indicates the wall-clock time.
Revised: 4 shows the convergence of (cid:107) β t − β ∗ (cid:107) 2 for problems in the test set. The x -axis indicates the wall-clock time.

Original: It can be seen that in terms of the ﬁnal recovery accuracy, PLISA outperforms all baseline methods.
Revised: In terms of the ﬁnal recovery accuracy, PLISA outperforms all baseline methods.

Original: In the more difﬁcult setting (i.e,when d = 1024 ), its advantage isvery obvious.
Revised: In the more difﬁcult setting (i.e, d = 1024 ), its advantage is obvious.

Original: AlthoughPLISA is slightly slower than other deep learning based models due to the computations of MCP and SCAD , PLISA achieves a better accuracy and it has been converging much faster than the classic algorithm APF. [Table] [Figure] Ablation study.
Revised: Although

Original: In the second experiment, we perform an ablation study to verify the effectiveness of the designs in PLISA. We consider two variants of PLISA . One is PLISA -single which employs a single regularization parameter across different entries, i.e., η 1 = · · · = η d and λ ∗ 1 = · · · = λ ∗ d . The other is PLISA - (cid:96) 1 which does not learn the penalty function but uses the (cid:96) 1 norm as the penalty function, i.e., P w ( λ , β ) =
Revised: PLISA is slightly slower than other deep learning based models due to the computations of

Original: (cid:107) λ ◦ β (cid:107) 1 .

Original: As reported in Fig. 5 and Table 1, the vanilla PLISA performs better  than the alternatives.
Revised: MCP and SCAD , PLISA achieves a better accuracy and it has been converging much faster than the classic algorithm APF.

Original: Especially, it has a much better accuracy than PLISA -single which does not use entry-wise regularization parameters. Therefore, this ablation study has validated the effectiveness of using entry-wise regularization parameters and learning the penalty function. [Figure]
Revised: APF is very slow mainly due to the use of line-search for selecting step sizes.

Original: Generalization gap.
Revised: Generalization gap.

Original: We are interested inseeing what is the generalization behavior of PLISA experimentally.
Revised: We are interested in the generalization behavior of PLISA .

Original: As this experiment is conducted for theoretical interest, we do not use the validation set to select the model.
Revised: As this experiment is conducted for theoretical interest, we do not use the validation set to select the model.

Original: We vary the number of layers ( K ) and cells ( T ) in PLISA to create a set of models with different depths.
Revised: We vary the number of layers ( K ) and cells ( T ) in PLISA to create a set of models with different depths.

Original: For each depth, we train the model with 2000 training problems, and then test it on a separate set of 100 problems to approximate the generalization gap.
Revised: For each depth, we train the model with 2000 training problems, and then test it on a separate set of 100 problems to approximate the generalization gap.

Original: In Fig.
Revised: In Fig.

Original: 6, we observe the interesting behavior of the generalization gap, where the left one increases in KT at the beginning and then decrease to a constant, and the right one increases fast and then decrease very slowly.
Revised: 5, we observe the interesting behavior of the generalization gap, where the left one increases in KT at the beginning and then decrease to a constant, and the right one increases fast and then decrease very slowly.

Original: This surprisingly matches the two different visualizations in Fig.
Revised: This surprisingly matches the two different visualizations in Fig.

Original: 3 of the predicted bound given by 
Revised: 3 of the predicted generalization gap given by Theorem 4.1. 7.1. S PARSE P RECISION M ATRIX E STIMATION (SPE) We compare PLISA with APF, GLASSO (Friedman et al., 2008), GISTA (Guillot et al., 2012), and GGM (Belilovsky et al., 2017) on sparse precision estimation tasks in Gaussian graphical models.

Revised: GLASSO estimates the precision matrix by block-coordinate decent methods.

Revised: GISTA is a proximal gradient method for precision matrix estimation.

Original: Theorem 4.1.
Revised: GGM utilizes convolutional neural network to estimate the precision matrix. Details of each model and the training can be found in Appendix H.4. [Table] [Table]

Revised: Table 1 reports the Frobenius error (cid:107)

Revised: Θ − Θ ∗ (cid:107) 2 F between the estimation Θ and the true precision matrix Θ ∗ , averaged over 100 test problems.

Revised: PLISA achieves consistent improvements.

Revised: Classic algorithm are slower because they perform linesearch.

Revised: GLASSO is faster than other classic algorithm because we use the sklearn package (Pedregosa et al., 2011) in which the implementations are optimized.

Revised: 7.1. D ISCUSSION ON T RAINING -T ESTING T IME

Revised: Test time.

Revised: Fig.

Revised: 4 and Table 1 shows the wall-clock time for solving test problems.

Revised: Overall, classic algorithms are slower because they need to perform line-search.

Revised: It is noteworthy that learning-based methods can solve a batch of problems parallelly but most classic algorithms cannot.

Revised: To allow more advantages for classic algorithms, the test problems are solved without batching in all methods.

Revised: Train time.

Revised: As metioned earlier, we also perform grid-search to select the hyperparameters in classic algorithms using the training and validation sets.

Revised: The training time for both classic methods and learning-based methods are summarized in Table 3 and Table 4 in Appendix H.4.

Revised: We can see that training time is not a bottleneck of this problem.

Revised: Most methods can be trained efﬁciently.

Revised: In

Revised: SPE, classic algorithms even require a longer training time than learning-based methods.

Revised: 7. U NSUPERVISED L EARNING ON R EAL - WORLD D ATASETS

Revised: We conduct experiments of unsupervised algorithm learning on 3 datasets: Gene expression dataset (Kouno et al., 2013), Parkinsons patient dataset (Tsanas et al., 2009), and School exam score dataset (Zhou et al., 2011).

Revised: See Appendix H.6 for details of datasets and the conﬁguration.

Revised: [Table]

Revised: The goal of the algorithm is to estimation the sparse linear regression parameters β ∗ for each problem.

Revised: Recovery accuracy is estimated by the least-square error on a set of held-out samples for each problem.

Revised: Table 2 shows the effectiveness of PLISA .

Revised: Note that these real-world datasets may not satisfy the assumptions in this paper.

Revised: This set of experiments are conducted only to demonstrate the robustness of the proposed method.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed an algorithm-unrolling based architecture called PLISA for learning to solve sparse parameter recovery problems.
Revised: We proposed an algorithm-unrolling based architecture called PLISA for learning to solve sparse parameter recovery problems.

Original: We presented a theoretical analysis for both the capacity and generalization ability of PLISA .
Revised: We presented a theoretical analysis for both the capacity and generalization ability of PLISA .

Original: The results can shed new light on the relation between generalization and the algorithmic properties, and the techniques could be used to derive guarantees for other algorithmunrolling based architectures.
Revised: The results can shed new light on the relation between generalization and the algorithmic properties, and the techniques could be used to derive guarantees for other algorithmunrolling based architectures.

Original: The current work can be extended in the future from several aspects.

Original: The model PLISA can be improved, for example, by using a more ﬂexible penalty function (e.g., a conventional neural network) as long as it satisﬁes Assumption B.1.
Revised: The model PLISA can be improved by using a more ﬂexible penalty function (e.g., a conventional neural network) as long as it satisﬁes Assumption B.1.

Original: Theoretically, it will be interesting to extend the current result to the unsupervised learning scenario when the true parameters are unobserved.

Original: A L IST OF DEFINITIONS AND NOTATIONS
Revised: A L IST OF DEFINITIONS AND NOTATIONS

Original: For the convenience of the reader
Revised: For the convenience of the reader

Original: [Equation]
Revised: [Equation]

Original: we summarize a list of notations blow.
Revised: we summarize a list of notations blow.

Original: (cid:107) β (cid:107) λ , 1 := (cid:80) dj
Revised: (cid:107) β (cid:107) λ , 1 := (cid:80) dj

Original: =

Revised: =

Original: λ j | β j | denotes element-wise (cid:96) 1 norm.
Revised: λ j | β j | denotes element-wise (cid:96) 1 norm.

Original: ( x ) S is the sub-vector of x with entries in the index set S .
Revised: ( x ) S is the sub-vector of x with entries in the index set S .

Original: x ∨ (cid:15) = max { x , (cid:15) } =
Revised: x ∨ (cid:15) = max { x , (cid:15) } =

Original: [max { x 1 , (cid:15) } , · · · , max { x d , (cid:15) } ] (cid:62) .
Revised: [max { x 1 , (cid:15) } , · · · , max { x d , (cid:15) } ] (cid:62) .

Original: Element-wise soft-threshold:
Revised: Element-wise soft-threshold:

Original: [Equation]
Revised: [Equation]

Original: Denote a single iteration of the modiﬁed proximal gradient step as
Revised: Denote a single iteration of the modiﬁed proximal gradient step as

Original: [Equation]
Revised: [Equation]

Original: B P ENALTY F UNCTIONS
Revised: B P ENALTY F UNCTIONS

Original: We provide detailed descriptions of the penalty functions used in PLISA .
Revised: We provide detailed descriptions of the penalty functions used in PLISA .

Original: We focus on learning the combination of three well-known penalty functions:
Revised: We focus on learning the combination of three well-known penalty functions:

Original: [Equation]
Revised: [Equation]

Original: where MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given below.
Revised: where MCP (Zhang, 2010a) and SCAD (Fan & Li, 2001) are nonconvex penalties whose analytical forms are given below.

Original: B. A NALYTICAL FORMS
Revised: B. A NALYTICAL FORMS

Original: The MCP penalty can be written as
Revised: The MCP penalty can be written as

Original: [Equation]
Revised: [Equation]

Original: where b > 0 is some hyperparameter.
Revised: where b > 0 is some hyperparameter.

Original: The SCAD penalty can be written as
Revised: The SCAD penalty can be written as

Original: [Equation]
Revised: [Equation]

Original: where a > 2 is some hyperparameter
Revised: where a > 2 is some hyperparameter

Original: [Equation]
Revised: [Equation]

Original: PLISA uses the concave components of these penalty functions.
Revised: PLISA uses the concave components of these penalty functions.

Original: Their analytical forms are as follows.
Revised: Their analytical forms are as follows.

Original: [Equation]
Revised: [Equation]

Original: B. I MPLICATIONS
Revised: B. I MPLICATIONS

Original: Based on the deﬁnitions of these penalty functions, it is easy to check that their convex combination satisﬁes a set of conditions as stated in the following Assumption B.1.
Revised: Based on the deﬁnitions of these penalty functions, it is easy to check that their convex combination satisﬁes a set of conditions as stated in the following Assumption B.1.

Original: Assumption B.1 (Regularization) .
Revised: Assumption B.1 (Regularization) .

Original: Let P ( λ , β ) =
Revised: Let P ( λ , β ) =

Original: Q ( λ , β )+ (cid:107) β (cid:107) λ , 1 where Q is concave in β , and that Q ( λ , β ) =
Revised: Q ( λ , β )+ (cid:107) β (cid:107) λ , 1 where Q is concave in β , and that Q ( λ , β ) =

Original: (cid:80) dj
Revised: (cid:80) dj

Original: =1 q ( λ j , β j ) .
Revised: =1 q ( λ j , β j ) .

Original: Denote the partial gradient of q as q (cid:48) λ ( β ) :
Revised: Denote the partial gradient of q as q (cid:48) λ ( β ) :

Original: =
Revised: =

Original: ∂q ( λ,β ) ∂β .
Revised: ∂q ( λ,β ) ∂β .

Original: Assume the following regularization conditions.
Revised: Assume the following regularization conditions.

Original: (a) There exists constants ξ ≥ 0 such that for any β (cid:48) >
Revised: (a) There exists constants ξ ≥ 0 such that for any β (cid:48) >

Original: β ,
Revised: β ,

Original: [Equation]
Revised: [Equation]

Original: (b) q ( λ, − β ) =
Revised: (b) q ( λ, − β ) =

Original: q ( λ, β ) .
Revised: q ( λ, β ) .

Original: (c) q (cid:48) λ
Revised: (c) q (cid:48) λ

Original: =
Revised: =

Original: q ( λ, 0) = 0 .
Revised: q ( λ, 0) = 0 .

Original: (d) | q (cid:48) λ
Revised: (d) | q (cid:48) λ

Original: ( β ) | ≤ λ .
Revised: ( β ) | ≤ λ .

Original: (e) | q (cid:48)
Revised: (e) | q (cid:48)

Original: λ 1 ( β ) − q (cid:48)
Revised: λ 1 ( β ) − q (cid:48)

Original: λ 2 ( β ) | ≤ | λ 1 − λ 2 | .
Revised: λ 2 ( β ) | ≤ | λ 1 − λ 2 | .

Original: Note that the penalty function P w ( λ , β ) in PLISA θ satisﬁes all these conditions, with the constant ξ in condition (a) being
Revised: Note that the penalty function P w ( λ , β ) in PLISA θ satisﬁes all these conditions, with the constant ξ in condition (a) being

Original: [Equation]
Revised: [Equation]

Original: where a and b are the hyperparameters in SCAD and MCP .
Revised: where a and b are the hyperparameters in SCAD and MCP .

Original: Next, we present an important lemma which states the restricted strongly convexity and smoothness of the modiﬁed loss
Revised: Next, we present an important lemma which states the restricted strongly convexity and smoothness of the modiﬁed loss

Original: [Equation]
Revised: [Equation]

Original: Lemma B.1 (Restricted Strongly Convex and Restricted Strongly Smooth) .
Revised: Lemma B.1 (Restricted Strongly Convex and Restricted Strongly Smooth) .

Original: Assume P satisﬁes Assumption 3.1 and ( Z n , β ∗ ) ∈ P .
Revised: Assume P satisﬁes Assumption 3.1 and ( Z n , β ∗ ) ∈ P .

Original: If Q ( λ , β ) satisﬁes the regularization conditions in Assumption B. with a constant ξ < ρ − .
Revised: If Q ( λ , β ) satisﬁes the regularization conditions in Assumption B. with a constant ξ < ρ − .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: is strongly convex and smooth in the subspace {(cid:107) β (cid:48) − β {+(cid:107) 
+}
Revised: is strongly convex and smooth in the subspace {(cid:107) β (cid:48) − β {+(cid:107) 
+}

Original: [Equation]

Original: 0 ≤ s ∗ + 2˜ s } .
Revised: ≤ s ∗ + 2˜ s } .

Original: That is, for any β , β (cid:48) such that (cid:107) β (cid:48) − β (cid:107) 

Revised: That is, for any β , β (cid:48) such that (cid:107) β (cid:48) − β (cid:107)

Original: ≤ s ∗ + 2˜ s , the following inequalities hold 
Revised: ≤ s ∗ + 2˜ s , the following inequalities hold [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is straightforward by using the condition (a) in Assumption B.1 and the sparseeigenvalue condition in Assumption 3.1.
Revised: The proof is straightforward by using the condition (a) in Assumption B.1 and the sparseeigenvalue condition in Assumption 3.1.

Original: C R ECOVERY G UARANTEE F OR P ARAMETERS I N C ONSTRAINED S PACE
Revised: C R ECOVERY G UARANTEE F OR P ARAMETERS I N C ONSTRAINED S PACE

Original: Our generalization analysis is conducted on a constrained parameter space.
Revised: Our generalization analysis is conducted on a constrained parameter space.

Original: In this section, we provide theoretical guarantees for the estimation error of PLISA θ when θ is in the constrained space.
Revised: In this section, we provide theoretical guarantees for the estimation error of PLISA θ when θ is in the constrained space.

Original: We ﬁrst formally deﬁne the parameter space as follows.
Revised: We ﬁrst formally deﬁne the parameter space as follows.

Original: Deﬁnition C.1 (Constrained parameter space) .
Revised: Deﬁnition C.1 (Constrained parameter space) .

Original: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .

Original: Given some positive values α min , η max , ξ max , and λ max .
Revised: Given some positive values α min , η max , ξ max , and λ max .

Original: We deﬁnite the constrained parameter space Θ
Revised: We deﬁnite the constrained parameter space Θ

Original: ( D m ) as Θ( D m ) :
Revised: ( D m ) as Θ( D m ) :

Original: = { θ = { η , λ ∗ , w , α } : conditions (i) (ii) (iii) (iv) are satisﬁed } , where the conditions are:
Revised: = { θ = { η , λ ∗ , w , α } : conditions (i) (ii) (iii) (iv) are satisﬁed } , where the conditions are:

Original: (i) α ∈ [ α min , 1 ρ + ] .
Revised: (i) α ∈ [ α min , 1 ρ + ] .

Original: (ii) η j ∈ [ η min :=
Revised: (ii) η j ∈ [ η min :=

Original: σ − 1 (0 . 9) , η max ] , for all j = 1 , · · · , d . (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , where b is the hyperparameter in MCP and a is the hyperparameter in SCAD .
Revised: σ − 1 (0 . 9) , η max ] , for all j = 1 , · · · , d . (iii) (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 ≤ ξ max < ρ − , where b is the hyperparameter in MCP and a is the hyperparameter in SCAD .

Original: (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]
Revised: (iv) λ ∗ j ∈ [8 sup ( Z 1: n, β ∗ ) ∈D m | [ ∇ β L n ( Z 1: n , β ∗ )]

Original: j | ∨ ε, λ max ] , for all j = 1 , · · · , d .
Revised: j | ∨ ε, λ max ] , for all j = 1 , · · · , d .

Original: For parameters in this constrained space, we can show the performance guarantee for the outputs of
Revised: For parameters in this constrained space, we can show the performance guarantee for the outputs of

Original: PLISA θ , as stated in the following Theorem C.1.
Revised: PLISA θ , as stated in the following Theorem C.1.

Original: Theorem C.1 (Recovery guarantee) .
Revised: Theorem C.1 (Recovery guarantee) .

Original: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .
Revised: Assume P satisﬁes Assumption 3.1 and D m ⊆ P .

Original: Let Θ( D m ) be the constrained parameter space deﬁned in Deﬁnition C.1 and assume θ = { η , λ ∗ , w ,
Revised: Let Θ( D m ) be the constrained parameter space deﬁned in Deﬁnition C.1 and assume θ = { η , λ ∗ , w ,

Original: α } ∈ Θ
Revised: α } ∈ Θ

Original: ( D m ) .
Revised: ( D m ) .

Original: Assume
Revised: Assume

Original: [Equation]
Revised: [Equation]

Original: where δ := 1 − α min ( ρ − − ξ max ) .
Revised: where δ := 1 − α min ( ρ − − ξ max ) .

Original: Then for any problem ( Z 1: n , β ∗ ) in the training set D m , all intermediate outputs of PLISA θ have the following error bounds for all t
Revised: Then for any problem ( Z 1: n , β ∗ ) in the training set D m , all intermediate outputs of PLISA θ have the following error bounds for all t

Original: = 1 , · · · , T .
Revised: = 1 , · · · , T .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this theorem is based on a series of key lemmas for the properties of the modiﬁed proximal gradient steps in each cell of PLISA .
Revised: The proof of this theorem is based on a series of key lemmas for the properties of the modiﬁed proximal gradient steps in each cell of PLISA .

Original: We state these key lemmas and their proofs in
Revised: We state these key lemmas and their proofs in

Original: Section F. With these lemmas, the proof of this theorem is straightforward, and we state the proof below.
Revised: Section F. With these lemmas, the proof of this theorem is straightforward, and we state the proof below.

Original: Given a ﬁxed problem ( Z 1: n , β ∗ ) ∈
Revised: Given a ﬁxed problem ( Z 1: n , β ∗ ) ∈

Original: D m and a ﬁxed set of parameters θ = { η , λ ∗ , w , α } ∈ Θ( D m ) , we simplify some notations in this proof by removing the dependency on Z 1: n and θ .
Revised: D m and a ﬁxed set of parameters θ = { η , λ ∗ , w , α } ∈ Θ( D m ) , we simplify some notations in this proof by removing the dependency on Z 1: n and θ .

Original: For example, we denote β t =
Revised: For example, we denote β t =

Original: β t
Revised: β t

Original: ( Z 1: n ; θ ) , (cid:101) β
Revised: ( Z 1: n ; θ ) , (cid:101) β

Original: kt = (cid:101) β kt ( Z 1: n ; θ ) , λ t = λ t ( Z 1: n ; θ ) , etc.
Revised: kt = (cid:101) β kt ( Z 1: n ; θ ) , λ t = λ t ( Z 1: n ; θ ) , etc.

Original: Notations.
Revised: Notations.

Original: Furthermore, we will use the following simpliﬁed notations and deﬁnitions.
Revised: Furthermore, we will use the following simpliﬁed notations and deﬁnitions.

Original: empirical loss L n ( β ) :
Revised: empirical loss L n ( β ) :

Original: = L n ( Z 1: n , β ) modiﬁed loss L λ
Revised: = L n ( Z 1: n , β ) modiﬁed loss L λ

Original: [Equation]
Revised: [Equation]

Original: β )
Revised: β )

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: regularized loss φ λ ( β )
Revised: regularized loss φ λ ( β )

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: : n
Revised: : n

Original: [Equation]
Revised: [Equation]

Original: local optimal
Revised: local optimal

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to state the proof.
Revised: Now we are ready to state the proof.

Original: We ﬁrst show the following statement holds true for all t ≤ T by mathematical induction.
Revised: We ﬁrst show the following statement holds true for all t ≤ T by mathematical induction.

Original: Statement ( t ) :
Revised: Statement ( t ) :

Original: ω λ t ( (cid:101) β 0 t ) ≤ 1 2 , and (cid:107) ( (cid:101) β kt ) S ∗ (cid:107) 

Revised: ω λ t ( (cid:101) β 0 t ) ≤ 1 2 , and (cid:107) ( (cid:101) β kt ) S ∗ (cid:107)

Original: ≤ ˜ s, ∀ k = 1 , · · · , K.
Revised: ≤ ˜ s, ∀ k = 1 , · · · , K.

Original: Statement (1) .
Revised: Statement (1) .

Original: First we verify that Statement (1) holds true.
Revised: First we verify that Statement (1) holds true.

Original: Recall that λ 0 = |∇
Revised: Recall that λ 0 = |∇

Original: L n ( 0 ) | .
Revised: L n ( 0 ) | .

Original: In the following, we prove that (cid:101) β 01 is a local solution of φ λ 0 ( β ) :
Revised: In the following, we prove that (cid:101) β 01 is a local solution of φ λ 0 ( β ) :

Original: = L
Revised: = L

Original: [Equation]
Revised: [Equation]

Original: 0 ( β ) + (cid:107) β (cid:107) λ 0 , 1 .
Revised: 0 ( β ) + (cid:107) β (cid:107) λ 0 , 1 .

Original: [Equation]
Revised: [Equation]

Original: Since − sign( ∇ L n ( 0 )) ∈
Revised: Since − sign( ∇ L n ( 0 )) ∈

Original: ∂ (cid:107) 

Revised: ∂ (cid:107)

Original: (cid:107) 1 , then we have 0 ∈ ∇ L λ 0 ( (cid:101) β 01 ) + λ 0 ◦
Revised: (cid:107) 1 , then we have 0 ∈ ∇ L λ 0 ( (cid:101) β 01 ) + λ 0 ◦

Original: ∂ (cid:107) (cid:101) β 01 (cid:107) 1 .
Revised: ∂ (cid:107) (cid:101) β 01 (cid:107) 1 .

Original: Therefore, ω λ 0 ( (cid:101) β 01 ) = 0 .
Revised: Therefore, ω λ 0 ( (cid:101) β 01 ) = 0 .

Original: Since λ 1 = λ 0 ◦ η 1 , by Lemma F.6, we have
Revised: Since λ 1 = λ 0 ◦ η 1 , by Lemma F.6, we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma F.5, we have φ λ 1 (
Revised: By Lemma F.5, we have φ λ 1 (

Original: [Equation]
Revised: [Equation]

Original: β 01 )
Revised: β 01 )

Original: [Equation]
Revised: [Equation]

Original: max , which implies that the conditions in
Revised: max , which implies that the conditions in

Original: Lemma F.4 are satisﬁed.
Revised: Lemma F.4 are satisﬁed.

Original: Therefore, we have proved that
Revised: Therefore, we have proved that

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof of Statement (1) .
Revised: This ﬁnishes the proof of Statement (1) .

Original: Statement ( t ) .
Revised: Statement ( t ) .

Original: Now we assume Statement ( t − 1) is true and prove Statement ( t ) .
Revised: Now we assume Statement ( t − 1) is true and prove Statement ( t ) .

Original: First, we prove that ω λ
Revised: First, we prove that ω λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: β Kt − 1 ) ≤ 1 / 4 .
Revised: β Kt − 1 ) ≤ 1 / 4 .

Original: By Lemma F.2 and Lemma F.5,
Revised: By Lemma F.2 and Lemma F.5,

Original: [Equation]
Revised: [Equation]

Original: Assume K ≥ log (cid:16) 4 (cid:113) 29 ρ + ρ − − ξ max (1 +
Revised: Assume K ≥ log (cid:16) 4 (cid:113) 29 ρ + ρ − − ξ max (1 +

Original: [Equation]
Revised: [Equation]

Original: + 1 , where
Revised: + 1 , where

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:101) β 0 t = (cid:101) β
Revised: Since (cid:101) β 0 t = (cid:101) β

Original: Kt − 1 , Lemma F.6 implies that
Revised: Kt − 1 , Lemma F.6 implies that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by Lemma F.5, ω λ t ( (cid:101) β 0 t ) ≤ 1 / 2 implies
Revised: Furthermore, by Lemma F.5, ω λ t ( (cid:101) β 0 t ) ≤ 1 / 2 implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which in turns implies that the conditions in Lemma F.4 are satisﬁed.
Revised: which in turns implies that the conditions in Lemma F.4 are satisﬁed.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of Statement ( t ) .
Revised: which completes the proof of Statement ( t ) .

Original: Now we derive the error bounds.
Revised: Now we derive the error bounds.

Original: Similar to how we derive Eq.
Revised: Similar to how we derive Eq.

Original: 19 and Eq.
Revised: 25 and Eq.

Original: 20, it is easy to show that ω λ t
Revised: 26, it is easy to show that ω λ t

Original: [Equation]
Revised: [Equation]

Original: β [Equation] ) ≤1 / 4 ,
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: , and φ λ t
Revised: , and φ λ t

Original: [Equation]
Revised: [Equation]

Original: β0 t
Revised: β

Original: [Equation]
Revised: [Equation]

Original:  −φ
Revised: 22 ρ −

Original: [Equation]
Revised: [Equation]

Original: β

Original: [Equation]
Revised: [Equation]

Original: [Equation] 22 ρ − − ξ max ∀ [Equation] = 1 , · · · , T.
Revised: T.

Original: Combining φ λ t ( (cid:101) β 0 t ) − φ λ t ( β ∗ ) ≤ 21 / 2 (cid:107) ( λ t ) S ∗ (cid:107) 22 ρ − − ξ max with Lemma F.3, we have[Equation] ρ [Equation]
Revised: Combining φ λ t ( (cid:101) β 0 t ) − φ λ t ( β ∗ ) ≤ 21 / 2 (cid:107) ( λ t ) S ∗ (cid:107) 22 ρ − − ξ max with Lemma F.3, we have

Original: ∀

Original: [Equation]
Revised: [Equation]

Original: K. Combining ω λ t ( (cid:101) β Kt ) ≤ 1 / 4 with Lemma F.5, we have (cid:107)
Revised: Combining ω λ t ( (cid:101) β Kt ) ≤ 1 / 4 with Lemma F.5, we have (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) (cid:107) (

Original: [Equation]
Revised: [Equation]

Original:  t ) [Equation] ∗ (cid:107)
Revised: 19 / 8 (cid:107) ( λ t ) S ∗ (cid:107)

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 19 / 8 (cid:107) ( λ t ) S ∗ (cid:107)ρ − − ξ max .
Revised: max .

Original: D C APACITY A NALYSIS : P ROOF OF T HEOREM 3. 
Revised: D C APACITY A NALYSIS : P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: This theorem is a direct result of Theorem C.1, by taking a suitable set of parameters θ .
Revised: This theorem is a direct result of Theorem C.1, by taking a suitable set of parameters θ .

Original: Similar to the proof of Theorem C.1, some notations are simpliﬁed.
Revised: Similar to the proof of Theorem C.1, some notations are simpliﬁed.

Original: Please refer to the proof of
Revised: Please refer to the proof of

Original: Theorem C.1 for detailed illustrations.
Revised: Theorem C.1 for detailed illustrations.

Original: Denote the union support set of the true parameters β ∗ by S ∗ m :=
Revised: Denote the union support set of the true parameters β ∗ by S ∗ m :=

Original: { j : j ∈ supp ( β ∗ ) , ( Z 1: n , β ∗ ) ∈
Revised: { j : j ∈ supp ( β ∗ ) , ( Z 1: n , β ∗ ) ∈

Original: D m } .
Revised: D m } .

Original: Then we specify a set of parameters θ = { η , λ ∗ , w , α } for PLISA θ as follows.
Revised: Then we specify a set of parameters θ = { η , λ ∗ , w , α } for PLISA θ as follows.

Original: • α = 1 ρ + .
Revised: • α = 1 ρ + .

Original: • w : The weight w can take any values as long as it satisﬁes (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 < ρ − .
Revised: • w : The weights w can take any values as long as they satisfy (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 < ρ − .

Revised: Since (cid:102) w 2 and (cid:102) w 3 and be arbitrary close to 0, there must exists a set of weights w that satisfy this constraint.

Original: In particular, we denote this value by ξ w := (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 .
Revised: In particular, we denote this value by ξ w := (cid:102) w 2 1 b + (cid:102) w 3 1 a − 1 .

Original: • λ ∗ :
Revised: • λ ∗ :

Original: For each j ∈ S ∗ m , we take λ ∗ j := 8 max ( Z n , β ∗ ) ∈D m |∇ L n ( Z n , β ∗ )
Revised: For each j ∈ S ∗ m , we take λ ∗ j := 8 max ( Z n , β ∗ ) ∈D m |∇ L n ( Z n , β ∗ )

Original: j | ∨ ε as the target regularization parameter.
Revised: j | ∨ ε as the target regularization parameter.

Original: For each j / ∈ S ∗ m , we take λ ∗ j = B 2 , which is the upper bound of (cid:107)∇ β L n ( Z 1: n , 0 ) (cid:107) ∞ by Assumption 3.1.
Revised: For each j / ∈ S ∗ m , we take λ ∗ j = B 2 , which is the upper bound of (cid:107)∇ β L n ( Z 1: n , 0 ) (cid:107) ∞ by Assumption 3.1.

Original: η
Revised: η

Original: For all j , take η j = log 9 so that the decrease ratio is σ ( η j ) = 0 . 9 .
Revised: For all j , take η j = log 9 so that the decrease ratio is σ ( η j ) = 0 . 9 .

Revised: Clearly, this set of parameters satisfy the conditions in Theorem C.1, so we can apply its result in this proof.

Original: In the following, we will show that, with this speciﬁcation of the parameters, PLISA θ can achieve the recovery accuracy stated in Theorem 3.1.
Revised: In the following, we will show that, with this speciﬁcation of the parameters, PLISA θ can achieve the recovery accuracy stated in Theorem 3.1.

Original: Let t 0 =
Revised: Let t 0 =

Original: min { t : t ≥ 0 , λ ∗ = max { 0 .
Revised: min { t : t ≥ 0 , λ ∗ = max { 0 .

Original: 9 t |∇ β L n ( Z 1: n , 0 ) | , λ ∗ } , Z 1: n ∈ D m , λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t ( Z 1: n ; θ )
Revised: 9 t |∇ β L n ( Z 1: n , 0 ) | , λ ∗ } , Z 1: n ∈ D m , λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t ( Z 1: n ; θ )

Original: = λ ∗ .
Revised: = λ ∗ .

Original: We know t 0 is a small number because the decrease rate is linear.
Revised: We know t 0 is a small number because the decrease rate is linear.

Original: To be more clear, it is easy to show that t 0 ≤ log (cid:0) B 2 ε (cid:1) / log(10 / 9) .
Revised: To be more clear, it is easy to show that t 0 ≤ log (cid:0) B 2 ε (cid:1) / log(10 / 9) .

Original: Therefore, after t 0 cells, the regularization parameters do not change anymore.
Revised: Therefore, after t 0 cells, the regularization parameters do not change anymore.

Original: That is, 
Revised: That is, λ

Revised: [Equation]

Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: [Equation]
Revised: [Equation]

Original:  Since the speciﬁed parameters satisfy the conditions in Theorem C.1, we can follow the same way as how we derive Eq.
Revised: ≥ [Equation] Since the speciﬁed parameters satisfy the conditions in Theorem C.1, we can follow the same way as how we derive Eq.

Original: 19 to obtain that
Revised: to obtain that

Original: [Equation]
Revised: [Equation]

Original: Since we have λ t = λ ∗ for t ≥ t 0 .
Revised: Since we have λ t = λ ∗ for t ≥ t 0 .

Original: The last ( T − t 0 ) cells can be viewed as a single cell with
Revised: The last ( T − t 0 ) cells can be viewed as a single cell with

Original: K ( T − t 0 ) many steps.
Revised: K ( T − t 0 ) many steps.

Original: Therefore, we can apply Lemma F.2 and Lemma F.5 to obtain
Revised: Therefore, we can apply Lemma F.2 and Lemma F.5 to obtain

Original: [Equation]
Revised: [Equation]

Original:  , where C θ = − 12 log (1 − α ( ρ − − ξ w )) , c θ = (cid:113) 29 ρ + ρ − − ξ w (1 + 1 αρ + )
Revised: ( − C θ K ( t − t 0 )) , where C θ = − 12 log (1 − α ( ρ − − ξ w )) , c θ = (cid:113) 29 ρ + ρ − − ξ w (1 + 1 αρ + )

Original: B [-
-]
Revised: B [-
-]2 1 − α ( ρ − − ξ w ) . By Lemma F.5, (cid:107)

Original: [Equation]
Revised: [Equation]

Original: α ( ρ − − ξ w ) .
Revised: t [Equation]

Original: By Lemma F.5,
Revised: ∗ [Equation] λ

Original: [Equation]
Revised: [Equation]

Original: where ˜ c θ = √ s ∗ B 2 ρ − − ξ w c θ and c (cid:48) θ = 17 8( ρ − − ξ w ) .
Revised: where ˜ c θ = √ s ∗ B 2 ρ − − ξ w c θ and c (cid:48) θ = 17 8( ρ − − ξ w ) .

Original: What remains is bounding (cid:107) λ ∗ S ∗ (cid:107) 2 .
Revised: What remains is bounding (cid:107) λ ∗ S ∗ (cid:107) 2 .

Original: First, we deﬁne a condition number as
Revised: First, we deﬁne a condition number as

Original: [Equation]
Revised: [Equation]

Original: In fact, κ m ≤ B 2 ε .
Revised: In fact, κ m ≤ B 2 ε .

Original: Then recall the speciﬁcation of λ ∗ at the beginning of this proof.
Revised: Then recall the speciﬁcation of λ ∗ at the beginning of this proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Plugging this back to Eq.
Revised: Plugging this back to Eq.

Original: 22, we have
Revised: 28, we have

Original: [Equation]
Revised: [Equation]

Original: This is equivalent to the statement to be proved.
Revised: This is equivalent to the statement to be proved.

Original: E G ENERALIZATION A NALYSIS : P ROOF OF T HEOREM 4. 
Revised: E G ENERALIZATION A NALYSIS : P ROOF OF T HEOREM 4.

Original: We ﬁrst state a well-known inequality that bounds the generalization gap by empirical Rademacher complexity .
Revised: We ﬁrst state a well-known inequality that bounds the generalization gap by empirical Rademacher complexity .

Original: Theorem E.1 (Adapted from Theorem 26.5 (Shalev-Shwartz & Ben-David, 2014)) .
Revised: Theorem E.1 (Adapted from Theorem 26.5 (Shalev-Shwartz & Ben-David, 2014)) .

Original: Assume that for all ( Z 1: n , β ∗ ) ∈ P and all θ ∈ Θ we have that (cid:96) ( β T ( Z 1: n ; θ ) , β ∗ ) ≤ c .
Revised: Assume that for all ( Z 1: n , β ∗ ) ∈ P and all θ ∈ Θ we have that (cid:96) ( β T ( Z 1: n ; θ ) , β ∗ ) ≤ c .

Original: Then with probability at least 1 − (cid:15) , for all θ ∈ Θ ,
Revised: Then with probability at least 1 − (cid:15) , for all θ ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } is the space of loss functions of our model PLISA θ , and R m ( (cid:96) Θ ) is its empirical Rademacher complexity.
Revised: where (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } is the space of loss functions of our model PLISA θ , and R m ( (cid:96) Θ ) is its empirical Rademacher complexity.

Original: It is deﬁned as
Revised: It is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where { σ i } mi =1 are m independent Rademacher random variables.
Revised: where { σ i } mi =1 are m independent Rademacher random variables.

Original: Therefore, it resorts to bound the empirical Rademacher complexity, which can be bounded via
Revised: Therefore, it resorts to bound the empirical Rademacher complexity, which can be bounded via

Original: Dudley’s integral.
Revised: Dudley’s integral.

Original: The following theorem states the bound for the empirical Rademacher complexity and its proof follows.
Revised: The following theorem states the bound for the empirical Rademacher complexity and its proof follows.

Original: Theorem E.2 (Empirical Rademacher complexity bound) .
Revised: Theorem E.2 (Empirical Rademacher complexity bound) .

Original: Assume the assumptions in Theorem 4.1.
Revised: Assume the assumptions in Theorem 4.1.

Original: Let Θ = Θ
Revised: Let Θ = Θ

Original: ( D m ) be the constrained space deﬁned in Deﬁnition C.1.
Revised: ( D m ) be the constrained space deﬁned in Deﬁnition C.1.

Original: Let (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } be the space of loss functions of our model PLISA θ .
Revised: Let (cid:96) Θ := { ( Z 1: n , β ∗ ) (cid:55)→ (cid:107) β T ( Z 1: n ; θ ) − β ∗ (cid:107) 22 : θ ∈ Θ } be the space of loss functions of our model PLISA θ .

Original: Then the empirical Rademacher complexity is bounded by R m (cid:96) Θ ≤ c 1 √
Revised: Then the empirical Rademacher complexity is bounded by R m (cid:96) Θ ≤ c 1 √

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 3 , c 4 are some constants independent of d, m, K and T .
Revised: where c 1 , c 2 , c 3 , c 4 are some constants independent of d, m, K and T .

Original: Proof.
Revised: Proof.

Original: The classical Dudley’s entropy integral bound gives us an upper bound for the empirical
Revised: The classical Dudley’s entropy integral bound gives us an upper bound for the empirical

Original: Rademacher complexity in terms of the covering number.
Revised: Rademacher complexity in terms of the covering number.

Original: [Equation]
Revised: [Equation]

Original: The notation P m means the empirical measure deﬁned based on the samples in D m , and
Revised: The notation P m means the empirical measure deﬁned based on the samples in D m , and

Original: [Equation]
Revised: [Equation]

Original: by Theorem C.
Revised: by Theorem C.

Original: [Equation]
Revised: [Equation]

Original: sup θ
Revised: sup θ

Original: [Equation]
Revised: [Equation]

Original: Here we use a constant C as the upper bound because we only care about the scales in d, m, K and
Revised: Here we use a constant C as the upper bound because we only care about the scales in d, m, K and

Original: T .
Revised: T .

Original: Next, we bound the covering number N (cid:16) 1 √ m , (cid:96) Θ , L 2 ( P m ) (cid:17) .
Revised: Next, we bound the covering number N (cid:16) 1 √ m , (cid:96) Θ , L 2 ( P m ) (cid:17) .

Original: By the key result in Lemma 4.1, it is easy to derive that
Revised: By the key result in Lemma 4.1, it is easy to derive that

Original: [Equation]
Revised: [Equation]

Original: From now on
Revised: From now on

Original: [Equation]
Revised: [Equation]

Original: we abuse the symbol c to generally represent some constant that does not depend on d, m, K, T
Revised: we abuse the symbol c to generally represent some constant that does not depend on d, m, K, T

Original: [Equation]
Revised: [Equation]

Original: For the covering number of a d -dimensional compact vector space T ⊆
Revised: For the covering number of a d -dimensional compact vector space T ⊆

Original: R d , we can bound it by the (cid:15)
Revised: R d , we can bound it by the (cid:15)

Original: -packing number.
Revised: -packing number.

Original: That is, N ( T, (cid:15), (cid:96) 2 ) ≤ M ( T, (cid:15), (cid:96) 2 ) ≤ c (cid:81) dj
Revised: That is, N ( T, (cid:15), (cid:96) 2 ) ≤ M ( T, (cid:15), (cid:96) 2 ) ≤ c (cid:81) dj

Original: =1 (cid:16) T j, max − T j, min (cid:15) + 1 (cid:17) for some constant c .
Revised: =1 (cid:16) T j, max − T j, min (cid:15) + 1 (cid:17) for some constant c .

Original: Applying this fact, we have
Revised: Applying this fact, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: which implies log
Revised: which implies log

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: 1 − exp( − C Θ KT ))
Revised: 1 − exp( − C Θ KT ))

Original: + 1 (cid:1) + 3 log (cid:16) c 
Revised: + 1 (cid:1) + 3 log (cid:16) c √ dm (1 − exp( − C Θ KT ))

Original: [Equation] (1 − exp( − C Θ KT )) (cid:17) ≤ log (cid:0) c √ mK ( T − t 0 ) exp( − C Θ K ( T − t 0 ))
Revised: (cid:17) ≤ log (cid:0) c √ mK ( T − t 0 ) exp( − C Θ K ( T − t 0 ))

Original: ∨ 1 (cid:1) + cd log (cid:0) c √ m (1 − exp( − C Θ KT ))
Revised: ∨ 1 (cid:1) + cd log (cid:0) c √ m (1 − exp( − C Θ KT ))

Original: + 1 (cid:1) .
Revised: + 1 (cid:1) .

Original: Therefore, R m (cid:96) Θ ≤ c
Revised: Therefore, R m (cid:96) Θ ≤ c

Original: [Equation]
Revised: [Equation]

Original: E. R OBUSTNESS TO P ARAMETER P ERTURBATION : P ROOF OF L EMMA 4. 
Revised: E. R OBUSTNESS TO P ARAMETER P ERTURBATION : P ROOF OF L EMMA 4.

Original: We split the parameters θ = { λ ∗ , w , η , α } into two sets θ = θ 1 ∪ θ 2 , where θ 1 = { α } and θ 2 = { λ ∗ , w , η } .
Revised: We split the parameters θ = { λ ∗ , w , η , α } into two sets θ = θ 1 ∪ θ 2 , where θ 1 = { α } and θ 2 = { λ ∗ , w , η } .

Original: Clearly, the robustness can be bounded by
Revised: Clearly, the robustness can be bounded by

Original: [Equation]
Revised: [Equation]

Original: We derive the upper bounds of these two terms in Lemma E.1 and
Revised: We derive the upper bounds of these two terms in Lemma E.1 and

Original: Lemma E.
Revised: Lemma E.

Original: [Equation]
Revised: [Equation]

Original: which imply
Revised: which imply

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .

Original: Lemma 4.1 is equivalent to this inequality by taking
Revised: Lemma 4.1 is equivalent to this inequality by taking

Original: C Θ = − log( δ ) > 0 .
Revised: C Θ = − log( δ ) > 0 .

Original: Therefore, the key derivation steps are in the proof of Lemma E.1 and
Revised: Therefore, the key derivation steps are in the proof of Lemma E.1 and

Original: Lemma E.2, as stated below.
Revised: Lemma E.2, as stated below.

Original: Lemma E.1.
Revised: Lemma E.1.

Original: Assume the assumptions in Lemma 4.1 are satisﬁed.
Revised: Assume the assumptions in Lemma 4.1 are satisﬁed.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: , α
Revised: , α

Original: [Equation]
Revised: [Equation]

Original: } be parameters in the constrained space Θ .
Revised: } be parameters in the constrained space Θ .

Original: Let { β t
Revised: Let { β t

Original: Tt =1 = PLISA θ
Revised: Tt =1 = PLISA θ

Original: ( Z 1: n ) and { β t (cid:48) } Tt =

Revised: ( Z 1: n ) and { β t (cid:48) } Tt =

Original: = PLISA θ (cid:48)
Revised: = PLISA θ (cid:48)

Original: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .
Revised: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .

Original: Assume
Revised: Assume

Original: K ≥ log
Revised: K ≥ log

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By triangular inequality,
Revised: By triangular inequality,

Original: [Equation]
Revised: [Equation]

Original: WLOG, assume α > α (cid:48) .
Revised: WLOG, assume α > α (cid:48) .

Original: Applying Lemma E.3 and Lemma E.4 to the above two terms on the right hand side
Revised: Applying Lemma E.3 and Lemma E.4 to the above two terms on the right hand side

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where c is the Lipschtiz constant.
Revised: where c is the Lipschtiz constant.

Original: c is a ﬁnite number because α is bounded and positive.
Revised: c is a ﬁnite number because α is bounded and positive.

Original: Applying it recursively
Revised: Applying it recursively

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we bound the term (cid:13)(cid:13)(cid:13) (cid:101) β kt − (cid:101)
Revised: Now we bound the term (cid:13)(cid:13)(cid:13) (cid:101) β kt − (cid:101)

Original: β k − 1 t (cid:13)(cid:13)(cid:13) 2 .
Revised: β k − 1 t (cid:13)(cid:13)(cid:13) 2 .

Original: By Lemma F.1 and Lemma F.2, (cid:13)(cid:13)(cid:13) (cid:101)
Revised: By Lemma F.1 and Lemma F.2, (cid:13)(cid:13)(cid:13) (cid:101)

Original: [Equation]
Revised: [Equation]

Original: kt
Revised: kt

Original: [Equation]
Revised: [Equation]

Original: k − 1 t
Revised: k − 1 t

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (By Lemma F.6)
Revised: (By Lemma F.6)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where c (cid:48) :
Revised: where c (cid:48) :

Original: = (cid:113) 2 ρ + (cid:113) 10 ρ − − ξ max B 2 δ − 1 .
Revised: = (cid:113) 2 ρ + (cid:113) 10 ρ − − ξ max B 2 δ − 1 .

Original: However, we can have a tighter bound for this term if t is larger than a certain integer.
Revised: However, we can have a tighter bound for this term if t is larger than a certain integer.

Original: More precisely, recall the deﬁnition that λ t = max { λ 0 ◦ σ ( η ) t , λ ∗ } .
Revised: More precisely, recall the deﬁnition that λ t = max { λ 0 ◦ σ ( η ) t , λ ∗ } .

Original: Since σ ( η max ) < 1 , the entries of λ 0 ◦ σ ( η ) t will decrease exponentially to reach the entry values of λ ∗ .
Revised: Since σ ( η max ) < 1 , the entries of λ 0 ◦ σ ( η ) t will decrease exponentially to reach the entry values of λ ∗ .

Original: Let t 0 =
Revised: Let t 0 =

Original: min { t : t ≥ 0 , λ ∗ = max { λ 0 ◦ σ ( η max ) t , λ ∗ } , ∀ λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t = λ ∗ .
Revised: min { t : t ≥ 0 , λ ∗ = max { λ 0 ◦ σ ( η max ) t , λ ∗ } , ∀ λ ∗ ∈ θ ∈ Θ( D M ) } be the number of blocks after which λ t = λ ∗ .

Original: We know t 0 is a small number because the decrease rate is linear.
Revised: We know t 0 is a small number because the decrease rate is linear.

Original: Now, the regularization parameters satisfy λ t = λ ∗ , ∀ t ≥ t 0 .
Revised: Now, the regularization parameters satisfy λ t = λ ∗ , ∀ t ≥ t 0 .

Original: Therefore, we can apply Lemma F. and Lemma F.2 again to obtain the following bound for all t ≥ t 0 :
Revised: Therefore, we can apply Lemma F. and Lemma F.2 again to obtain the following bound for all t ≥ t 0 :

Original: [Equation]
Revised: [Equation]

Original: Lemma F.6)
Revised: Lemma F.6)

Original: [Equation]
Revised: [Equation]

Original: Combining the two different bounds for (cid:13)(cid:13)(cid:13)
Revised: Combining the two different bounds for (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: β kt − (cid:101) β k−
Revised: β kt − (cid:101) β k

Original: [Equation]
Revised: [Equation]

Original:  (cid:13)(cid:13)(cid:13) 2 with Eq. 24, we have
Revised: [Equation] (cid:13)(cid:13)(cid:13) 2 with Eq. 30, we have

Original: [Equation]
Revised: [Equation]

Original: where C = cc (cid:48)
Revised: where C = cc (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We apply the above inequality recursively and obtain that
Revised: We apply the above inequality recursively and obtain that

Original: [Equation]
Revised: [Equation]

Original: Lemma E.2.
Revised: Lemma E.2.

Original: Assume the assumptions in Lemma 4.1 are satisﬁed.
Revised: Assume the assumptions in Lemma 4.1 are satisﬁed.

Original: Let θ
Revised: Let θ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: be parameters in the constrained space Θ .
Revised: be parameters in the constrained space Θ .

Original: Let { β t
Revised: Let { β t

Original: Tt =1 = PLISA θ
Revised: Tt =1 = PLISA θ

Original: ( Z 1: n ) and { β t (cid:48) } Tt =

Revised: ( Z 1: n ) and { β t (cid:48) } Tt =

Original: = PLISA θ (cid:48)
Revised: = PLISA θ (cid:48)

Original: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .
Revised: ( Z 1: n ) be the intermediate outputs of PLISA with different parameters θ and θ (cid:48) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 and C 1 , 2 , 4 are some constants.
Revised: (cid:112) 1 − α min ( ρ − − ξ max ) < 1 and C 1 , 2 , 4 are some constants.

Original: Proof.
Revised: Proof.

Original: Let { β t
Revised: Let { β t

Original: Tt =1 ∪ { (cid:101) β kt } Tt =

Revised: Tt =1 ∪ { (cid:101) β kt } Tt =

Original: Kk =1 and { β t (cid:48) }
Revised: Kk =1 and { β t (cid:48) }

Original: Tt =1 ∪ { (cid:101) β kt (cid:48) }
Revised: Tt =1 ∪ { (cid:101) β kt (cid:48) }

Original: Tt =

Revised: Tt =

Original: Kk =1 be the intermediate outputs of PLISA θ
Revised: Kk =1 be the intermediate outputs of PLISA θ

Original: ( Z 1: n ) and PLISA θ (cid:48)
Revised: ( Z 1: n ) and PLISA θ (cid:48)

Original: ( Z 1: n ) respectively.
Revised: ( Z 1: n ) respectively.

Original: By triangle inequality,
Revised: By triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma E
Revised: Applying Lemma E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma E.
Revised: Lemma E.

Original: [Equation]
Revised: [Equation]

Original: and Lemma E.6 to the above 3 terms on the right hand side, we obtain
Revised: and Lemma E.6 to the above 3 terms on the right hand side, we obtain

Original: [Equation]
Revised: [Equation]

Original: where C is some absolute constant
Revised: where C is some absolute constant

Original: [Equation]
Revised: [Equation]

Original: Term ( i ) .
Revised: Term ( i ) .

Original: By deﬁnition and triangular inequality,
Revised: By deﬁnition and triangular inequality,

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds by Assumption 3.1 and the bounded ﬁrst derivative of σ ( · ) .
Revised: The last inequality holds by Assumption 3.1 and the bounded ﬁrst derivative of σ ( · ) .

Original: Therefore, for some constants C 2 = 2 ρ + and ˜ C 1 = C 2 B 2 4 ,
Revised: Therefore, for some constants C 2 = 2 ρ + and ˜ C 1 = C 2 B 2 4 ,

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds because the value σ ( η max ) t t is bounded by some constant.
Revised: The last inequality holds because the value σ ( η max ) t t is bounded by some constant.

Original: Term ( ii ) .
Revised: Term ( ii ) .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original:  Plugging the bounds for (i)-(iii) into Eq. 25, we have
Revised: ≤ [Equation] Plugging the bounds for (i)-(iii) into Eq. 31, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: direct computation
Revised: direct computation

Original: [Equation]
Revised: [Equation]

Original: Recall that (cid:101)
Revised: Recall that (cid:101)

Original: β
Revised: β

Original: Kt = β t and (cid:101) β 0 t = β t − 1 .
Revised: Kt = β t and (cid:101) β 0 t = β t − 1 .

Original: We can apply the above inequality recursively and obtain that
Revised: We can apply the above inequality recursively and obtain that

Original: [Equation]
Revised: [Equation]

Original: Lemma E.3 (Robustness to β ) .
Revised: Lemma E.3 (Robustness to β ) .

Original: Assume ( Z 1: n , β ∗ ) ∈ P and P satisﬁes Assumption 3.1.
Revised: Assume ( Z 1: n , β ∗ ) ∈ P and P satisﬁes Assumption 3.1.

Original: Assume β , β (cid:48) satisfy (cid:107) β − β (cid:48)
Revised: Assume β , β (cid:48) satisfy (cid:107) β − β (cid:48)

Original: (cid:107) 

Revised: (cid:107)

Original: ≤ s = s ∗ + 2˜ s and α ≤ 1 ρ + .
Revised: ≤ s = s ∗ + 2˜ s and α ≤ 1 ρ + .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , w ,
Revised: , w ,

Original: [Equation]
Revised: [Equation]

Original: ; λ
Revised: ; λ

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: α )
Revised: α )

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since on the restricted subspace { β : (cid:107) β − β ∗ (cid:107) 

Revised: Since on the restricted subspace { β : (cid:107) β − β ∗ (cid:107)

Original: ≤ s ∗ + 2˜ s }
Revised: ≤ s ∗ + 2˜ s }

Original: [Equation]
Revised: [Equation]

Original: L λ is ( ρ − − ξ w ) -strongly convex and ρ + -smooth, by Lemma F
Revised: L λ is ( ρ − − ξ w ) -strongly convex and ρ + -smooth, by Lemma F

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq.
Revised: Combining this inequality with Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By the assumption that α ≤ 1 ρ + , the second term in the above inequality is non-positive.
Revised: By the assumption that α ≤ 1 ρ + , the second term in the above inequality is non-positive.

Original: Furthermore, this assumption also implies that 1 − 2 α ( ρ − − ξ w ) ρ
Revised: Furthermore, this assumption also implies that 1 − 2 α ( ρ − − ξ w ) ρ

Original: + ρ − − ξ w + ρ + ≥ 0 .
Revised: + ρ − − ξ w + ρ + ≥ 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma E.4 (Robustness to α ) .
Revised: Lemma E.4 (Robustness to α ) .

Original: With out loss of generalization, assume α ≥ α (cid:48) .
Revised: With out loss of generalization, assume α ≥ α (cid:48) .

Original: Denote β + := MPG ( β ; λ , w , α ) and β + (cid:48) :
Revised: Denote β + := MPG ( β ; λ , w , α ) and β + (cid:48) :

Original: =
Revised: =

Original: MPG ( β ; λ , w , α (cid:48) ) .
Revised: MPG ( β ; λ , w , α (cid:48) ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne the quadratic approximation function as
Revised: Deﬁne the quadratic approximation function as

Original: [Equation]
Revised: [Equation]

Original: Clearly,
Revised: Clearly,

Original: [Equation]
Revised: [Equation]

Original: Since ψ α, λ , w ( z , β ) is 1 α -strongly convex in z , and that β
Revised: Since ψ α, λ , w ( z , β ) is 1 α -strongly convex in z , and that β

Original: + :=
Revised: + :=

Original: MPG ( β ; λ , w , α ) is its optimal point,
Revised: MPG ( β ; λ , w , α ) is its optimal point,

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Taking z = β + (cid:48) in Eq.
Revised: Taking z = β + (cid:48) in Eq.

Original: 30 and z = β + in Eq.
Revised: 36 and z = β + in Eq.

Original: 31 yields
Revised: 37 yields

Original: [Equation]
Revised: [Equation]

Original: Summing the above two inequalities
Revised: Summing the above two inequalities

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Lemma E.5 (Robustness to λ ) .
Revised: Lemma E.5 (Robustness to λ ) .

Original: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)
Revised: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)

Original: =
Revised: =

Original: MPG ( β ; λ (cid:48) , w , α ) .
Revised: MPG ( β ; λ (cid:48) , w , α ) .

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where S :
Revised: 38) where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) )
Revised: = supp( β + ) ∪ supp( β + (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the quadratic approximation in Eq.
Revised: Recall the quadratic approximation in Eq.

Original: 44 and the deﬁnition that MPG ( β ; λ , w , α ) =
Revised: 50 and the deﬁnition that MPG ( β ; λ , w , α ) =

Original: arg min
Revised: arg min

Original: z ψ α, λ , w ( z , β ) .
Revised: z ψ α, λ , w ( z , β ) .

Original: By the 1 α -strongly convexity of ψ α, λ , w in z , it holds true for all z that
Revised: By the 1 α -strongly convexity of ψ α, λ , w in z , it holds true for all z that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking z = β + (cid:48) in Eq. 33, taking z = β + in Eq.
Revised: Taking z = β + (cid:48) in Eq. 39, taking z = β + in Eq.

Original: [Equation]
Revised: [Equation]

Original: and summing up the two inequalities, we have
Revised: and summing up the two inequalities, we have

Original: [Equation]
Revised: [Equation]

Original: Assumption B.1)
Revised: Assumption B.1)

Original: [Equation]
Revised: [Equation]

Original: where S :
Revised: where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) ) .
Revised: = supp( β + ) ∪ supp( β + (cid:48) ) .

Original: Dividing both side by (cid:107) β + (cid:48)
Revised: Dividing both side by (cid:107) β + (cid:48)

Original: − β + (cid:107) 2 draws the conclusion.
Revised: − β + (cid:107) 2 draws the conclusion.

Original: Lemma E.6 (Robustness to w ) .
Revised: Lemma E.6 (Robustness to w ) .

Original: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)
Revised: Let β + = MPG ( β ; λ , w , α ) and β + (cid:48)

Original: =
Revised: =

Original: MPG ( β ; λ , w (cid:48) , α ) .
Revised: MPG ( β ; λ , w (cid:48) , α ) .

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where S :
Revised: where S :

Original: = supp( β + ) ∪ supp( β + (cid:48) ) and C is some absolute constant.
Revised: = supp( β + ) ∪ supp( β + (cid:48) ) and C is some absolute constant.

Original: Proof.
Revised: Proof.

Original: Recall the quadratic approximation in Eq.
Revised: Recall the quadratic approximation in Eq.

Original: 44 and the deﬁnition that MPG ( β ; λ , w , α ) =
Revised: 50 and the deﬁnition that MPG ( β ; λ , w , α ) =

Original: arg min
Revised: arg min

Original: z ψ α, λ , w ( z , β ) .
Revised: z ψ α, λ , w ( z , β ) .

Original: By the 1 α -strongly convexity of ψ α, λ , w in β , it holds true for all z in the restricted subspace that
Revised: By the 1 α -strongly convexity of ψ α, λ , w in β , it holds true for all z in the restricted subspace that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Taking z = β + (cid:48) in Eq.
Revised: Taking z = β + (cid:48) in Eq.

Original: [Equation]
Revised: [Equation]

Original: taking z = β + in Eq.
Revised: taking z = β + in Eq.

Original: [Equation]
Revised: [Equation]

Original: and summing up the two inequalities
Revised: and summing up the two inequalities

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that q (cid:48) w ( λ j , β j )
Revised: Note that q (cid:48) w ( λ j , β j )

Original: = (cid:80) 

Revised: = (cid:80)

Original: i =1 exp( w i )
Revised: i =1 exp( w i )

Original: Z ( w ) q
Revised: Z ( w ) q

Original: ( i ) (cid:48) ( λ j , β j ) where Z ( w ) =
Revised: ( i ) (cid:48) ( λ j , β j ) where Z ( w ) =

Original: (cid:80) 3 i =1 exp( w i ) .
Revised: (cid:80) 3 i =1 exp( w i ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: 3 (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )
Revised: 3 (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )

Original: Z ( w ) − exp( w (cid:48) i )
Revised: Z ( w ) − exp( w (cid:48) i )

Original: Z ( w (cid:48) ) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) q
Revised: Z ( w (cid:48) ) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) q

Original: ( i ) (cid:48) ( λ j , β j ) (cid:12)(cid:12)(cid:12) (by Assumption B.1) ≤ 3 (cid:88)
Revised: ( i ) (cid:48) ( λ j , β j ) (cid:12)(cid:12)(cid:12) (by Assumption B.1) ≤ 3 (cid:88)

Original: i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )
Revised: i = (cid:12)(cid:12)(cid:12)(cid:12) exp( w i )

Original: Z ( w ) − exp( w (cid:48) i )
Revised: Z ( w ) − exp( w (cid:48) i )

Original: Z ( w (cid:48) ) (cid:12)(cid:12)(cid:12)(cid:12) λ j .
Revised: Z ( w (cid:48) ) (cid:12)(cid:12)(cid:12)(cid:12) λ j .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: for some constant C.
Revised: for some constant C

Original: [Equation]
Revised: [Equation]

Original:  This section supplies some fundamental results for the modiﬁed proximal gradient algorithm.
Revised: [Equation] This section supplies some fundamental results for the modiﬁed proximal gradient algorithm.

Original: All the results in this section are based on the following assumption and notations.
Revised: All the results in this section are based on the following assumption and notations.

Original: Assumption F.1.
Revised: Assumption F.1.

Original: Assume P satisﬁes Assumption 3.1 and consider a problem ( Z n , β ∗ ) ∈ P .
Revised: Assume P satisﬁes Assumption 3.1 and consider a problem ( Z n , β ∗ ) ∈ P .

Original: Assume we have a penalty function P ( λ , β ) whose concave component Q ( λ , β )
Revised: Assume we have a penalty function P ( λ , β ) whose concave component Q ( λ , β )

Original: =
Revised: =

Original: P ( λ , β ) −(cid:107) λ ◦ β (cid:107)satisﬁes the regularization conditions in Assumption B
Revised: P ( λ , β ) −(cid:107) λ ◦ β (cid:107)satisﬁes the regularization conditions in Assumption B

Original: [Equation]
Revised: [Equation]

Original: 1 with a Lipschitz constant ξ for the condition (a).
Revised: 1 with a Lipschitz constant ξ for the condition (a).

Original: We adopt the following notations for the statements in this sections.
Revised: We adopt the following notations for the statements in this sections.

Original: [Equation]
Revised: [Equation]

Original: Lemma F.1 (Contraction of MPG) .
Revised: Lemma F.1 (Contraction of MPG) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:

Original: [Equation]
Revised: [Equation]

Original: Assume (cid:107) ( β k ) S ∗ (cid:107) 

Revised: Assume (cid:107) ( β k ) S ∗ (cid:107)

Original: ≤ ˜ s .
Revised: ≤ ˜ s .

Original: Then (cid:18)
Revised: Then (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where δ α ( β ) :
Revised: where δ α ( β ) :

Original: =
Revised: =

Original: α − 1 ( β − T α λ ( β − α ∇ L λ ( β ))) .
Revised: α − 1 ( β − T α λ ( β − α ∇ L λ ( β ))) .

Original: It is easy to observe that δ α ( β ) ∈ ∇ L λ ( β
Revised: It is easy to observe that δ α ( β ) ∈ ∇ L λ ( β

Original: [Equation]
Revised: [Equation]

Original: + λ ◦ ∂
Revised: + λ ◦ ∂

Original: [Equation]
Revised: [Equation]

Original: β − α δ α
Revised: β − α δ α

Original: [Equation]
Revised: [Equation]

Original: β )
Revised: β )

Original: [Equation]
Revised: [Equation]

Original: By RSS in Lemma B.1, it holds that
Revised: By RSS in Lemma B.1, it holds that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by RSC in Lemma B.1, for any z in the space { (cid:13)(cid:13)
Revised: Furthermore, by RSC in Lemma B.1, for any z in the space { (cid:13)(cid:13)

Original: z − β k − 1 (cid:13)(cid:13) 

Revised: z − β k − 1 (cid:13)(cid:13)

Original: ≤ s ∗ + 2˜ s } ,
Revised: ≤ s ∗ + 2˜ s } ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Denote v = δ α ( β k − 1 ) − ∇ L λ ( β k − 1 ) .
Revised: Denote v = δ α ( β k − 1 ) − ∇ L λ ( β k − 1 ) .

Original: We have v ∈ λ ◦ ∂ (cid:13)(cid:13)
Revised: We have v ∈ λ ◦ ∂ (cid:13)(cid:13)

Original: β k − 1 − α δ α ( β k − 1 ) (cid:13)(cid:13) 1 since δ α ( β k −
Revised: β k − 1 − α δ α ( β k − 1 ) (cid:13)(cid:13) 1 since δ α ( β k −

Original: [Equation]
Revised: [Equation]

Original: ∈ ∇ L λ (
Revised: ∈ ∇ L λ (

Original: [Equation]
Revised: [Equation]

Original: + λ
Revised: + λ

Original: [Equation]
Revised: [Equation]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: β k − 1 − α δ α
Revised: β k − 1 − α δ α

Original: ( β k − 1 ) (cid:13)(cid:13) 1 .
Revised: ( β k − 1 ) (cid:13)(cid:13) 1 .

Original: By the convexity of (cid:107)·(cid:107) λ , 1 , (cid:13)(cid:13)
Revised: By the convexity of (cid:107)·(cid:107) λ , 1 , (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq.
Revised: Combining this inequality with Eq.

Original: 41, we have
Revised: 47, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking z = β k − 1 in Eq.
Revised: Taking z = β k − 1 in Eq.

Original: 43 implies that
Revised: 49 implies that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.2 (Convergence of modiﬁed proximal gradient) .
Revised: Lemma F.2 (Convergence of modiﬁed proximal gradient) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the modiﬁed proximal gradient updates:

Original: [Equation]
Revised: [Equation]

Original: with a step size α in (0 , 1 ρ + ] .
Revised: with a step size α in (0 , 1 ρ + ] .

Original: Assume (cid:107) ( β k ) S ∗ (cid:107) 0 ≤ ˜ s .
Revised: Assume (cid:107) ( β k ) S ∗ (cid:107) 0 ≤ ˜ s .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the quadratic approximation function
Revised: Consider the quadratic approximation function

Original: [Equation]
Revised: [Equation]

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Since ψ α, λ ( z , β k − 1 ) is 1 α -strongly convex with respect to z
Revised: Since ψ α, λ ( z , β k − 1 ) is 1 α -strongly convex with respect to z

Original: [Equation]
Revised: [Equation]

Original: and that β k is its optimal point
Revised: and that β k is its optimal point

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Note that by Lemma B.1,
Revised: Note that by Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Similarly, by Lemma B.1,
Revised: Similarly, by Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two results with Eq.
Revised: Combining the above two results with Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Taking z = c (cid:98) β λ
Revised: Taking z = c (cid:98) β λ

Original: + (1 − c ) β k − 1 for some c ∈ [0 , 1] , then φ λ ( β k ) ≤
Revised: + (1 − c ) β k − 1 for some c ∈ [0 , 1] , then φ λ ( β k ) ≤

Original: [Equation]
Revised: [Equation]

Original: Taking c
Revised: Taking c

Original: = α ( ρ − − ξ ) , it implies φ
Revised: = α ( ρ − − ξ ) , it implies φ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assume α ≤ 1 ρ + .
Revised: Assume α ≤ 1 ρ + .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Taking z = (cid:98) β λ in Eq.
Revised: Taking z = (cid:98) β λ in Eq.

Original: 46 implies φ λ
Revised: 52 implies φ λ

Original: [Equation]
Revised: [Equation]

Original: β k ) ≤ φ
Revised: β [Equation] ) ≤ φ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: α −
Revised: α −

Original: [Equation]
Revised: [Equation]

Original: ρ − − ξ ) (cid:107) (cid:98) β 
Revised: ρ − − ξ ) (cid:107) (cid:98) β λ − β

Original: λ − β k − 1 (cid:107) 22 − 1 [-
-] α (cid:107)
Revised: [Equation] − 1 (cid:107) 22 − 1 [-
-]2 α (cid:107)

Original: [Equation]
Revised: [Equation]

Original: By the optimality of (cid:98) β λ , we have
Revised: By the optimality of (cid:98) β λ , we have

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we derive the upper bound for ω λ ( β k ) .
Revised: we derive the upper bound for ω λ ( β k ) .

Original: Since β k = arg min z ψ α, λ ( z , β k − 1 ) , by the optimality condition, for each j = 1 , · · · , d
Revised: Since β k = arg min z ψ α, λ ( z , β k − 1 ) , by the optimality condition, for each j = 1 , · · · , d

Original: [Equation]
Revised: [Equation]

Original: there exists a subgradient ξ j
Revised: there exists a subgradient ξ j

Original: [Equation]
Revised: [Equation]

Original: ∂ (cid:12)(cid:12) β kj (cid:12)(cid:12)
Revised: ∂ (cid:12)(cid:12) β kj (cid:12)(cid:12)

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of ω λ and combining it with the above equality,
Revised: By the deﬁnition of ω λ and combining it with the above equality,

Original: [Equation]
Revised: [Equation]

Original: by RSS) ≤
Revised: by RSS) ≤

Original: [Equation]
Revised: [Equation]

Original: By Lemma F.1,
Revised: By Lemma F.1,

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: − β k )
Revised: − β k )

Original: [Equation]
Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original: k )) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.3 (Statistical (cid:96) 2 error) .
Revised: Lemma F.3 (Statistical (cid:96) 2 error) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: If the following conditions are satisﬁed
Revised: If the following conditions are satisﬁed

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the restricted strong convexity of L λ in Lemma B.1,
Revised: By the restricted strong convexity of L λ in Lemma B.1,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the term (i),
Revised: For the term (i),

Original: [Equation]
Revised: [Equation]

Original: S ∗ , 1 − (cid:107) β S∗ (cid:107) λ
Revised: S ∗ , 1 − [Equation] S

Original:  S ∗ ,= (cid:107) ( β − β ∗ )
Revised: [Equation] (cid:107) λ S ∗ ,= (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 − (cid:107) ( β − β ∗ )
Revised: S ∗ , 1 − (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 .
Revised: S ∗ , 1 .

Original: For the term (ii), −∇
Revised: For the term (ii), −∇

Original: [Equation]
Revised: [Equation]

Original: β
Revised: −∇

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: −∇ [Equation] β 
Revised: ( β − β ∗ )

Original: [Equation] ( β − β ∗ [Equation] − ∇ Q λ ( β ∗ 
Revised: (By Eq. 55) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ ) (cid:62) ( β − β ∗ ) (since q (cid:48)

Original: [Equation] ( β − β ∗ ) (By Eq. 49) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ ) (cid:62) ( β − β ∗ ) (since q (cid:48) λ j (0) = 0 )
Revised: λ j (0) = 0 )

Original: = 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ S ∗ ) (cid:62) ( β − β ∗ )
Revised: = 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 − ∇ Q λ ( β ∗ S ∗ ) (cid:62) ( β − β ∗ )

Original: S ∗ (since (cid:12)(cid:12)(cid:12) q (cid:48)
Revised: S ∗ (since (cid:12)(cid:12)(cid:12) q (cid:48)

Original: λ j ( β ) (cid:12)(cid:12)(cid:12) ≤ λ j ) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 + (cid:107) ( β − β ∗ )
Revised: λ j ( β ) (cid:12)(cid:12)(cid:12) ≤ λ j ) ≤ 1 8 (cid:107) β − β ∗ (cid:107) λ , 1 + (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ S ∗ ,=8 (cid:107) ( β − β ∗ )
Revised: S ∗ (cid:107) λ S ∗ ,=8 (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 + 1 8 (cid:107) ( β − β ∗ )
Revised: S ∗ , 1 + 1 8 (cid:107) ( β − β ∗ )

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 .
Revised: S ∗ , 1 .

Original: Combining the upper bounds of term (i) and term (ii) with Eq.
Revised: Combining the upper bounds of term (i) and term (ii) with Eq.

Original: 50, we have
Revised: 56, we have

Original: [Equation]
Revised: [Equation]

Original: Let a ≥ 0 be a constant.
Revised: Let a ≥ 0 be a constant.

Original: If (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ

Original: S ∗ , 1 ≤
Revised: S ∗ , 1 ≤

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ∗ (cid:107) 22 , then
Revised: ∗ (cid:107) 22 , then

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ

Original: S ∗
Revised: S ∗

Original: [Equation]
Revised: [Equation]

Original: 1 >
Revised: 1 >

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:107) β − β ∗ (cid:107) 2 ≤
Revised: which implies (cid:107) β − β ∗ (cid:107) 2 ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: + 174 (cid:1)
Revised: + 174 (cid:1)

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: ξ .
Revised: ξ .

Original: Combining the two cases, we know that
Revised: Combining the two cases, we know that

Original: [Equation]
Revised: [Equation]

Original: Taking a = 7 it is easy to derive that
Revised: Taking a = 7 it is easy to derive that

Original: [Equation]
Revised: [Equation]

Original: Lemma F.4 (Retain in the restricted space) .
Revised: Lemma F.4 (Retain in the restricted space) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: Let β 0 , · · · , β k be a sequence of vectors obtained by the following update:
Revised: Let β 0 , · · · , β k be a sequence of vectors obtained by the following update:

Original: [Equation]
Revised: [Equation]

Original: with a step sizes α in [ α min , 1 ρ + ] .
Revised: with a step sizes α in [ α min , 1 ρ + ] .

Original: Assume the following conditions:
Revised: Assume the following conditions:

Original: [Equation]
Revised: [Equation]

Original: where κ :=
Revised: where κ :=

Original: ρ + ρ − − ξ .
Revised: ρ + ρ − − ξ .

Original: Then for all k = 1 , · · · , K ,
Revised: Then for all k = 1 , · · · , K ,

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: ≤ ˜ s, and
Revised: ≤ ˜ s, and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that β k = T α λ (cid:0) ¯ β k (cid:1) where
Revised: Recall that β k = T α λ (cid:0) ¯ β k (cid:1) where

Original: [Equation]
Revised: [Equation]

Original: To show (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13) 0 ≤ ˜ s , we need to prove that, for j ∈ S ∗ , the number of j ’s such that | ¯ β kj | > αλ j is no more than ˜ s .
Revised: To show (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13) 0 ≤ ˜ s , we need to prove that, for j ∈ S ∗ , the number of j ’s such that | ¯ β kj | > αλ j is no more than ˜ s .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, ∀ c ∈ [0 , 1] ,
Revised: Therefore, ∀ c ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: 53) where Card () represents the size of the set.
Revised: 59) where Card () represents the size of the set.

Original: For the term in Eq.52,
Revised: For the term in Eq.

Original: Card
Revised: Card

Original: [Equation]
Revised: [Equation]

Original: Bounding (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) 1 .
Revised: Bounding (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) 1 .

Original: Following the derivation of Eq.
Revised: Following the derivation of Eq.

Original: 51, we can obtain
Revised: 57, we can obtain

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Let a ≥ 0 be a constant.
Revised: Let a ≥ 0 be a constant.

Original: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ

Original: S ∗
Revised: S ∗

Original: [Equation]
Revised: [Equation]

Original: ≤ a ρ − − ξ (cid:107) λ
Revised: ≤ a ρ − − ξ (cid:107) λ

Original: S ∗ (cid:107) 22 , then
Revised: S ∗ (cid:107) 22 , then

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ
Revised: If (cid:107) ( β k − 1 − β ∗ ) S ∗ (cid:107) λ

Original: S ∗ , 1 > a ρ − − ξ (cid:107) λ
Revised: S ∗ , 1 > a ρ − − ξ (cid:107) λ

Original: S ∗ (cid:107) 22 , then
Revised: S ∗ (cid:107) 22 , then

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases yields
Revised: Combining the above two cases yields

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is obtained by taking a = 25534 .
Revised: where the last inequality is obtained by taking a = 25534 .

Original: Therefore, the term in Eq.
Revised: Therefore, the term in Eq.

Original: 52 can be bounded by
Revised: 58 can be bounded by

Original: [Equation]
Revised: [Equation]

Original: Now we bound the term in Eq.
Revised: Now we bound the term in Eq.

Original: 53 as follows
Revised: 59 as follows

Original: [Equation]
Revised: [Equation]

Original: Denote the set as S (cid:48) :
Revised: Denote the set as S (cid:48) :

Original: = { j ∈ S ∗ :
Revised: = { j ∈ S ∗ :

Original: [Equation]
Revised: [Equation]

Original: > (1 − c ) · 78 λ j } and its size as s (cid:48) := | S (cid:48) | .
Revised: > (1 − c ) · 78 λ j } and its size as s (cid:48) := | S (cid:48) | .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds because L λ is ( ρ + ) -smooth in the restricted subspace by Lemma B.1.
Revised: The last inequality holds because L λ is ( ρ + ) -smooth in the restricted subspace by Lemma B.1.

Original: Applying Lemma F.
Revised: Applying Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Combining the above bounds for the terms in Eq.
Revised: Combining the above bounds for the terms in Eq.

Original: 52 and Eq.
Revised: 58 and Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: 49 cα (cid:19) ρ − − ξ + 49(1 − c )(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ (cid:32) α min ( ρ − − ξ ) +(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ ˜ s
Revised: 49 cα (cid:19) ρ − − ξ + 49(1 − c )(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ (cid:32) α min ( ρ − − ξ ) +(cid:18) ρ + ρ − − ξ (cid:19) 2 (cid:33) (cid:18) (cid:107) λ S ∗ (cid:107)min( λ S ∗ ) (cid:19)≤ ˜ s

Original: The second last inequality is obtained by taking c = 57 .
Revised: The second last inequality is obtained by taking c = 57 .

Original: Finally, if we can show φ λ ( β k ) − φ λ ( β ∗ ) ≤ 21 / 2 ρ − − ξ (cid:107) λ S ∗ (cid:107) 22 , then we can show by math induction that (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13) 

Revised: Finally, if we can show φ λ ( β k ) − φ λ ( β ∗ ) ≤ 21 / 2 ρ − − ξ (cid:107) λ S ∗ (cid:107) 22 , then we can show by math induction that (cid:13)(cid:13) ( β k ) S ∗ (cid:13)(cid:13)

Original: ≤ ˜ s for all k .
Revised: ≤ ˜ s for all k .

Original: Taking z = β k − 1 in Eq.
Revised: Taking z = β k − 1 in Eq.

Original: 46, we have
Revised: 52, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Lemma F.5 (Statistical error) .
Revised: Lemma F.5 (Statistical error) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then the following relations hold true.
Revised: then the following relations hold true.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since (cid:107) β − β ∗ (cid:107) 

Revised: Since (cid:107) β − β ∗ (cid:107)

Original: ≤ s ∗ + 2˜ s , then by the RSC and RSS in Lemma B.1,
Revised: ≤ s ∗ + 2˜ s , then by the RSC and RSS in Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ ( β ) .
Revised: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ ( β ) .

Original: Adding and  β − β ∗ ) (cid:62)  ) to the right-hand side of 
Revised: Adding and ( β − β ∗ ) (cid:62) ( ξ ) to the right-hand side of Eq.

Original: 54 yields
Revised: 60 yields

Original: [Equation]
Revised: [Equation]

Original: Now we bound the terms (i)-(iii).
Revised: Now we bound the terms (i)-(iii).

Original: Bound (i) using ω λ ( β ) .
Revised: Bound (i) using ω λ ( β ) .

Original: Since ξ attains the minimum in ω λ ( β ) , by deﬁnition,
Revised: Since ξ attains the minimum in ω λ ( β ) , by deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Bound (ii).
Revised: Bound (ii).

Original: Note that L λ =
Revised: Note that L λ =

Original: L n + Q λ .
Revised: L n + Q λ .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Since λ j ≥ 8 |
Revised: Since λ j ≥ 8 |

Original: [ L n ( β ∗ )]
Revised: [ L n ( β ∗ )]

Original: j | , we have
Revised: j | , we have

Original: [Equation]
Revised: [Equation]

Original: Since q (cid:48) λ j (0) = 0 and q (cid:48) λ j ( β ) ≤ λ j (Assumption B.1), it holds true that
Revised: Since q (cid:48) λ j (0) = 0 and q (cid:48) λ j ( β ) ≤ λ j (Assumption B.1), it holds true that

Original: [Equation]
Revised: [Equation]

Original: Therefore, |
Revised: Therefore, |

Original: [Equation]
Revised: [Equation]

Original: ii
Revised: ii

Original: [Equation]
Revised: [Equation]

Original: Bound (iii) by the deﬁnition of subgradient.
Revised: Bound (iii) by the deﬁnition of subgradient.

Original: [Equation]
Revised: [Equation]

Original: By Holder’s inequality and the fact
Revised: By Holder’s inequality and the fact

Original: [Equation]
Revised: [Equation]

Original: Since ξ ∈ ∂ (cid:107) β (cid:107) 1 and that β ∗ S ∗ = 0 ,
Revised: Since ξ ∈ ∂ (cid:107) β (cid:107) 1 and that β ∗ S ∗ = 0 ,

Original: [Equation]
Revised: [Equation]

Original: Combining the above three equations, it holds true that
Revised: Combining the above three equations, it holds true that

Original: [Equation]
Revised: [Equation]

Original: Combine (i)-(iii).
Revised: Combine (i)-(iii).

Original: Combining the bounds for (i)-(iii), it holds true that
Revised: Combining the bounds for (i)-(iii), it holds true that

Original: [Equation]
Revised: [Equation]

Original: Using the fact that (cid:107) ( β − β
Revised: Using the fact that (cid:107) ( β − β

Original: [Equation]
Revised: [Equation]

Original: S ∗ (cid:107) λ
Revised: S ∗ (cid:107) λ

Original: S ∗ , 1 ≤ (cid:107) λ
Revised: S ∗ , 1 ≤ (cid:107) λ

Original: S ∗ (cid:107) 2 (cid:107) ( β − β ∗ ) S ∗ (cid:107) 2 , it implies the ﬁrst conclusion:
Revised: S ∗ (cid:107) 2 (cid:107) ( β − β ∗ ) S ∗ (cid:107) 2 , it implies the ﬁrst conclusion:

Original: [Equation]
Revised: [Equation]

Original: Now we prove the objective value.
Revised: Now we prove the objective value.

Original: Since L λ is convex on β , β ∗ and (cid:107)·(cid:107) 1 is convex,
Revised: Since L λ is convex on β , β ∗ and (cid:107)·(cid:107) 1 is convex,

Original: [Equation]
Revised: [Equation]

Original: , which implies
Revised: , which implies

Original: [Equation]
Revised: [Equation]

Original: Now we bound the norm (cid:107) β − β ∗ (cid:107) λ , 1 as follows.
Revised: Now we bound the norm (cid:107) β − β ∗ (cid:107) λ , 1 as follows.

Original: By triangle inequality,
Revised: By triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: To bound (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ S ∗ , 1 , moving the term (cid:107) ( β
Revised: To bound (cid:107) ( β − β ∗ ) S ∗ (cid:107) λ S ∗ , 1 , moving the term (cid:107) ( β

Original: [Equation]
Revised: [Equation]

Original: ) S
Revised: ) S

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: , 1 in Eq.
Revised: , 1 in Eq.

Original: 56 to the left hand side yields
Revised: 62 to the left hand side yields

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging it into Eq.
Revised: Plugging it into Eq.

Original: 59, we have
Revised: 65, we have

Original: [Equation]
Revised: [Equation]

Original: (by Eq. 57) ≤
Revised: (by Eq. 63) ≤

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Combining it with Eq.
Revised: Combining it with Eq.

Original: 58, we have
Revised: 64, we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we derive the bound for the term
Revised: Finally, we derive the bound for the term

Original: φ λ ( β ) − φ λ ( (cid:98) β λ ) .
Revised: φ λ ( β ) − φ λ ( (cid:98) β λ ) .

Original: Since (cid:107) β − (cid:98) β λ (cid:107) 

Revised: Since (cid:107) β − (cid:98) β λ (cid:107)

Original: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then
Revised: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we only need to bound the term (cid:107) β ∗ − (cid:98)
Revised: Now we only need to bound the term (cid:107) β ∗ − (cid:98)

Original: β λ (cid:107) λ , 1 .
Revised: β λ (cid:107) λ , 1 .

Original: We can derive its bound following the same steps as we derive Eq. 61 and using the fact that ω λ ( (cid:98) β λ ) = 0 , which will imply that
Revised: We can derive its bound following the same steps as we derive Eq. 67 and using the fact that ω λ ( (cid:98) β λ ) = 0 , which will imply that

Original: [Equation]
Revised: [Equation]

Original: Plugging it into Eq.
Revised: Plugging it into Eq.

Original: 62, we have
Revised: 68, we have

Original: [Equation]
Revised: [Equation]

Original: Assuming (cid:15) ≤ 1 / 2 , then
Revised: Assuming (cid:15) ≤ 1 / 2 , then

Original: [Equation]
Revised: [Equation]

Original: Lemma F.6 (Optimality when λ decreases) .
Revised: Lemma F.6 (Optimality when λ decreases) .

Original: Assume Assumption F.1.
Revised: Assume Assumption F.1.

Original: If
Revised: If

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with η j
Revised: with η j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 65) then
Revised: 71) then

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ρ − − ξ .
Revised: ρ − − ξ .

Original: If (cid:15) t − 1 ≤ 1 /
Revised: If (cid:15) t − 1 ≤ 1 /

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the deﬁnition of ω λ t − 1 ( β ) .
Revised: Recall the deﬁnition of ω λ t − 1 ( β ) .

Original: [Equation]
Revised: [Equation]

Original: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ t − 1 ( β ) .
Revised: Let ξ ∈ ∂ (cid:107) β (cid:107) 1 be the subgradient that attains the minimum in ω λ t − 1 ( β ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Now we consider λ t =
Revised: Now we consider λ t =

Original: η ◦ λ t − 1 .
Revised: η ◦ λ t − 1 .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Using the fact that (cid:107) β − β (cid:48) (cid:107) λ t , 1 ≥ 0 .
Revised: Using the fact that (cid:107) β − β (cid:48) (cid:107) λ t , 1 ≥ 0 .

Original: 9 (cid:107) β − β (cid:48)
Revised: 9 (cid:107) β − β (cid:48)

Original: (cid:107) λ t − 1 , 1 , term (i) can be bounded as follows.
Revised: (cid:107) λ t − 1 , 1 , term (i) can be bounded as follows.

Original: [Equation]
Revised: [Equation]

Original: Using the fact | ξ j | ≤ 1 and the fact that [ λ t −
Revised: Using the fact | ξ j | ≤ 1 and the fact that [ λ t −

Original: [Equation]
Revised: [Equation]

Original: [ λ
Revised: [ λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , term (ii) can be bounded as follows.
Revised: , term (ii) can be bounded as follows.

Original: [Equation]
Revised: [Equation]

Original: (cid:18)0 . 9 −(cid:19)
Revised: (cid:18)0 . 9 −(cid:19)

Original: =0 .
Revised: =0 .

Original: 9 − 1 .
Revised: 9 − 1 .

Original: Term (iii) is bounded similarly as term (ii).
Revised: Term (iii) is bounded similarly as term (ii).

Original: Since (cid:12)(cid:12)(cid:12) q λ j ( β ) − q λ (cid:48)
Revised: Since (cid:12)(cid:12)(cid:12) q λ j ( β ) − q λ (cid:48)

Original: j ( β ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12) λ
Revised: j ( β ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12) λ

Original: j − λ (cid:48)
Revised: j − λ (cid:48)

Original: j (cid:12)(cid:12) , we have (iii) ≤
Revised: j (cid:12)(cid:12) , we have (iii) ≤

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: [ λ
Revised: [ λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining the bounds for (i)-(iii), we have
Revised: Combining the bounds for (i)-(iii), we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst part of the proof is ﬁnished.
Revised: The ﬁrst part of the proof is ﬁnished.

Original: Now we bound the term φ λ t ( β ) − φ λ t ( (cid:98) β λ t ) .
Revised: Now we bound the term φ λ t ( β ) − φ λ t ( (cid:98) β λ t ) .

Original: Since (cid:107) β − (cid:98) β λ (cid:107) 

Revised: Since (cid:107) β − (cid:98) β λ (cid:107)

Original: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then
Revised: ≤ s ∗ + 2˜ s and φ λ is convex on the restricted space, then

Original: [Equation]
Revised: [Equation]

Original: ) ≤ ω λ
Revised: ) ≤ ω λ

Original: [Equation]
Revised: [Equation]

Original: β )
Revised: β )

Original: [Equation]
Revised: [Equation]

Original: β −
Revised: β −

Original: [Equation]
Revised: [Equation]

Original: Following the same way we derive Eq. 60, we can obtain that
Revised: Following the same way we derive Eq. 66, we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using the same arguments for the term (cid:107) β ∗ − (cid:98) β λ t (cid:107) λ t , 1 and using the fact that ω λ t ( (cid:98) β λ t ) ≤ 0 · λ t , we obtain that
Revised: Using the same arguments for the term (cid:107) β ∗ − (cid:98) β λ t (cid:107) λ t , 1 and using the fact that ω λ t ( (cid:98) β λ t ) ≤ 0 · λ t , we obtain that

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with Eq. 66 and Eq.
Revised: Combining this inequality with Eq. 72 and Eq.

Original: 68, we have
Revised: 74, we have

Original: [Equation]
Revised: [Equation]

Original: Assuming (cid:15) t − 1 ≤ 1 / 4 , then
Revised: Assuming (cid:15) t − 1 ≤ 1 / 4 , then

Original: [Equation]
Revised: [Equation]

Original: Lemma F.7 (Coercivity of the gradient) .
Revised: Lemma F.7 (Coercivity of the gradient) .

Original: [Lemma 3.11 in Bubeck (2014)] Let f be a -strongly convex b -smooth and on B ⊆ R d .
Revised: [Lemma 3.11 in Bubeck (2014)] Let f be a -strongly convex b -smooth and on B ⊆ R d .

Original: Then for all β , β (cid:48) ∈ B , one has
Revised: Then for all β , β (cid:48) ∈ B , one has

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: ) , β − β
Revised: ) , β − β

Revised: [Equation]

Revised: G D ETAILS OF S ECTION 4.1: E XTENSION T O U NSUPERVISED L EARNING - TO - LEARN S ETTING

Revised: In addition to the assumptions in the supervised setting, in this unsupervised setting, we assume for each estimation problem, both L n 1 ( Z 1: n 1 ) and L n 2 ( Z 1: n 2 ) satisfy condition (b) and (c) in Assumption 3.1.

Revised: Similar to the supervised setting, the generalization error of the optimizer θ ∗ U can be bounded by

Original: [Equation]
Revised: [Equation]

Revised: However, in the unsupervised setting, the minimizer θ ∗ U only guarantees that unsupervised training loss function 1 m (cid:80) mi =

Revised: [Equation]

Revised: L n 2 (cid:16) Z

Revised: ( i ) 1: n 2 , β T ( Z ( i ) 1: n 1 ;

Revised: θ ∗ U ) (cid:17) is small.

Revised: It requires further derivations to show that the corresponding supervised training loss in Eq.

Revised: 76 is small, too.

Revised: Therefore, in the following, we bridge the gap between the supervised training loss and the unsupervised training loss.

Original: {+Based on the proof of Theorem C.1, we know that (cid:107) β T ( Z ( i ) 1: n 1 ; θ ∗ U ) | S ∗ (cid:107) 
+}
Revised: {+Based on the proof of Theorem C.1, we know that (cid:107) β T ( Z ( i ) 1: n 1 ; θ ∗ U ) | S ∗ (cid:107) 
+}

Revised: ≤ ˜ s .

Revised: Therefore, we can apply the restricted strong convexity of L n 2 to obtain the following inequality for β T = β T ( Z ( i ) 1: n 1 ; θ ∗ U ) , β ∗ = β ∗ ( i ) , and Z =

Revised: Z ( i ) 1: n 2 for each i = 1 , · · · , m .

Revised: [Equation]

Revised: Aggregating this inequality for i = 1 , · · · , m , we can obtain the following inequality that bounds the supervised training loss using the unsupervised loss.

Revised: [Equation]

Revised: The right-hand-side of this inequality consists of a statistical error and the unsupervised training error.

Revised: Finally, to characterize how small the unsupervised training error can be

Revised: [Equation]

Revised: we can combine the following inequality (by restricted strongly smooth) with the result of Theorem 3.1.

Revised: [Equation]

Revised: Overall, the generalization error can be bounded by

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: G E XPERIMENT D ETAILS G. I MPLEMENTATION
Revised: [Table] [Equation] [Table] H D ETAILS OF S YNTHETIC E XPERIMENTS H. A G ENTLE R EVIEW OF L INEAR R EGRESSION AND P RECISION E STIMATION H.1. S PARSE L INEAR R EGRESSION

Original: ALISTA : We follow the implementation in Liu et al. (2019a).
Revised: A sparse linear regression model reads [Equation] where β ∗ is a sparse vector and (cid:15) is Gaussian noise. Given n observations Z 1: n := { ( x 1 , y 1 ) , · · · , ( x n , y n ) } , the goal is to estimate the vector β ∗ .

Revised: In our method, we use the the least-square error to deﬁne the empirical loss function.

Revised: [Equation]

Revised: H.1. S PARSE P RECISION E STIMATION

Revised: The sparse precision matrix estimation problem in Gaussian graphical models assumes the observation of n samples from a distribution N (0 , Σ) .

Revised: Given the n samples, X 1: n := { x 1 , · · · , x n } from

Revised: N (0 , Σ) , the goal is to estimate the precision matrix Θ ∗ :

Revised: = Σ − 1 which is the inverse covariance matrix.

Revised: This precision matrix represents the conditional independency among random variables.

Revised: A commonly used objective for estimating the precision matrix is the (cid:96) 1 -penalized log determinant divergence:

Revised: [Equation]

Revised: where ˆΣ n is the sample covariance matrix.

Revised: In our method, we use the likelihood term as the empirical loss.

Revised: [Equation]

Revised: H. D ATA P REPARATION

Revised: For the sparse linear recovery problem, we follow the setting in Wang et al.

Revised: (2014b).

Revised: We create the synthetic data by sampling a set of estimation problems { (( X ( i ) , Y ( i ) ) , β ∗ ( i ) ) } .

Revised: In each problem, the design matrix X ( i ) ∈ R n × d contains n = 64 independent realizations of a random vector x ∈ R p with p = 256 , 1024 in easy or difﬁcult setting, respectively.

Revised: x follows a zero mean Gaussian distribution with covariance matrix (Σ) i,j = 0 .

Revised: 9 · 1 { i (cid:54) =

Revised: j } + 1 · 1 { i = j } .

Revised: The true parameter vector β ∗ ( i ) has a sparsity s ∗ =

Original: {+(cid:107) β ∗ ( i ) (cid:107) 
+}
Revised: {+(cid:107) β ∗ ( i ) (cid:107) 
+}

Revised: = 16 and its nonzero entries take values uniformly sampled from ( − 2 , − 12 ) ∪ ( 12 , 2) .

Revised: The support set of each β ∗ ( i ) is independently sampled from a union support set S , with | S | = 128 to allow some similarity among the problems.

Revised: The observation Y ( i ) is sampled such that Y ( i ) − X ( i ) β ∗ ( i ) is a n -dimensional Gaussian random vector with zero mean and covariance matrix I n .

Revised: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.

Revised: In sparse precision matrix estimation problem, we follow the setting in Guillot et al.

Revised: We create the synthetic data by sampling a set of estimation problems { (ˆΣ ( i ) n , Θ ( i ) ) } .

Revised: In each problem, the ground truth precision matrix is generated in the following ways: the lower diagonal part (lower triangular parting, exluding diagonal) of true precision matrix Θ has a sparsity s ∗ , that are uniformly selected from a union support with size S .

Revised: After selecting the nonzero entries, we assign them values uniformly from ( − 2 , − 12 ) ∪ ( 12 , 2) and let the upper diagonal part has the same value as lower diagonal part.

Revised: Finally, a multiple of the identity was added to the resulting matrix so that the smallest eigenvalue was equal to 1.

Revised: In this way, Θ was insured to be sparse, positive deﬁnite, and wellconditioned.

Revised: After the precision matrix is generated, we generate the observational samples for each precision matrix, by generating n independent samples of the random vector x ∼ N (0 , (Θ ( i ) ) − 1 ) .

Revised: The samples are denoted by X ( i ) ∈ R n × d .

Revised: Using the samples, we can compute the sample covariance matrix ˆ Σ ( i ) n = 1 n (cid:80) ni

Revised: =1 ( X ( i ) )

Revised: T X ( i ) .

Revised: In all the experiments, 2000 such problems are used for training, 200 such problems are used for validation, and 100 such problems are used for test.

Revised: We use ( n, S, s ∗ ) =

Revised: (100 , 375 , 75) , (200 , 600 , 150) for d = 50 , 200 , respectively.

Revised: H. B ASELINE I MPLEMENTATION

Revised: ALISTA : We follow the implementation in Liu et al. (2019a).

Original: RNN : The RNN is designed following Andrychowicz et al.
Revised: RNN : The RNN is designed following Andrychowicz et al.

Original: Initialize the β 0 as 0 .
Revised: Initialize the β 0 as 0 .

Original: At every step, we compute the gradient ∇ β L ( β t ) w.r.t.
Revised: At every step, we compute the gradient ∇ β L ( β t ) w.r.t.

Original: to objective as the input to a LSTM with hidden dimension equals to 128.
Revised: to objective as the input to a LSTM with hidden dimension equals to 128.

Original: Then we use the output of the LSTM as the increment:
Revised: Then we use the output of the LSTM as the increment:

Original: [Equation]
Revised: [Equation]

Original: We repeat this iteration 20 steps and use β 20 as our ﬁnal output.
Revised: We repeat this iteration 20 steps and use β 20 as our ﬁnal output.

Original: RNN- (cid:96) 1 : RNN- (cid:96) 1 use the same architecture as RNN.
Revised: RNN- (cid:96) 1 : RNN- (cid:96) 1 use the same architecture as RNN.

Original: The only difference is we add an extra learnable parameter λ t for soft thresholding at each step:
Revised: The only difference is we add an extra learnable parameter λ t for soft thresholding at each step:

Original: [Equation]
Revised: [Equation]

Original: ) where the softthresholding η λ is an elementwise operator maps η λ ( x ) = 0{+
+} is | x | < λ and η λ ( x ) = sign ( x )( | x
Revised: [Equation] [Equation] where the softthresholding η λ is an elementwise operator maps η λ ( x ) = {+
+} is | x | < λ and η λ ( x ) = sign ( x )( | x

Original: | − λ ) .
Revised: | − λ ) .

Revised: GLASSO : we use the implementation in the sklearn package.

Revised: GGM : We follow the implementation in Belilovsky et al.

Original: APF : We follow the implementation in Wang et al. 
Revised: APF : We follow the implementation in Wang et al. (2014b).

Revised: The speciﬁc algorithm steps are summarized in Algorithm 2.

Revised: Algorithm 2: The Approximate Path Following Method

Original: {+Input: λ tgt > 0 , (cid:15) opt > 
+}
Revised: {+Input: λ tgt > 0 , (cid:15) opt > 
+}

Original: G. T RAINING AND E VALUATION{+(Here we assume (cid:15) opt (cid:28) λ tgt / 4 ), η ∈ [0 . 9 , 
 ) 1 Initialize ˜ β 0 ← 0 , L 0 ← L min , λ 
 = (cid:107)∇L ( 0 ) (cid:107) ∞ , N ← ( λ 0 /λ tgt ) / log( η − 1 ) . 2 for t = 1 , ..., N − 1 do λ t ← η t λ (cid:15) t ← λ t / { ˜ β t , L t } ← Proximal-Gradient ( λ t , (cid:15) t , ˜ β t − 1 , L t − 1 , R ) as in Algorithm 3.+}
Revised: {+(Here we assume (cid:15) opt (cid:28) λ tgt / 4 ), η ∈ [0 . 9 , 
 ) 1 Initialize ˜ β 0 ← 0 , L 0 ← L min , λ 
 = (cid:107)∇L ( 0 ) (cid:107) ∞ , N ← ( λ 0 /λ tgt ) / log( η − 1 ) . 2 for t = 1 , ..., N − 1 do λ t ← η t λ (cid:15) t ← λ t / { ˜ β t , L t } ← Proximal-Gradient ( λ t , (cid:15) t , ˜ β t − 1 , L t − 1 , R ) as in Algorithm 3.+}

Revised: 6 end 7 λ N ← λ tgt 8 (cid:15) N ← (cid:15) opt 9 { ˜ β N , L N } ← Proximal-Gradient ( λ N , (cid:15) N , ˜ β

Revised: N − 1 , L N − 1 , R ) .

Revised: 10 return { ˜ β t } Nt =1 .

Revised: Input: λ t > 0 , (cid:15) t > 0 , β 0 t ∈ R p , L 0 t >1 Initialize k ← 0 .

Revised: 2 repeat k ← k +

Revised: L init ← max { L min , L k − 1 t / 2 } β kt , L kt ← Line-Search ( λ t , β k − 1 t , L init )

Revised: as in Algorithm 4.

Revised: 6 until ω λ t ( β kt ) ≤ (cid:15) t as deﬁned in Equation 81 ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .

Revised: Input:

Revised: [Equation]

Revised: 1 repeat

Revised: [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: (cid:1) as deﬁned in Equation if φ λ t (cid:0) β kt (cid:1) >

Revised: ψ L kt ,λ t (cid:0) β kt ; β k − 1 t (cid:1) then

Revised: L kt ← L kt .

Revised: end 6 until φ λ t ( β kt ) ≤ ψ L kt

Revised: [Equation]

Revised: λ t ( β kt

Revised: [Equation]

Revised: β k − 1 t ) as deﬁned in Equations 83 and 84 ; 7 ˜ β t ← β kt 8 L t ← L kt 9 return { ˜ β t , L t } .

Revised: [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: H. T RAINING AND E VALUATION

Original: We trained the neural networks to minimize the weightedl2 loss.
Revised: We trained the learning-based methods to minimize the weighted loss.

Original: That’s to say,  given the sequence of output ( β 1 , ..., β T ) , the loss is computed as:
Revised: That’s to say, for each sparse linear recovery problem, given the sequence of outputs ( β 1 , ..., β T ) , the loss is computed as:

Original: [Equation]
Revised: [Equation]

Original: where γ = 0 .
Revised: where γ decreasing from 0 . 9 to 0 .

Revised: 1 during the training process.

Original:  We use training set containing 2000 problems and  valid set containing 200 instances.
Revised: The procedure is similar in the sparse precision matrix estimation problem. We use optimizer Adam (Kingma & Ba, 2014). For sparse linear recovery problem, we use batch size 10, we train 500 epochs with learning rate 1e4 and select the model based on the l2 loss on valid data. For sparse precision matrix estimation problem, we use batch size 40, we train 200 epochs with learning rate 1e-3 and select the model based on Frobenius loss on valid data.

Original: We use optimizer Adam (Kingma & Ba, 2014) to train 500 epochs with learning rate 1e-4 and select the model based on the l2 loss on 200 valid problems.
Revised: For classical algorithms, we select their parameters, λ for APF and ρ for GISTA, based on their performance on the validation set. Speciﬁcally, we evaluate APF or GISTA with ρ or λ from { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } and use one with the best recovery error in test. [Table]

Original: The evaluation is performed on a server with CPU:
Revised: [Table] We report the total training time for learning based methods as well as the parameter tuning time for classical algorithms [Table] From Table 3 and Table

Revised: [Table]

Revised: we can see that training a learning-based method is cheap in our experiments, as 1)

Revised: A single forward for PLISA or GGM is very fast as stated in Table 1.

Revised: We can easily parallel the computations to handle a batch of problems during the training time, as the learning-based methods do not require line-search.

Original:  Intel(R) 
Revised: The evaluation is performed on a server with CPU: Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz, GPU: Nvidia GTX 2080TI, Memory 264G, in single thread.

Revised: H. A BLATION S TUDY

Revised: [Table]

Revised: [Figure]

Revised: We consider two variants of PLISA to perform the ablation study.

Revised: One is PLISA -single which employs a single regularization parameter across different entries, i.e., η 1 = · · · = η d and λ ∗ 1 = · · · = λ ∗ d .

Revised: The other is PLISA - (cid:96) 1 which does not learn the penalty function but uses the (cid:96) 1 norm, i.e., P w ( λ , β ) =

Revised: (cid:107) λ ◦ β (cid:107) 1 .

Revised: Fig.

Revised: 6 and Table 5 show the vanilla PLISA performs better than alternatives.

Revised: Especially, it has a much better accuracy than PLISA -single.

Revised: Therefore, this ablation study has validated the effectiveness of using entry-wise regularization parameters and learning the penalty function.

Revised: H. R EAL - WORLD E XPERIMENTS

Revised: We use the following 3 real-world datasets.

Revised: (1) Gene - a single-cell gene expression dataset that contains expression levels of 45 transcription factors measured at different time-points.

Revised: We follow Ollier & Viallon (2017) to pick the transcription factor, EGR2, as the response variable and the other 44 factors as the covariates.

Revised: In this dataset, each time-point is considered as an estimation problem.

Revised: In each estimation problem, the goal is learning the regression weights β ∗ on the 44 factors to predict the expression level of the target transcription factor, EGR2.

Revised: Therefore, in this problem, p = 44 .

Revised: This dataset contains the gene expression data for 120 single cells at 8 different time points.

Revised: For each time point, we randomly split the 120 samples into 6 sets, so that each set contains 20 samples.

Revised: By doing this, we construct 48 estimation problems each of which contains 20 samples.

Revised: 10 samples are used for recovering the parameters β ∗ and the other 10 samples are used for evaluate it by computing the least-square error.

Revised: We use 36 problems for training, 6 problems for validation, and 6 problems for testing.

Revised: (2) Parkinsons - a disease dataset that contains symptom scores of Parkinson for different patients.

Revised: Each patient is considered as an estimation problem.

Revised: In each estimation problem, the goal is learning the regression weights β ∗ on 19 bio-medical features to predict the symptom score.

Revised: This dataset contains 42 patients, so there are 42 estimation problems in total.

Revised: Each patient is examined at different time-point and each time a sample is generated.

Revised: For each patient we randomly selectsamples, so that eventually our dataset contains 42 estimation problems each of which containssamples.

Revised: 50 are used for recovering the parameter β ∗ and 50 are used for evaluation.

Revised: We useproblems for training, 7 problems for validation, and 7 problems for testing.

Revised: (3) School - an examination score dataset of students from 139 secondary schools in London.

Revised: Each

Revised: school is considered as an estimation problem.

Revised: In each estimation problem, the goal is learning the regression weights β ∗ on 28-dimensional school and student features to present the exam scores for all students.

Revised: We use the dataset from Malsar package (Zhou et al., 2011).

Revised: For each school, we randomly select 40 students as the samples.

Revised: Since some schools contain less than 40 students, we ﬁnally obtain 125 estimation problems (schools) each of wich contains 40 samples.

Original: Xeon
Revised: 20 are used for recovering We On each dataset, we train each learning-based algorithm for 200 epochs using Adam with learning rata 1e-3. The batch size is set to be 6.