Original: P ROJECTIVE P ROXIMAL G RADIENT D ESCENT FOR  N ONCONVEX N ONSMOOTH O PTIMIZATION  : F AST C ONVERGENCE W ITHOUT 
Revised: P ROJECTIVE P ROXIMAL G RADIENT D ESCENT FOR A C LASS OF N ONCONVEX N ONSMOOTH O PTIMIZATION P ROBLEMS : F AST C ONVERGENCE W ITHOUT K URDYKA -

Original: K URDYKA -L OJASIEWICZ (KL) P ROPERTY
Revised: Ł OJASIEWICZ (KŁ) P ROPERTY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Nonconvex and nonsmooth optimization problems are important and challenging for statistics and machine learning.
Revised: Nonconvex and nonsmooth optimization problems are important and challenging for statistics and machine learning.

Original: In this paper, we propose Projected Proximal Gradient Descent (PPGD) which solves a class of nonconvex and nonsmooth optimization problems, where the nonconvexity nd nonsmoothness come from a nonsmooth regularization term which is nonconvex but piecewise convex.
Revised: In this paper, we propose Projected Proximal Gradient Descent (PPGD) which solves a class of nonconvex and nonsmooth optimization problems, where the nonconvexity nd nonsmoothness come from a nonsmooth regularization term which is nonconvex but piecewise convex.

Original: In contrast with existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems based on the Kurdyka-Lojasiewicz (KL) property, we provide a new theoretical analysis showing  that PPGD achieves optimal convergence rate  on a class of nonconvex and nonsmooth problems under mild assumptions, which is  the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with Lipschitz continuous gradient.
Revised: In contrast with existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems based on the Kurdyka-Łojasiewicz (KŁ) property, we provide a new theoretical analysis showing local fast convergence of PPGD. It is proved that PPGD achieves a fast convergence rate of O (1 /k 2 ) when the iteration number k ≥ k 0 for a finite k 0 on a class of nonconvex and nonsmooth problems under mild assumptions, which is locally the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with Lipschitz continuous gradient.

Original: Experimental results demonstrate the effectiveness ofthe PPGD.
Revised: Experimental results demonstrate the effectiveness of PPGD.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Nonconvex and nonsmooth optimization problems are challenging ones which have received a lot of attention in statistics and machine learning (Bolte et al., 2014; Ochs et al., 2015).
Revised: Nonconvex and nonsmooth optimization problems are challenging ones which have received a lot of attention in statistics and machine learning (Bolte et al., 2014; Ochs et al., 2015).

Original: In this paper, we consider fast optimization algorithms for a class of nonconvex and nonsmooth problems presented as
Revised: In this paper, we consider fast optimization algorithms for a class of nonconvex and nonsmooth problems presented as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: (1) where  h ( x ) =
Revised: where g is convex, h ( x ) =

Original: d (cid:80) j
Revised: d (cid:80) j

Original: =1 h j ( x j ) is a separable regularizer, each h j is piecewise convex, and inf x ∈ R d F ( x ) > −∞ .
Revised: =1 h j ( x j ) is a separable regularizer, each h j is piecewise convex.

Original: A piecewise convex function is defined in Definition 1.1.
Revised: A piecewise convex function is defined in Definition 1.1.

Original: Without loss of generality we let h j = f for all j ∈ [ d ] , and f is a piecewise convex function. 
Revised: For simplicity of analysis we let h j = f for all j ∈ [ d ] , and f is a piecewise convex function. Here [ d ] is the set of natural numbers between 1 and n inclusively.

Original: f can be either nonconvex or convex, and all the results in this paper can be straightforwardly extended to the case when { h j } are different.
Revised: f can be either nonconvex or convex, and all the results in this paper can be straightforwardly extended to the case when { h j } are different.

Original: Definition 1.1.
Revised: Definition 1.1.

Original: A univariate function f : R → R is piecewise convex if f is lower semicontinuous and there exist intervals {R m } Mm =1 such that R =
Revised: A univariate function f : R → R is piecewise convex if f is lower semicontinuous and there exist intervals {R m } Mm =1 such that R =

Original: (cid:83) Mm =1 R m , and f restricted on R m is convex for each m ∈
Revised: (cid:83) Mm =1 R m , and f restricted on R m is convex for each m ∈

Original: [ M ] .
Revised: [ M ] .

Original: The left and right endpoints of R m are denoted by q m − 1 and q m for all m ∈
Revised: The left and right endpoints of R m are denoted by q m − 1 and q m for all m ∈

Original: [ M ] , where { q m } Mm =0 are the endpoints such that q 0 =
Revised: [ M ] , where { q m } Mm =0 are the endpoints such that q 0 =

Original: −∞ ≤ q 1 < q 2 < . . .
Revised: −∞ ≤ q 1 < q 2 < . . .

Original: <
Revised: <

Original: q M = + ∞ .
Revised: q M = + ∞ .

Original: Furthermore, f is either left continuous or right continuous at each endpoint q m for m ∈ [ M − 1] .
Revised: Furthermore, f is either left continuous or right continuous at each endpoint q m for m ∈ [ M − 1] .

Original: {R m } Mm =1 are also referred to as convex pieces throughout this paper.
Revised: {R m } M m =1 are also referred to as convex pieces throughout this paper.

Original: It is important to note that for  m ∈
Revised: It is important to note that for all m ∈

Original: [ M − 1] , when f is continuous at some endpoint q m for m∈ [ M − 1] ,then q m ∈ R m ∩ R m +1 .
Revised: [ M − 1] , when f is continuous at the endpoint q m or f is only left continuous at q m , q m ∈ R m and q m / ∈ R m +1 .

Original: If f is onlyleft continuous at q m , q m ∈ R m and q m / ∈ R m +1 . If f is only right continuous at q m , q m / ∈ R m and q m ∈ R m +1 . 
Revised: If f is only right continuous at q m , q m / ∈ R m and q m ∈ R m +1 . This ensures that any point in R lies in only one convex piece.

Original: When M = 1 , f becomes a convex function on R , and problem (1) is a convex problem.
Revised: When M = 1 , f becomes a convex function on R , and problem (1) is a convex problem.

Original: We consider M > 1 throughout this paper, and our proposed algorithm trivially extends to the case when M = 1 .
Revised: We consider M > 1 throughout this paper, and our proposed algorithm trivially extends to the case when M = 1 .

Original: We allow a special case that an interval R m = { q m } for m ∈ [ M − 1] is a singlepoint set, in this case q m − 1 = q m .
Revised: We allow a special case that an interval R m = { q m } for m ∈ [ M − 1] is a singlepoint set, in this case q m − 1 = q m .

Original: When there are no single-point intervals in {R m } Mm =1 , the minimum interval length is defined by R 0 = min m ∈
Revised: When there are no single-point intervals in {R m } Mm =1 , the minimum interval length is defined by R 0 = min m ∈

Original: [ M − 1] |R m | .
Revised: [ M ] |R m | .

Original: Otherwise,the minimum interval length is R 0 = min m ∈
Revised: Otherwise,the minimum interval length is R 0 = min m ∈

Original: [ M − 1]: |R m |̸ =

Revised: [ M ]: |R m |̸ =

Original: |R m | , where |·| denotes the length of an interval.
Revised: |R m | , where |·| denotes the length of an interval.

Original: Below are three examples of piecewise convex functions with the corresponding regularizers which have been widely used in constrained optimization and sparse learning problems.
Revised: Below are three examples of piecewise convex functions with the corresponding regularizers which have been widely used in constrained optimization and sparse learning problems.

Original: [Figure]
Revised: [Figure]

Original: Example 1.1.
Revised: Example 1.1.

Original: The indicator penalty function f ( x ) =
Revised: The indicator penalty function f ( x ) =

Original: λ 1I { x<τ } is piecewise convex with R 1 = ( −∞ , τ ) , R 2 =
Revised: λ 1I { x<τ } is piecewise convex with R 1 = ( −∞ , τ ) , R 2 =

Original: [ τ, ∞ ) .
Revised: [ τ, ∞ ) .

Original: The capped- ℓ 1 penalty function f ( x ) =
Revised: The capped- ℓ 1 penalty function f ( x ) =

Original: f
Revised: f

Original: ( x ; λ, b ) =
Revised: ( x ; λ, b ) =

Original: λ min {| x | , b } is piecewise convex with R 1 = ( −∞ , − b ] , R 2 =
Revised: λ min {| x | , b } is piecewise convex with R 1 = ( −∞ , − b ] , R 2 =

Original: [ − b, b ] , R 3 =
Revised: [ − b, b ] , R 3 =

Original: [ b, ∞ ) .
Revised: [ b, ∞ ) .

Original: The leaky cappedℓ 1 penalty function (Wangni & Lin, 2017) f
Revised: The leaky cappedℓ 1 penalty function (Wangni & Lin, 2017) f

Original: = λ min {| x | , b } + β 1I {| x |≥ b } | x − b | with R 1 = ( −∞ , − b ] , R 2 =
Revised: = λ min {| x | , b } + β 1I {| x |≥ b } | x − b | with R 1 = ( −∞ , − b ] , R 2 =

Original: [ − b, b ] , R 3 =
Revised: [ − b, b ] , R 3 =

Original: [ b, ∞ ) .
Revised: [ b, ∞ ) .

Original: The three functions are illustrated in Figure 1.
Revised: The three functions are illustrated in Figure 1.

Original: While not illustrated, f ( x ) = 1I { x ̸ =0 } for the ℓ 0 -norm with h ( x ) =
Revised: While not illustrated, f ( x ) = 1I { x ̸ =0 } for the ℓ 0 -norm with h ( x ) =

Original: ∥ x ∥ 0 is also piecewise convex.
Revised: ∥ x ∥ 0 is also piecewise convex.

Original: A piecewise convex function f on each of its convex piece R m can be viewed as an restriction of a globally convex function f i to that convex piece. That is, there exists a convex function f
Revised: We propose a novel method termed Projective Proximal Gradient Descent (PPGD) to solve problem (1), and PPGD enjoys fast convergence rate by a novel projection operator and a negative curvature exploitation procedure. Our main results are summarized in the subsection below.

Original: i : R → R such that f i ( x ) = f ( x ) for all i ∈ [ m ] and x ∈ R m .
Revised: 1. M AIN A SSUMPTION

Original: The main assumption of this paper is that g and h satisfy the following conditions.
Revised: The main assumption of this paper is that g and h satisfy the following conditions.

Original: Assumption 1 (Main Assumption) .
Revised: Assumption 1 (Main Assumption) .

Original: (a) g is convex 
Revised: (a) g is convex with

Original: with L -smooth gradient, that is, ∥∇ g ( x ) − ∇ g ( y ) ∥ 2 ≤L g ∥ x − y ∥ 2 .
Revised: L -smooth gradient, that is ∥∇ g ( x ) − ∇ g ( y ) ∥ 2 ≤

Original:  ghas bounded gradient, and ∥∇ g ( x ) ∥∞ ≤ G for a positive constant G for all x ∈ R d .
Revised: L g ∥ x −

Original: (b) f : R → R is a piecewise convex function.
Revised: y ∥ 2 . F is coercive that

Original: Furthermore, f satisfies the following two conditions.
Revised: is, F ( x → ∞ when ∥ x ∥ 2 → ∞ , and inf x ∈ R d F ( x ) > −∞ .

Original: (1) There exists a small positive constant s 0 < R 0 such that f is differential on ( q m − s 0 , q m ) and ( q m , q m + s 0 ) for all m ∈
Revised: (b) f : R → R is a piecewise convex function and lower semicontinuous. Furthermore, there exists a small positive constant s 0 < R 0 such that f is differentiable on [ q m − s 0 , q m ) and ( q m , q m + s 0 ] for all m ∈

Original: [ M − 1] .
Revised: [ M − 1] .

Original: (c)
Revised: (c)

Original: The proximal mapping prox sf i has closed form solution for all m ∈ [ − M, M ] , where prox sf i ( x ) :=
Revised: The proximal mapping prox sf i has a closed-form solution for all m ∈ [ − M, M ] , where prox sf i ( x ) :

Original:  argmin v ∈ R 12 s ( v − x ) 2 + f i ( v ) .
Revised: = argmin v ∈ R 12 s ( v − x ) 2 + f i ( v ) .

Original: (d) f has “negative curvature” at each endpoint q m where f is continuousat for all m ∈
Revised: (d) f has “negative curvature” at each endpoint q m where f is continuous for all m ∈

Original: [ M − 1] , that is, lim x → q − m f ′
Revised: [ M − 1] , that is, lim x → q − m f ′

Original: ( x ) >
Revised: ( x ) >

Original: lim x → q + m f ′ ( x ) .
Revised: lim x → q + m f ′ ( x ) .

Original: We define
Revised: We define

Original: [Equation]
Revised: [Equation]

Original: In addition,

Original: f

Original:  has bounded Fr´echet subdifferential , that is, sup x 
Revised: In addition, f has bounded Fr´echet subdifferential, that is, sup x ∈

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: Mm =1 R om sup

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∈ ˜ ∂f

Original: [Equation]
Revised: [Equation]

Original:  ∥ 2 ≤ F  for some absolute constant F  > 0 , where R o denotes the interior of an interval.
Revised: x ) ∥ v ∥ 2 ≤ F 0 for some absolute constant F 0 > 0 , where R o denotes the interior of an interval.

Original: Fr´echet subdifferential is formally defined in Section 3.
Revised: Fr ´ echet subdifferential is formally defined in Section 3.

Original: It is noted that on each R om , the convex differential of f coincides with its Fr´echet subdifferential.
Revised: It is noted that on each R om , the convex differential of f coincides with its Fr´echet subdifferential.

Original: We define the minimum jump value of f at noncontinuous endpoints by
Revised: We define the minimum jump value of f at noncontinuous endpoints by

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Assumption 1 is mild and explained as follows.
Revised: Assumption 1 is mild and explained as follows.

Original: When g is a G -Lipschitz continuous function, its gradient is bounded by G

Original: [Table] and Assumption 1(a) holds. Smooth gradient  is a commonly required in the standard analysis of proximal methods for non-convex problems
Revised: Smooth gradient in Assumption 1(a) is a commonly required in the standard analysis of proximal methods for non-convex problems, such as (Bolte et al., 2014; Ghadimi & Lan, 2016a).

Original: [Table] such as (Bolte et al., 2014;
Revised: The objective F is also assumed to be coercive in (Li & Lin, 2015;

Original: Ghadimi & Lan, 2016a).
Revised: Li et al., 2017), which leads to bounded iterates.

Original: In addition, Assumption 1(b)-(d) are mild, and they hold for all the three piecewise convex functions in Example 1.1 as well as the ℓ 0 -norm.
Revised: In addition, Assumption 1(b)-(d) are mild, and they hold for all the three piecewise convex functions in Example 1.1 as well as the ℓ 0 -norm.

Original: It is noted that (1) covers a broad range of optimization problems in machine learning and statistics.
Revised: It is noted that (1) covers a broad range of optimization problems in machine learning and statistics.

Original: The nonnegative programming problem min x ∈ R d : x i ≥ 0 ,i ∈
Revised: The nonnegative programming problem min x ∈ R d : x i ≥ 0 ,i ∈

Original: [ d ] g ( x ) can be reduced to the regularized problem g ( x ) + h ( x ) with f being the indicator penalty f
Revised: [ d ] g ( x ) can be reduced to the regularized problem g ( x ) + h ( x ) with f being the indicator penalty f

Original: ( x ) = λ 1I { x<τ } for a properly large λ .
Revised: ( x ) = λ 1I { x<τ } for a properly large λ .

Original: When g is the squared loss, that is, g ( x ) =
Revised: When g is the squared loss, that is, g ( x ) =

Original: ∥
Revised: ∥

Original: y − Dx ∥ 22 where y ∈ R n and D ∈ R n × d is the design matrix, (1) is the well-known regularized sparse linear regression problems with convex or nonconvex regularizers such as the capped- ℓ 1 or ℓ 0 -norm penalty.
Revised: y − Dx ∥ 22 where y ∈ R n and D ∈ R n × d is the design matrix, (1) is the well-known regularized sparse linear regression problems with convex or nonconvex regularizers such as the capped- ℓ 1 or ℓ 0 -norm penalty.

Original: We propose a novel method termed Projective Proximal Gradient Descent (PPGD) to solve problem (1), and PPGD enjoys fast convergence rate by a novel projection operator and a negative curvature exploitation procedure.

Original: Our main results are summarized in the subsection below.

Original: 1. M AIN R ESULTS AND C ONTRIBUTIONS
Revised: 1. M AIN R ESULTS AND C ONTRIBUTIONS

Original: We extend the existing Accelerated Proximal Gradient descent method (APG) (Beck & Teboulle, 2009b; Nesterov, 2013) on problem (1), showing improved convergence rates.
Revised: We extend the existing Accelerated Proximal Gradient descent method (APG) (Beck & Teboulle, 2009b; Nesterov, 2013) on problem (1), showing improved convergence rates.

Original: More concretely, 1.
Revised: More concretely, 1.

Original: We propose Projected Proximal Gradient Descent (PPGD).
Revised: We propose Projected Proximal Gradient Descent (PPGD).

Original: Using the novel and carefully designed projection operator and the Negative-Curvature-Exploitation algorithm (Algorithm 3),
Revised: Using the novel and carefully designed projection operator and the Negative-Curvature-Exploitation algorithm (Algorithm 2),

Original: PPGD achieves a fast convergence rate for the nonconvex and nonsmooth optimization problem (1) which  matches the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with Lipschitz continuous gradient.
Revised: PPGD achieves a fast convergence rate for the nonconvex and nonsmooth optimization problem (1) which locally matches the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with Lipschitz continuous gradient.

Original: In particular, it is proved that under two mild assumptions, Assumption 1 and Assumption 2 to be introduced in Section 3, there exists a finite k 0 ≥ 1 such that for all k > k0 ,
Revised: In particular, it is proved that under two mild assumptions, Assumption 1 and Assumption 2 to be introduced in Section 3, there exists a finite k 0 ≥ 1 such that for all k > k

Original: [Equation]
Revised: [Equation]

Original: Details are deferred to Theorem 3.5 in Section 3.
Revised: Details are deferred to Theorem 3.4 in Section 3.

Original: It should be emphasized that this is the same convergence rate as that of regular APG on convex problems (Beck & Teboulle
Revised: It should be emphasized that this is the same convergence rate as that of regular APG on convex problems (Beck & Teboulle, 2009a;b).

Original: , 2009a b). Our analysis provides insights into accelerated PGD methods for challenging nonconvex and nonsmooth problems.
Revised: Our analysis provides insights into accelerated PGD methods for challenging nonconvex and nonsmooth problems.

Original: In contrast to most existing accelerated PGD methods (Li & Lin, 2015; Li et al., 2017) which employ the Kurdyka-Lojasiewicz (KL) property to analyze the convergence rates, PPGD provides an new perspective of convergence analysis without the KL property while  enjoying the Nesterov’s optimal convergence rate.
Revised: In contrast to most existing accelerated PGD methods (Li & Lin, 2015; Li et al., 2017) which employ the Kurdyka-Łojasiewicz (KŁ) property to analyze the convergence rates, PPGD provides a new perspective of convergence analysis without the KŁ property while locally enjoying the Nesterov’s optimal convergence rate.

Original: Our analysis reveals that the objective function makes progress, that is, its value is decreased by a positive amount, when the variable sequence generated by PPGD transits from one convex piece to another.
Revised: Our analysis reveals that the objective function makes progress, that is, its value is decreased by a positive amount, when the variable sequence generated by PPGD transits from one convex piece to another.

Original: Such observation opens the door for future research in analyzing convergence rate of accelerated APG methods without the KL property.
Revised: Such observation opens the door for future research in analyzing convergence rate of accelerated APG methods without the KL property

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Assuming that both g and h are The accelerated PGD algorithms in (Li & Lin, 2015; Li et al., 2017) have linear convergence rate with θ ∈
Revised: Assuming that both g and h satisfy the KŁ Property defined in Definition A.1 in Section A.1 of the supplementary, the accelerated PGD algorithms in (Li & Lin, 2015; Li et al., 2017) have linear convergence rate with θ ∈

Original: [1 / 2 , 1) , and sublinear rate O ( k − 1 1 − 2 θ ) with θ ∈ (0 , 1 / 2) , where θ is theKurdyka-Lojasiewicz (KL) exponent and both rates are for objective values. The KL property of proper and lower semicontinuous functions is defined below. Definition 1.2 (Kurdyka-Lojasiewicz (KL) Property) . Let u : R d → ( −∞ , ∞ ] be proper and lower semicontinuous.
Revised: [1 / 2 , 1) , and sublinear rate O ( k − 1 1 − 2 θ ) with θ ∈ (0 , 1 / 2) , where θ is the

Original: u is said to satisfy the Kurdyka-Lojasiewicz (KL) property at u ∈ dom ∂u := { u ′ ∈ R d : ∂u ( u ′ ) ̸ = ∅} , if there exist η ∈ (0 , ∞ ] , a neighborhood U of u and a function ϕ ∈ Φ η , such that for all ¯ u ∈ U ∪ [ u ( u ) < u (¯ u ) < u ( u ) + η ] , the following inequality holds: ϕ
Revised: Kurdyka-Lojasiewicz (KŁ) exponent and both rates are for objective values.

Original: [Equation] Here η ∈ (0 , ∞ ] , Φ η is defined to be the class of all concave and continuous functionsϕ : [0 , η ) → R + which satisfy the following conditions: ϕ (0) = 0 ; 2. ϕ is continuously differentiable on (0 , η ) and continuous at 0 ;
Revised: The KŁ property of proper and lower semicontinuous functions is defined below.

Original: ϕ ′ ( r ) > 0 for all r ∈ (0 , η )

Original: If u satisfies the KL property at each point of dom ∂u , then u is called a KL function.

Original: To the best of our knowledge, most existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems, where the nonconvexity and nonsmoothness are due to the regularizer h , are based on the KL property.
Revised: To the best of our knowledge, most existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems, where the nonconvexity and nonsmoothness are due to the regularizer h , are based on the KŁ property.

Original: While (Ghadimi & Lan, 2016b) provides analysis of an accelerated PGD method for nonconvex and nonsmooth problems, the nonconvexity comes from the smooth part of the objective function ( g in our objective function), so it cannot handle the problems discussed in this paper.
Revised: While (Ghadimi & Lan, 2016b) provides analysis of an accelerated PGD method for nonconvex and nonsmooth problems, the nonconvexity comes from the smooth part of the objective function ( g in our objective function), so it cannot handle the problems discussed in this paper.

Original: Furtheremore, the fast PGD method by (Yang & Yu, 2020) is restricted to ℓ 0 -regularized problems.
Revised: Furtheremore, the fast PGD method by (Yang & Yu, 2020) is restricted to ℓ 0 -regularized problems.

Original: As a result, it remains an interesting and important open problem regarding the convergence rate of accelerated PGD methods for problem (1).
Revised: As a result, it remains an interesting and important open problem regarding the convergence rate of accelerated PGD methods for problem (1).

Original: Another line of related works focuses on accelerated gradient methods for smooth objective functions.
Revised: Another line of related works focuses on accelerated gradient methods for smooth objective functions.

Original: For example, (Jin et al., 2018) proposes perturbed accelerated gradient method which achieves faster convergence by decreasing a quantity named Hamiltonian, which is the sum of the current objective value and the scaled squared distance between consecutive iterates, by exploiting the negative curvature.
Revised: For example, (Jin et al., 2018) proposes perturbed accelerated gradient method which achieves faster convergence by decreasing a quantity named Hamiltonian, which is the sum of the current objective value and the scaled squared distance between consecutive iterates, by exploiting the negative curvature.

Original: The very recent work (Li & Lin, 2022) further removes the polylogarithmic factor in the time complexity of (Jin et al., 2018) by a restarting strategy.
Revised: The very recent work (Li & Lin, 2022) further removes the polylogarithmic factor in the time complexity of (Jin et al., 2018) by a restarting strategy.

Original: Because our objective is nonsmooth, the Hamiltonian cannot be decreased by exploiting the negative curvature of the objective in the same way as (Jin et al., 2018).
Revised: Because our objective is nonsmooth, the Hamiltonian cannot be decreased by exploiting the negative curvature of the objective in the same way as (Jin et al., 2018).

Original: However, we still manage to design a “Negative-Curvature-Exploitation” algorithm which decreases the objective value by an absolute positive amount when the iterates generated by PPGD cross endpoints.
Revised: However, we still manage to design a “Negative-Curvature-Exploitation” algorithm which decreases the objective value by an absolute positive amount when the iterates of PPGD cross endpoints.

Original: Our results are among the very few results in the optimization literature about fast optimization methods for nonconvex and nonsmooth problems which are not limited to ℓ 0 -regularized problems.
Revised: Our results are among the very few results in the optimization literature about fast optimization methods for nonconvex and nonsmooth problems which are not limited to ℓ 0 -regularized problems.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: Throughout this paper, we use bold letters for matrices and vectors, regular lower letters for scalars.
Revised: Throughout this paper, we use bold letters for matrices and vectors, regular lower letters for scalars.

Original: The bold letter with subscript indicates the corresponding element of a matrix or vector.
Revised: The bold letter with subscript indicates the corresponding element of a matrix or vector.

Original: ∥ · ∥ p denotes the ℓ p -norm of a vector, or the p -norm of a matrix.
Revised: ∥·∥ p denotes the ℓ p -norm of a vector, or the p -norm of a matrix.

Original: | A | denotes the cardinality of a set A .
Revised: | A | denotes the cardinality of a set A .

Original: [ n ] is the set of natural numbers between 1 and n inclusively.

Original: WhenR ⊆
Revised: When

Original: R  is an interval, R is the closure of R , R + denotes the set of all the points in R to the right of R while not in R , and R − is defined similarly denoting the set of all the points to the left of R .
Revised: R ⊆ R is an interval, R is the closure of R , R + denotes the set of all the points in R to the right of R while not in R , and R − is defined similarly denoting the set of all the points to the left of R .

Original: The domain of any function u is denoted by dom u .
Revised: The domain of any function u is denoted by dom u .

Original: lim x → a + and lim x
Revised: lim x → a + and lim x

Original: → a − denote left limit and right limit at a point a ∈ R .
Revised: → a − denote left limit and right limit at a point a ∈ R .

Original: N is the set of all natural numbers.
Revised: N is the set of all natural numbers.

Original:  ALGORITHMS
Revised:  ALGORITHMS

Original: Before presenting the proposed algorithm, we first present the definition of proximal mapping, and then describe how to build surrogate functions for each convex piece.
Revised: Before presenting the proposed algorithm, we first define proximal mapping, and then describe how to build surrogate functions for each convex piece.

Original: These surrogate functions are used in the proposed PPGD algorithm.
Revised: These surrogate functions have much simpler structure than the original f , and PPGD uses surrogate functions to perform proximal mapping.

Original: Definition 2.1 (Proximal Mapping) .
Revised: Definition 2.1 (Proximal Mapping) .

Original: The proximal mapping associated with function u is defined as prox u ( x ) :
Revised: The proximal mapping associated with function u is defined as prox u ( x ) :

Original: = argmin v ∈ R d u ( v ) + 12 ∥ v − x ∥ 22 for x ∈ R d , and prox u ( · ) is called the proximal mapping associated with u .
Revised: = argmin v ∈ R d u ( v ) + 12 ∥ v − x ∥ 22 for x ∈ R d , and prox u ( · ) is called the proximal mapping associated with u .

Original: 2. C ONSTRUCTION OF THE S URROGATE F UNCTIONS { f m } Mm = 
Revised: 2. C ONSTRUCTION OF THE S URROGATE F UNCTIONS { f m } Mm =

Revised: A piecewise convex function f on each of its convex piece R m can be viewed as an restriction of a globally convex function

Revised: f i to that convex piece.

Original:  Given that f is convex on each convex piece R m , we describe how to construct a surrogate function f m : R → R such that f m ( x ) =
Revised: That is, there exists a convex function f i : R → R such that f i ( x ) = f ( x ) for all i ∈ [ m ] and x ∈ R m . Given that f is convex on each convex piece R m , we describe how to construct a surrogate function f m : R → R such that f m ( x ) =

Original: f
Revised: f

Original: ( x ) for all x ∈ R m .
Revised: ( x ) for all x ∈ R m .

Original: If the right endpoint q = q m is not + ∞ and f is continuous at q , then f m extends f | R m such that f m on ( q, + ∞ ) is linear.
Revised: If the right endpoint q = q m is not + ∞ and f is continuous at q , then f m extends f | R m such that f m on ( q, + ∞ ) is linear.

Original: Similar extension applies to the case when q = q m − 1 is the right endpoint of R m .
Revised: Similar extension applies to the case when q = q m − 1 is the left endpoint of R m .

Original: Formally, we letf m ( x ) =  [Equation] f
Revised: Formally, we let

Original: [Equation]

Original: ) f is continuous at q m , lim

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ R + m if q m ̸ = + ∞ , where v − =
Revised: for all x ∈ R + m if q m ̸ = + ∞ , where v − =

Original: lim x → q − m f ′ ( x ) .
Revised: lim x → q − m f ′ ( x ) .

Original: Note that when lim y
Revised: Note that when lim y

Original: → q + m f ( y ) > f ( q m ) , f is not continuous at q m .
Revised: → q + m f ( y ) > f ( q m ) , f is not continuous at q m .

Original: According to Assumption 1, f is either left continuous or right continuous at q m .
Revised: According to Assumption 1(b), f is either left continuous or right continuous at q m .

Original: By the lower semicontinuity of f , it follows that f must be left continuous at q m ∈ R m .
Revised: By the lower semicontinuity of f , it follows that f must be left continuous at q m ∈ R m .

Original: The surrogate function f m is defined similarly on R − m byf m ( x ) =
Revised: The surrogate function f m is defined similarly on R − m by

Original:   f ( q m − 1 ) + v + ( x − q m − 1 ) f is continuous at q m − 1 , lim y

Original: [Equation]
Revised: [Equation]

Original: q + m − 1 f ( y ) + v + ( x − q m − 1 ) f

Original: ( q m − 1 ) <

Original: lim y

Original: → q + m − 1 f ( y ) lim y → q − m − 1 f ( y ) f ( q m − 1 ) <
Revised: for all x ∈ R − m if q m − 1 ̸ =

Original: lim y

Original: → q − m − 1 f ( y ) , (6) for all x ∈ R − m if q m − 1 ̸ =

Original: −∞ , where v + = lim x
Revised: −∞ , where v + = lim x

Original: → q + m − 1 f ′ ( x ) .
Revised: → q + m − 1 f ′ ( x ) .

Original: Figure 2 illustrates the surrogate function f m ( x ) with x ∈ R + m for the three different cases in (5).
Revised: Figure 2 illustrates the surrogate function f m ( x ) with x ∈ R + m for the three different cases in (5).

Original: xm q xm q xm q m ( ) ( ) =
Revised: xm q xm q xm q m ( ) ( ) =

Original: li m y m q f x f y + → ( ) f x ( ) m f
Revised: li m y m q f x f y + → ( ) f x ( ) m f

Original: x is continuous at m f q ( ) ( ) ( ) m m m f x f q v x q − =
Revised: x is continuous at m f q ( ) ( ) ( ) m m m f x f q v x q − =

Original: + − ( ) ( ) m f x f x = lim
Revised: + − ( ) ( ) m f x f x = lim

Original: ( ) ( ) ) ( m m m y q f f y x v x q − − → + − =
Revised: ( ) ( ) ) ( m m m y q f f y x v x q − − → + − =

Original: ( ) lim ( ) m m
Revised: ( ) lim ( ) m m

Original: y q f q f y − → < ( ) lim
Revised: y q f q f y − → < ( ) lim

Original: ( ) m m y q f q f y + → <
Revised: ( ) m m y q f q f y + → <

Original: Example 2.1 explains how to build surrogate functions for two popular piecewise convex functions in the optimization literature.
Revised: Example 2.1 explains how to build surrogate functions for two popular piecewise convex functions in the optimization literature.

Original: It can be observed from the definition of surrogate functions and
Revised: It can be observed from the definition of surrogate functions and

Original: Figure 2 that surrogate functions always have a simpler geometric structure than the original f , so it is usually the case that proximal mapping associated with the surrogate functions have closedform expressions, given that proximal mapping associated with f has closed-form expressions.
Revised: Figure 2 that surrogate functions always have a simpler geometric structure than the original f , so it is usually the case that proximal mapping associated with the surrogate functions have closedform expressions, given that proximal mapping associated with f has closed-form expressions.

Original: The proximal mappings associated with the surrogate functions are also given in Example 2.1.[Equation] −
Revised: The proximal mappings associated with the surrogate functions are also given in Example 2.1.

Original: − s ∇

Original: Example 2.1 (Examples of Surrogate Functions) .
Revised: Example 2.1 (Examples of Surrogate Functions) .

Original: The capped- ℓ 1 penalty function f ( x ) =
Revised: The capped- ℓ 1 penalty function f ( x ) =

Original: λ min {| x | , b } has three surrogate functions as f 1 = f 3 ≡ λb , and
Revised: λ min {| x | , b } has three surrogate functions as f 1 = f 3 ≡ λb , and

Original: f 2 = λ | x | .
Revised: f 2 = λ | x | .

Original: The proximal mappings for all { f i } 3 i =1 has closed-form expressions, with prox sf 1 ( x ) =
Revised: The proximal mappings for all { f i } 3 i =1 has closed-form expressions, with prox sf 1 ( x ) =

Original: prox sf 3 ( x )
Revised: prox sf 3 ( x )

Original: = x , and prox sf 2 ( x ) =
Revised: = x , and prox sf 2 ( x ) =

Original: (1 − λs/ | x | ) + x being the soft thresholding operator, where ( a ) + = a when a ≥and ( a ) + = 0 when a < 0 .
Revised: (1 − λs/ | x | ) + x being the soft thresholding operator, where ( a ) + = a when a ≥and ( a ) + = 0 when a < 0 .

Original: The indicator penalty function f ( x ) =
Revised: The indicator penalty function f ( x ) =

Original: λ 1I { x<τ } has surrogate functions f 1 ≡ λ , f 2 = f .
Revised: λ 1I { x<τ } has surrogate functions f 1 ≡ λ , f 2 = f .

Original: The proximal mappings are prox sf 1 ( x ) = x , prox sf 2 ( x ) =
Revised: The proximal mappings are prox sf 1 ( x ) = x , prox sf 2 ( x ) =

Original: τ when τ − √ 2 λs < x < τ , and prox sf 2 ( x ) =
Revised: τ when τ − √ 2 λs < x < τ , and prox sf 2 ( x ) =

Original: x otherwise except for the point x = τ − √ 2 λs .
Revised: x otherwise except for the point x = τ − √ 2 λs .

Original: While prox sf 2 ( x ) has two values at x = τ − √ 2 λs , it will not be evaluated at x = τ − √ 2 λs in our PPGD algorithm to be introduced. 
Revised: While prox sf 2 ( x ) has two values at x = τ − √ 2 λs , it will not be evaluated at x = τ − √ 2 λs in our PPGD algorithm to be introduced. 2. P ROJECTIVE P ROXIMAL G RADIENT D ESCENT

Original: 1: z (1) =
Revised: 1: z (1) =

Original: x (1) =
Revised: x (1) =

Original: x (0) , t 0 = 0 , t 1 = 1  .
Revised: x (0) , t 0 = 0 , t 1 = 1 , satisfies (16), constants w 0 ∈ (0 , 1] .

Original: 2: for k = 1 , . . 
Revised: 2: for k = 1 , . . .

Original: , dou ( k ) =
Revised: , do

Original: x ( k ) + t k − 1 t k ( z ( k ) − x ( k ) ) + t k − 1 − 1 t k ( x ( k ) − x ( k − 1) ) (7) w ( k ) = P x ( k ) ( u ( k ) )
Revised: [Equation]

Original: (8) for i = 1 , . . .
Revised: 8) for i = 1 , . . .

Original: , d, do z ( k+1)
Revised: , d, do [Equation] k

Original: i = prox sf P ( x ( k ) i ) (cid:16)(cid:104) w ( k ) − s ∇ g ( w ( k ) (cid:105) i (cid:17) (9)
Revised: [Equation]

Original:  t k +1 =
Revised: 9) t k +1 =

Original: (cid:112) 1 + 4 t 2 k{+
+}
Revised: (cid:112) {+
+}

Original: + .
Revised: + 4 t

Original: if F P ( x ( k ) ) ( z ( k +1) ) ≤ F ( x ( k ) ) then x ( k +1) = Negative-Curvature-Exploitation (cid:16) x ( k ) i , z ( k +1)
Revised: [Equation]

Original: i (cid:17) (10) else x ( k +1) = x ( k ) (cid:17) 1: if P ( z ( k +1) ) = P ( x ( k ) )
Revised: +

Original: then 2: x ( k +1) = z ( k +1) 3: return x ( k +1) 4: Flag = false 5: for i = 1 , . . . , d do 6: if P ( z ( k +1) i ) ̸
Revised: [Equation]

Original: = P ( x ( k )i )
Revised: if F P ( x ( k ) )

Original: = m i 7: if f is continuous at q ( w ( k  ) i ) and d i, 1 ≥ w 0 d i, 0 then 8: Flag = true 9: if q ( w ( k )i )is a right endpoint of R m i and f
Revised: ( z ( k +1) ) ≤ F ( x ( k ) )

Original: q w ( k ) i )) <
Revised: then x ( k +1) =

Original: lim y → (  w  ( ) i ) − f
Revised: Negative-Curvature-Exploitation (cid:16) x ( k ) , z ( k +1) , w 0 (cid:17) (10) else x ( k +

Original: or q ( ) i ) 10: is a left endpoint of R m i f ( q ( w ( k ) i )) < lim y
Revised: [Equation]

Original: → q ( w ( k ) i ) + f ( y ) then 11: Flag =true 12: if R m i = (cid:110) q ( w ( k ) i ) (cid:111)
Revised: =

Original: then 13: x ( k +1) i = q ( w ( k ) i ) . 14: if Flag = false then 15: x ( k +1) =
Revised: [Equation]

Original: x ( k )
Revised: ( k )

Original: 2. P ROJECTIVE P ROXIMAL G RADIENT D ESCENT

Original: We first define a basic operator P which returns the index of the convex piece that the input lies in, that is, P: R → N , P ( x ): = m such that x ∈ R m .
Revised: We first define a basic operator P which returns the index of the convex piece that the input lies in, that is, P ( x ) = m for x ∈ R m .

Original: While P ( x )can take any of the two values when x is an endpoint at which f is continuous, this does not affect our analysis and theoretical results because P ( x ) ̸ = P ( y ) indicates x, y are on different convex pieces .
Revised: It is noted that P ( x ) is a single-valued function, so P ( x ) ̸ = P ( y ) indicates x, y are on different convex pieces, that is, there is no convex piece which contain both x and y .

Original: Given a vector x ∈ R d , we define P ( x ) ∈ R d such that [ P ( x )] i = P ( x i ) for all
Revised: Given a vector x ∈ R d , we define P ( x ) ∈ R d such that [ P ( x )] i = P ( x i ) for all

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: We then introduce a novel projection operator P x  ( · ) for any x ∈ R d .
Revised: We then introduce a novel projection operator P x ,R 0 ( · ) for any x ∈ R d .

Original: That is, P x  : R d → R d , and P x  ( u ) for all u ∈ R d is defined as
Revised: That is, P x ,R 0 : R d → R d , and P x ,R 0 ( u ) for all u ∈ R d is defined as

Original: [Equation]
Revised: [Equation]

Original: (11) for all i ∈ [ d ] .
Revised: for all i ∈ [ d ] , where B ( x i , R 0 ) :

Original:  Since R P ( x i )  is a closed set, (11) is  well-defined.
Revised: = [ x i − R 0 , x i + R 0 ] . Since R P ( x i ) ∩ B ( x i , R 0 ) is a closed set, (11) is the projection of u i onto the closed convex set R P ( x i ) ∩ B ( x i , R 0 ) which is well-defined.

Original: P x  ( u ) can be computed very efficiently by setting the endpoint of R P ( x i )  closer to u i to [ P x  ( u )] 
Revised: P x ,R 0 ( u ) can be computed very efficiently by setting the endpoint of R P ( x i ) ∩ B ( x i , R 0 ) closer to u i to [ P x ,R 0 ( u )] i .

Original: i .
Revised: Algorithm 1: if P ( z ( k +1) ) = P ( x ( k ) ) then 2: x ( k +1)

Original: In fact, [ P x ( u )]
Revised: = z ( k +1) 3: return x ( k +1) 4: Flag = false 5: z ′ = z ( k +1) 6: for i = 1 , . . . , d do 7: if P ( z ( k +1) i ) ̸

Original: i is the projection of u i onto the closed convex set R P ( x  i ) .
Revised: = P ( x ( k ) i ) = m i then 8: if f is continuous at q ( w ( k ) i ) and d i, 1 ≥ w 0 d i, 0 then 9:

Original: The Projected Proximal Gradient Descent (PPGD) algorithm is described in Algorithm 2.
Revised: Flag = true 10: if f is not continuous at q ( w ( k ) i ) then 11: Flag = true 12: if R P ( z ( k +1) i ) = (cid:110) q ( w ( k ) i ) (cid:111) then 13: z ′ i = q ( w ( k ) i ) . 14: if Flag = false then 15: x ( k +1) =

Revised: x ( k ) 16:

Revised: else 17: x ( k +1)

Revised: =

Revised: z ′ 18: return x ( k +1)

Revised: The Projected Proximal Gradient Descent (PPGD) algorithm is described in Algorithm 1.

Original: The following functions and quantities are defined which are useful for PPGD.
Revised: The following functions and quantities are defined which are useful for PPGD.

Original: Let q ( w ( k ) i ) be the endpoint closest to w ( k ) i which lies in [ w ( k ) i , z ( k +1) i ] or [ z ( k +1)
Revised: Let q ( w ( k ) i ) be the endpoint closest to w ( k ) i which lies in [ w ( k ) i , z ( k +1) i ] or [ z ( k +1)

Original: i , w ( k ) i ] when P ( z ( k +1) i ) [Equation]
Revised: i , w ( k ) i ] when P ( z ( k +1) i ) ̸

Original: = P ( x ( k ) i ) at iteration k of Algorithm 2.
Revised: = P ( x ( k ) i ) at iteration k of Algorithm 1.

Original: We define F P ( x 
Revised: We define F P ( x ) ( v ) :

Original: [Equation]
Revised: = g ( v ) + (cid:80) di =1 f P ( x i ) ( v i ) , d i, 0 := (cid:12)(cid:12)(cid:12) z ( k +1) i − w ( k ) i (cid:12)(cid:12)(cid:12) , d i, 1 :=

Revised: (cid:12)(cid:12)(cid:12) z ( k +1)

Original:  [Equation]
Revised: i [Equation]

Original: It is noted that (7) is also in the regular monotone APG (Beck & Teboulle, 2009a)  described in Algorithm 1 for convex problems.
Revised: It is noted that (7) is also in the regular monotone APG (Beck & Teboulle, 2009a) for convex problems described in Algorithm 4 in Section A.2 of the supplementary.

Original: Compared to Algorithm 1, the projection of u ( k ) onto a convex piece is used to compute z ( k +1) .
Revised: Compared to Algorithm 4, the projection of u ( k ) onto a convex piece is used to compute z ( k +1) .

Original: In addition, we have a novel Negative-Curvature-Exploitation algorithm described in Algorithm 3 which decides if PPGD should update the next iterate x k +1 .
Revised: In addition, we have a novel Negative-Curvature-Exploitation algorithm described in Algorithm 2 which decides if PPGD should update the next iterate x k +1 .

Original: In particular, the Negative-Curvature-Exploitation procedure  onlyreturns true if there is no change in convex pieces between z k +1 and w k or the objective value will be decreased by an absolute positive amount by setting x k +1 to z k +1 .
Revised: In particular, the Negative-Curvature-Exploitation procedure sets x ( k +1) = z ( k +1) if and only if there is no change in convex pieces between z k +1 and w k or the objective value will be decreased by an absolute positive amount by setting x k +1 to z k +1 .

Revised: The

Original: The Negative-Curvature-Exploitation algorithm guarantees that either the next iterate x ( k +1) remains in the same convex pieces as x ( k ) , or the objective value is decreased by letting some coordinate of x ( k +1) transit to another convex piece.
Revised: Negative-Curvature-Exploitation algorithm guarantees that either the next iterate x ( k +1) remains in the same convex pieces as x ( k ) , or the objective value is decreased by letting some coordinate of x ( k +1) transit to another convex piece.

Original: Such guarantee will be proved in our theoretical analysis in the next section.
Revised: Such guarantee will be proved in our theoretical analysis in the next section.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, we present the analysis for the convergence rate of PPGD in Algorithm 2.
Revised: In this section, we present the analysis for the convergence rate of PPGD in Algorithm 1.

Original: Before that, we present the definition of Fr ´ echet subdifferential which is useful for the proof of necessary lemmas.
Revised: Before that, we present the definition of Fr´echet subdifferential which is useful for the proof of necessary lemmas.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Fr ´ echet subdifferential and critical points (Rockafellar & Wets, 2009))
Revised: (Fr´echet subdifferential and critical points (Rockafellar & Wets, 2009))

Original: Given a non-convex function u : R d → R ∪ { + ∞} which is a proper and lower semicontinuous function.
Revised: Given a non-convex function u : R d → R ∪ { + ∞} which is a proper and lower semicontinuous function.

Original: • For a given x ∈ dom u , its Fr ´ echet subdifferential of u at x , denoted by ˜ ∂u ( x ) , is the set of all vectors u ∈ R
Revised: • For a given x ∈ dom u , its Fr´echet subdifferential of u at x , denoted by ˜ ∂u ( x ) , is the set of all vectors u ∈ R

Original: d
Revised: d

Original: which satisfy
Revised: which satisfy

Original: [Equation]
Revised: [Equation]

Original: • The limiting-subdifferential of u at x ∈ R d , denoted by written ∂u ( x ) , is defined by
Revised: • The limiting-subdifferential of u at x ∈ R d , denoted by written ∂u ( x ) , is defined by

Original: [Equation]
Revised: [Equation]

Original: The point x is a critical point of u if 0 ∈ ∂u ( x ) .
Revised: The point x is a critical point of u if 0 ∈ ∂u ( x ) .

Original: The following assumption is useful for our analysis when the convex pieces have one or more endpoints at which f is continuous.
Revised: The following assumption is useful for our analysis when the convex pieces have one or more endpoints at which f is continuous.

Original: Assumption 2 (Nonvanishing Gradient at Continuous Endpoints) .
Revised: Assumption 2 (Nonvanishing Gradient at Continuous Endpoints) .

Original: Let q m with m ∈ [ M − 1] be an endpoint at which f is continuous.
Revised: Let q m with m ∈ [ M − 1] be an endpoint at which f is continuous.

Original: There exists a positive constant ε 0 such that the following two conditions hold.
Revised: There exists a positive constant ε 0 such that the following two conditions hold.

Original: If q m is a left endpoint, then inf x ∈ R d , x i =
Revised: If q m is a left endpoint, then inf x ∈ R d , x i =

Original: q m | [ ∇ g ( x )]
Revised: q m | [ ∇ g ( x )]

Original: i + p + | ≥ ε 0 for all
Revised: i + p + | ≥ ε 0 for all

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: If q m is a right endpoint, then inf x ∈ R d , x i =
Revised: If q m is a right endpoint, then inf x ∈ R d , x i =

Original: q m | [ ∇ g ( x )]
Revised: q m | [ ∇ g ( x )]

Original: i + p − | ≥ ε 0 .
Revised: i + p − | ≥ ε 0 .

Original: Here p + := lim y
Revised: Here p + := lim y

Original: → q + m f ′ ( y ) , and p − :=
Revised: → q + m f ′ ( y ) , and p − :=

Original: lim y
Revised: lim y

Original: → q − m f ′ ( y ) .
Revised: → q − m f ′ ( y ) .

Original: We note that Assumption 2 is rather mild.
Revised: We note that Assumption 2 is rather mild.

Original: For example, when f ( x ) = λ min {| x | , b } is the cappedℓ 1 penalty, then Assumption 2 holds when λ > G and ε 0 can be set to λ − G .
Revised: For example, when f ( x ) = λ min {| x | , b } is the cappedℓ 1 penalty, then Assumption 2 holds when λ > G and ε 0 can be set to λ − G .

Original: Such requirement for large λ is commonly used for analysis of consistency of sparse linear models, such as (Loh, 2017).
Revised: Such requirement for large λ is commonly used for analysis of consistency of sparse linear models, such as (Loh, 2017).

Original: In addition, when there are no endpoints at which f is continuous, then Assumption 2 is not required for the provable convergence rate of PPGD.
Revised: In addition, when there are no endpoints at which f is continuous, then Assumption 2 is not required for the provable convergence rate of PPGD.

Original: Details are deferred to Remark 3.6.
Revised: Details are deferred to Remark 3.5.

Original: We then have thean important Theorem 3.4, which directly follows by Lemma 3.1, Lemma 3.2, and 
Revised: We then have the important Theorem 3.3, which directly follows by Lemma 3.1 and Lemma 3.

Original: Lemma 3.3 below.
Revised: below.

Original: Theorem 3.4 states that the Negative-Curvature-Exploitation algorithm guarantees that, if convex pieces change across consecutive iterates x ( k ) and x ( k +1) , then the objective value is decreased by a positive amount.
Revised: Theorem 3.3 states that the Negative-Curvature-Exploitation algorithm guarantees that, if convex pieces change across consecutive iterates x ( k ) and x ( k +1) , then the objective value is decreased by a positive amount.

Original: Lemma 3.1. Suppose s < min (cid:110) [Equation] ) ̸ = P
Revised: Before presenting these results, we note that the level set

Original: [Equation] x  ( [Equation] [Equation]
Revised: L := (cid:8) x (cid:12)(cid:12) F ( x ) ≤ F ( x (0) )

Original: = m i for some k ≥and some i ∈ [ d ] , and f is not continuous at q ( w ( k ) i ) . Then f ( q ( w ( k ) i )) <
Revised: (cid:9) is bounded because F is coercive.

Original: lim y → q ( w ( k ) i ) − f ( y ) if q ( w ( k ) i ) is  the right endpoint of R m i for 1 ≤ m i < M , and f ( q ( w ( k ) i ))
Revised: Since ∇ g is smooth, we define the supremum of ∥∇ g ∥ 2 over the closure of L as G :

Original: < lim y
Revised: [Equation]

Original: → q ( w ( k ) i ) + f ( y ) if q ( w ( k ) i ) is  the left endpoint of R m i for 1 < m i ≤ M .
Revised: G is finite due to the fact that L R 0 is compact.

Original: Lemma 3.2 (Decrease of f at continuous endpoint) .
Revised: Lemma 3.1 (Decrease of the objective function when convex pieces change) .

Original: Suppose[-s < min (cid:110) s 
 F + G , ε 
 L g (1 − w-]
Revised: Suppose[-s < min (cid:110) s 
 F + G , ε 
 L g (1 − w-]

Original: [Equation]
Revised: [Equation]

Original: )( G [Equation] F ) (cid:111) ,
Revised: ) (cid:13)(cid:13) 2 ≤ G .

Original:  P ( x ( k +1)i ) ̸
Revised: If P ( x ( k +1) ) ̸

Original: = P ( x ( k )i )
Revised: = P ( x ( k ) )

Original: = m i for some k ≥ 0 and some

Original: i ∈ [ d ] , and f is continuous at q ( w ( k ) i ) . Then f ( x ( k+1) i ) ≤ f m i ( z ( k+1) i ) . Furthermore, if d i, 1 ≥w 0 d i, 0 , then f ( x ( k +1) i )
Revised: for some k ≥ 0 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 0 generated by the PPGD algorithm, then under Assumption 1 and Assumption 2,

Original: [Equation]
Revised: [Equation]

Original: f

Original: [Equation]
Revised: [Equation]

Original: ( z (

Original: [Equation] [Equation] [Equation] , (12) where κ  1 := sCw 0 ( ε 0 − sL g (1 − w 0 )( G + F )) .
Revised: where κ := min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε 0 − sL g (1 − w 0 )

Original: Lemma 3.3 (Decrease of f at discontinuous endpoints) .

Original: Suppose s < min (cid:110) s 0 G + F , J 2 F ( G + F ) (cid:111) , P ( x ( k +1) i ) ̸ = P ( x ( k ) i )
Revised: ( G + F 0 )) , κ 2 :=

Original: = m i for some k ≥ 0and some i ∈ [ d ] . Suppose the following condition hold: when q ( w ( k ) i ) is a right endpoint of R m i , then f ( q ( w ( k ) i )) <
Revised: J − 2 sF 0 ( G + F 0 ) .

Original: lim y → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of R m i , then f ( q ( w ( k ) i )) < lim y
Revised: Lemma 3.2.

Original: → q ( w ( k ) i ) + f ( y ) . Then f ( x ( k +1) i
Revised: The sequences (cid:8) x ( k ) (cid:9) k ≥ 1 and (cid:8) w ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm described in Algorithm 1 satisfies

Original: [Equation]
Revised: [Equation]

Original: ( k

Original: [Equation]

Original: 13) where κ 2 : = J − sF ( G + F ) . Theorem 3.4 (Decrease of the objective function when convex pieces change) .
Revised: Theorem 3.3.

Original: Suppose
Revised: Suppose

Original: [Equation]
Revised: [Equation]

Original: If P ( x ( k +1) ) ̸
Revised: If P ( x ( k +1) ) ̸

Original: = P ( x ( k ) )
Revised: = P ( x ( k ) )

Original: for some k ≥ 0 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 0 generated by the PPGD algorithm, then 
Revised: for some k ≥ 0 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 0 generated by the PPGD algorithm, then under Assumption 1 and Assumption 2,

Original: [Equation]
Revised: [Equation]

Original: where κ := min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε0 − sL g
Revised: where κ := min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε

Original: [Equation]
Revised: [Equation]

Original: 1 − 
Revised: − sL g

Original: [Equation]
Revised: [Equation]

Original: ( G [Equation] [Equation] [Equation]
Revised: 1 − w 0 )( G + F 0 )) , κ

Original: =

Original: [Equation]
Revised: [Equation]

Original:  −  sF  ( G + F  ) .
Revised: = J − 2 sF 0 ( G + F 0 ) .

Original: Due to Theorem 3.4, there can only be a finite number of k ’s such that the convex pieces change across consecutive iterates x ( k ) and x ( k +1) .
Revised: Due to Theorem 3.3, there can only be a finite number of k ’s such that the convex pieces change across consecutive iterates x ( k ) and x ( k +1) .

Original: As a result, after finite iterations all the iterates generated by PPGD lie on the same convex pieces, so the nice properties of convex optimization apply to these iterates.
Revised: As a result, after finite iterations all the iterates generated by PPGD lie on the same convex pieces, so the nice properties of convex optimization apply to these iterates.

Original: Formally, we have the following theorem stating the convergence rate of PPGD.
Revised: Formally, we have the following theorem stating the convergence rate of PPGD.

Original: Theorem 3.5 (Convergence rate of PPGD) .
Revised: Theorem 3.4 (Convergence rate of PPGD) .

Original: Suppose s satisfies (14), Assumption 1 and Assump- tion 2 hold.
Revised: Suppose s < min (cid:110) s [Equation] with s 1 defined by (16), Assumption 1 and Assumption 2 hold.

Original: Then there exists a finite k 0 ≥ 1 such that the following statements hold.
Revised: Then there exists a finite k 0 ≥ 1 such that the following statements hold.

Original:  P ( x ( k ) )
Revised: (1) P ( x ( k ) )

Original: = m ∗ for some m ∗ ∈ N d for all k ≥ k 0 .
Revised: = m ∗ for some m ∗ ∈ N d for all k > k 0 .

Original: (2) Let  x∗ with P ( x ∗ )
Revised: (2) Let Ω := (cid:110) x

Original:  = m 
Revised: (cid:12)(cid:12)(cid:12) x is a limit point of (cid:8) x ( k ) (cid:9) k ≥ 1 , P ( x ) = m ∗ (cid:111) be the set of all limit points of the sequence (cid:8) x

Original: ∗ be a limiting point of { x ( k ) } ∞ k =0 generated by Algorithm 2, then
Revised: ( k ) (cid:9) k ≥ 1 generated by Algorithm 1. Then for any x ∗ ∈ Ω , F ( x ∗ ) = inf { x ∈ R d | P ( x )= m ∗ } F ( x ) , and F (

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  for all k > k 0 , where U ( k 0 ) := (cid:0) 12 s ∥ ( t k 0 − 1 − 1) x ( k 0 − 1) − t k 0 − 1 z ( k 0 )
Revised: (18) for all k > k 0 , where U ( k 0 ) : = (cid:0) 12 s ∥ ( t k 0 − 1 − 1) x ( k 0 − 1) − t k 0 − 1 z ( k 0 )

Original: + x ∗ ∥ 22 + t 2 k 0 − 1 ( F ( x ( k 0 ) )
Revised: + x ∗ ∥ 22 + t 2 k 0 − 1 ( F ( x ( k 0 ) )

Original: −
Revised: −

Original: F ( x ∗ ))
Revised: F ( x ∗ ))

Original: (cid:1) .
Revised: (cid:1) .

Revised: Moreover, if f m ∗ i does not take the third case in (5) or (6) for all i ∈

Revised: [ d ] , then x ∗ is a critical point of F , that is, 0 ∈ ∂F ( x ∗ ) .

Original: Remark 3.6.
Revised: Remark 3.5.

Original: If the convex pieces {R m } Mm =1 has no endpoints at which f is continuous, then Theorem 3.5 holds without requiring Assumption 2.
Revised: If the convex pieces {R m } Mm =1 has no endpoints at which f is continuous, then Theorem 3.4 holds without requiring Assumption 2.

Original: In particular, Theorem 3.5 holds without Assumption 2 for problem (1) with capped- ℓ 1 regularizer or ℓ 0 -norm regularizer.
Revised: In particular, Theorem 3.4 holds without Assumption 2 for problem (1) with indicator penalty or ℓ 0 -norm regularizer.

Original:  EXPERIMENTAL RESULTS
Revised:  EXPERIMENTAL RESULTS

Original:  We conduct empirical study using the proposed PPGD methods in this section.
Revised: [Figure] We conduct empirical study using the proposed PPGD methods in this section.

Original: PPGD and the baseline optimization methods are used to solve the regularized logistic regression problem, [Equation] i =1 log(1 + exp( − y i x ′ x i ))
Revised: PPGD and the baseline optimization methods are used to solve the regularized logistic regression problem, min x 1 n n (cid:80) i =1 log(1 + exp( − y i x ′ x i ))

Original: + h ( x ) , where g ( x ) = 1 n n (cid:80) i
Revised: + h ( x ) , where g ( x ) = 1 n n (cid:80) i

Original: =
Revised: =

Original: 1 log(1 + exp( − y i x ′ x i )) , and[Figure] [Figure] h ( x )=
Revised: 1 log(1 + exp( − y i x ′ x i )) , and h ( x )

Original:  (cid:80) di
Revised: = (cid:80) di

Original: =1 f ( x i ) for three choices of f ( x ) : the indicator penalty, f ( x ) =
Revised: =1 f ( x i ) for three choices of f ( x ) : the indicator penalty, f ( x ) =

Original: λ 1I { x< 0 } ; the capped- ℓ penalty, f ( x ) =
Revised: λ 1I { x< 0 } ; the capped- ℓ penalty, f ( x ) =

Original: λ min {| x | , b } ; the ℓ 0 -norm, f ( x ) =
Revised: λ min {| x | , b } ; the ℓ 0 -norm, f ( x ) =

Original: λ 1I { x ̸ =0 } .
Revised: λ 1I { x ̸ =0 } .

Original: Three three choices for f correspond to nonnegative logistic regression, capped-
Revised: Three three choices for f correspond to nonnegative logistic regression, capped-

Original: ℓ regularized logistic regression, and ℓ 0 -regularized logistic regression. 
Revised: ℓ regularized logistic regression, and ℓ 0 -regularized logistic regression. We set λ = 0 .

Original: We conduct experiments on the MNIST handwritten digits dataset (LeCun et al., 2010) and the CIFAR-10 dataset (Krizhevsky, 2009). 
Revised: 2 and conduct experiments on the MNIST handwritten digits dataset (LeCun et al., 2010) and the CIFAR-10 dataset (Krizhevsky, 2009). [Figure]

Original: The MNIST data set contains 70 , 000 examples of handwritten digit images of 10 classes.
Revised: The MNIST data set contains 70 , 000 examples of handwritten digit images of 10 classes.

Original: Each image is of size 28 × 28 and represented by a 784 -dimensional vector.
Revised: Each image is of size 28 × 28 and represented by a 784 -dimensional vector.

Original: The CIFAR-10 dataset contains 60000 images of size 32 × 32 with 10 classes.
Revised: The CIFAR-10 dataset contains 60000 images of size 32 × 32 with 10 classes.

Original: For each dataset, we randomly sample two classes with 5000 images in each class, which form the training data { x i , y i } 10000 i = with { x i } 10000 i = being the sampled images and { y i } 10000 i = being the corresponding class labels ( ± 1 ).
Revised: For each dataset, we randomly sample two classes with 5000 images in each class [Figure] which form the training data { x i , y i } 10000 i = with { x i } 10000 i = being the sampled images and { y i } 10000 i = being the corresponding class labels ( ± 1 )

Original: We compare PPGD to Proximal Gradient Descent (PGD), the regular monotone Accelerated Proximal Gradient (APG) described in Algorithm1, and the monotone Accelerated Proximal Gradient (mAPG) algorithm
Revised: We compare PPGD to Proximal Gradient Descent (PGD), the regular monotone Accelerated Proximal Gradient (APG) described in Algorithm

Original: (Li & Lin, 2015).
Revised: [Figure]

Original:  Due to the observation that monotone version of accelerated gradient methods usually converge faster than its nonmonotone counterparts, the regular nonmonotone APG and the nonmonotone Accelerated Proximal Gradient (nmAPG) method (Li & Lin, 2015) are not included in the baselines.
Revised: and the monotone Accelerated Proximal Gradient (mAPG) algorithm (Li & Lin, 2015). Due to the observation that monotone version of accelerated gradient methods usually converge faster than its nonmonotone counterparts, the regular nonmonotone APG and the nonmonotone Accelerated Proximal Gradient (nmAPG) method (Li & Lin, 2015) are not included in the baselines.

Original: PPGD is also compared to the fast PGD method (Yang & Yu, 2020), referred to as
Revised: PPGD is also compared to the fast PGD method (Yang & Yu, 2020), referred to as

Original: APGD- ℓ 0 , in our experiments for ℓ 0 -regularized logistic regression.
Revised: APGD- ℓ 0 , in our experiments for ℓ 0 -regularized logistic regression.

Original: Figure 3 and Figure 4 illustrate the decrease of objective value with respect to the iteration number on the MNIST dataset and the
Revised: Figure 3 and Figure 4 illustrate the decrease of objective value with respect to the iteration number on the MNIST dataset and the

Original: CIFAR-10 dataset respectively.
Revised: CIFAR-10 dataset respectively.

Original: It can be observed that PPGD always converges faster than all the competing baselines, evidencing the proved fast convergence results in this paper.
Revised: It can be observed that PPGD always converges faster than all the competing baselines, evidencing the proved fast convergence results in this paper

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We present fast Projective Proximal Gradient Descent (PPGD) methods to solve a class of challenging nonconvex and nonsmooth optimization methods.
Revised: We present fast Projective Proximal Gradient Descent (PPGD) methods to solve a class of challenging nonconvex and nonsmooth optimization methods.

Original: Using a novel projection operator and a carefully designed Negative-Curvature-Exploitation algorithm, PPGD achieves Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function.
Revised: Using a novel projection operator and a carefully designed Negative-Curvature-Exploitation algorithm, PPGD locally matches Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function.

Original: The effectiveness of PPGD is evidenced by experiments.
Revised: The effectiveness of PPGD is evidenced by experiments

Original: A ALGORITHMS IN THE PAPER A. P ROXIMAL G RADIENT D ESCENT
Revised: A M ORE DETAILS ABOUT THIS PAPER A. K URDYKA -Ł OJASIEWICZ

Revised: KŁ P ROPERTY

Revised: Definition A.1 (Kurdyka-Łojasiewicz (KŁ) Property) .

Revised: Let u : R d → ( −∞ , ∞ ] be proper and lower semicontinuous.

Revised: u is said to satisfy the Kurdyka-Lojasiewicz (KŁ) property at u ∈ dom ∂u := { u ′ ∈ R d : ∂u ( u ′ ) ̸ = ∅} , if there exist η ∈ (0 , ∞ ] , a neighborhood U of u and a function ϕ ∈

Revised: Φ η , such that for all ¯ u ∈ U ∪ [ u ( u ) <

Revised: u (¯ u ) < u ( u ) + η ] , the following inequality holds:

Revised: [Equation]

Revised: Here η ∈ (0 , ∞ ] , Φ η is defined to be the class of all concave and continuous functions ϕ : [0 , η ) → R + which satisfy the following conditions:

Revised: ϕ (0) = 0 ; 2.

Revised: ϕ is continuously differentiable on (0 , η ) and continuous at 0 ; 3.

Revised: ϕ ′ ( r ) > 0 for all r ∈ (0 , η ) .

Original:  In the k -th iteration of PGD for k ≥ 1 , gradient descent is performed on the squared loss term g ( x ) to obtain an intermediate variable as the result of gradient descent, i.e. x ( k ) − s ∇ g ( x ( k ) ) , where s > 0 is the step size, and 1 s is usually chosen to be larger than L g .
Revised: If u satisfies the KŁ property at each point of dom ∂u , then u is called a KŁ function. A. P ROXIMAL G RADIENT D ESCENT (PGD) AND MONOTONE A CCELERATED P ROXIMAL G RADIENT (APG) D ESCENT We describe PGD and the regular monotone APG in this subsection. In the k -th iteration of PGD for k ≥ 1 , gradient descent is performed on the squared loss term g ( x ) to obtain an intermediate variable as the result of gradient descent, i.e. x ( k ) − s ∇ g ( x ( k ) ) , where s > 0 is the step size, and 1 s is usually chosen to be larger than L g .

Original: x ( k +1) is computed by proximal mapping on x ( k ) − s ∇ g ( x ( k ) ) .
Revised: x ( k +1) is computed by proximal mapping on x ( k ) − [Equation] g ( x ( k ) ) [Equation]

Original: 1: for k=1,..., do
Revised: 1: for k=1,..., do

Original: [Equation]
Revised: [Equation]

Revised: 1: z (1) =

Revised: x (1) =

Revised: x (0) , t 0 = 0.

Revised: 2: for k = 1 , . . .

Revised: , do

Revised: [Equation]

Original:  The iterations start from k = 1 and continue until the sequence { F ( x ( k ) )
Revised: x [Equation] otherwise . The iterations start from k = 1 and continue until the sequence { F ( x ( k ) )

Original: } k or { x ( k ) } k converges or maximum iteration number is achieved.
Revised: } k or { x ( k ) } k converges or maximum iteration number is achieved.

Original: The optimization algorithm for the ℓ 0 sparse approximation problem (1) by PGD is described in Algorithm 4.
Revised: The optimization algorithm for the ℓ 0 sparse approximation problem (1) by PGD is described in Algorithm 3.

Original: In practice, the time complexity of optimization by PGD is O ( Mdn ) where M is the number of iterations (or maximum number of iterations) for
Revised: In practice, the time complexity of optimization by PGD is O ( Mdn ) where M is the number of iterations (or maximum number of iterations) for

Original: PGD.
Revised: PGD.

Revised: The regular monotone Accelerated Proximal Gradient (APG) descent algorithm is described in Algorithm 4.

Original: B P ROOFS
Revised: B P ROOFS

Original: The following lemma shows that the Fermat’s rule still applies to functions in the sense of Fr ´ echet subdifferential, that is, local mininum has 0 in its Fr´echet subdifferential and also its (limiting- )subdifferential.
Revised: The following lemma shows that the Fermat’s rule still applies to functions in the sense of Fr´echet subdifferential, that is, local mininum has 0 in its Fr´echet subdifferential and also its (limiting- )subdifferential.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: (Fermat’s rule for Fr ´ echet subdifferential and (limiting-)subdifferential) Let u be a real-valued function defined on Ω ⊆ R d , u := Ω → R , has a local minimum point at x ∈ Ω .
Revised: (Fermat’s rule for Fr´echet subdifferential and (limiting-)subdifferential, also in (Rockafellar & Wets, 2009, Theorem 10.1)) Let u be a real-valued function defined on Ω ⊆ R d , u := Ω → R , has a local minimum point at x ∈ Ω .

Original: Then 0 ∈ ˜
Revised: Then 0 ∈ ˜

Original: ∂u ( x ) ⊆
Revised: ∂u ( x ) ⊆

Original: ∂u ( x ) .
Revised: ∂u ( x ) .

Original: Proof.
Revised: Proof.

Original: Since x ∈ Ω is a local minimum point, then there exisits a neighborhood x ∈ U ( x ) such that u ( y ) ≥ u ( x ) for all y ∈ U ( x ) .
Revised: Since x ∈ Ω is a local minimum point, then there exisits a neighborhood x ∈ U ( x ) such that u ( y ) ≥ u ( x ) for all y ∈ U ( x ) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: By (18), 0 ∈ ˜ ∂u ( x ) .
Revised: By (20), 0 ∈ ˜ ∂u ( x ) .

Original: Because ˜ ∂u ( x ) ⊆
Revised: Because ˜ ∂u ( x ) ⊆

Original: ∂u ( x ) , 0 ∈ ˜ ∂u ( x ) ⊆
Revised: ∂u ( x ) , 0 ∈ ˜ ∂u ( x ) ⊆

Original: ∂u ( x ) .
Revised: ∂u ( x ) .

Original: Proof.
Revised: We need to have the following two lemmas before proving Lemma 3.1.

Original: We prove the case when q ( w ( k ) i ) is the right endpoint of R m i .The case when q ( w ( k ) i ) is the left endpoint of R m i can be proved in a similar manner.
Revised: Lemma B.2 (Decrease of f at continuous endpoint) .

Original: Suppose the claimed result f ( q ( w ( k ) i )) < {+min (cid:110) s 0 G + F 0 , ε 
+}
Revised: Suppose Assumption 1 and Assumptionhold, s < {+min (cid:110) s 0 G + F 0 , ε 
+}

Original: lim y → q ( w ( k )i ) − f ( y ) does not hold, so we must have lim y → q ( w ( k ) i )
Revised: L g (1 − w 0 )( G + F 0 ) (cid:111) , (cid:13)(cid:13) ∇ g ( w ( k ) ) (cid:13)(cid:13) 2 ≤ G , P ( x ( k +1) i )

Original: + f ( y ) > f ( q ( w ( k ) i )) .
Revised: ̸

Original: In this case, x ( k +1) i  = z ( k +1)
Revised: = P ( x ( k ) i ) = m i for some k ≥ 0 and some

Original: i  , and z ( k +1) i ∈ ( q ( w ( k ) i ), ∞ ) .Define p ( v ) :
Revised: i ∈ [ d ] , and f is continuous at q ( w ( k ) i ) .

Original: = 12 s (cid:0) v − (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i (cid:1) 2 + f m i ( v ) . 
Revised: Then f ( x ( k +1) i ) ≤ f m i ( z ( k +1) i ) . Furthermore, if d i, 1 ≥ w 0 d

Original: We have z
Revised: i, 0 , then

Original: [Equation]
Revised: [Equation]

Original: i (cid:17) . The optimality of z ( k +1)
Revised: where κ 1 := sCw 0 ( ε 0 − sL g (1 − w 0 )

Original: i by Fermat’s rule indicates that z ( k +1) i = [Equation] w [Equation]
Revised: ( G + F 0 )) .

Original: k ) − s ∇ g ( w ( k ) (cid:3) i ,and p ( z ( k +1) i ) ≤ p ( w ( k ) i ) . On the other hand, we have
Revised: Lemma B.3 (Decrease of f at discontinuous endpoints) . Suppose Assumption 1 holds, s < min (cid:110) s 0 G + F 0 ,

Original: [Equation]
Revised: [Equation]

Original: and [Equation]
Revised: and some i ∈ [ d ] .

Original: (21) follows because w ( k ) i ∈ ( q ( w ( k ) i ) − s 0 , q ( w ( k ) i )) due to s < s 0 G . It follows by (20) and (21) that p ( z (
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: because s <
Revised: where κ 2 :

Original: J FG + G 2 / 2 . This contradiction shows that we must have lim y → q ( w ( k ) i )
Revised: =

Original: + f ( y ) ≤ f ( q ( w ( k )i )) .
Revised: J − 2 sF 0 ( G + F 0 ) .

Original: B. P ROOF OF L EMMA 3. 
Revised: B. P ROOF OF L EMMA B.

Original: Proof.
Revised: Proof of Lemma B.2 .

Original: Let q = q ( w ( k ) i ) .
Revised: Let q = q ( w ( k ) i ) .

Original: We must have x ( k +1) i =
Revised: We must have x ( k +1) i =

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: k +1)
Revised: k +1)

Original: i with the PPGD algorithm and the Negative-Curvature-Exploitation algorithm described in Algorithm 2 and Algorithm 3.
Revised: i with the PPGD algorithm and the Negative-Curvature-Exploitation algorithm described in Algorithm 1 and Algorithm 2.

Original: Without loss of generality, we assume x ( k +1)
Revised: Without loss of generality, we assume x ( k +1)

Original: i ∈
Revised: i ∈

Original: R
Revised: R

Original: + m i .
Revised: + m i .

Original: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.
Revised: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original:  bythe definition (5), where v − =
Revised: (23) by definition (5), where v − =

Original: lim x → q − f ′ ( x ) .
Revised: lim x → q − f ′ ( x ) .

Original: On the other hand, we have z ( k +
Revised: On the other hand, we have z ( k +

Original: [Equation]
Revised: [Equation]

Original: It follows by (24) that
Revised: It follows by (24) that

Original: [Equation]
Revised: [Equation]

Original: As a result, s < s 0 G + F  guarantees that x ( k +1) i ∈ ( q, q + s 0 ) .
Revised: As a result, s < s 0 G + F 0 guarantees that x ( k +1) i ∈ ( q, q + s 0 ) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation][Equation]
Revised: [Equation]

Original: when x ∈ ( q, x ( k +1) i ) .
Revised: when x ∈ ( q, x ( k +1) i ) .

Original: Letting x
Revised: Letting x

Original: → q in (26), we have
Revised: → q in (26), we have

Original: [Equation] 
Revised: [Equation] [Equation]

Original: 27) where v + = lim x → q + f ′ ( x ) .
Revised: where v + = lim x → q + f ′ ( x ) .

Original: By the negative curvature condition in Assumption 1(d), we have v − − v + ≥ C > 0 , therefore,
Revised: By the negative curvature condition in Assumption 1(d), we have v − − v + ≥ C > 0 , therefore,

Original: [Equation]
Revised: [Equation]

Original: Because z ( k +1) i ∈ 
Revised: Because z ( k +1) i ∈ R

Original: [Equation]

Original: + m [Equation] +
Revised: + m i , z ( k

Original: [Equation]
Revised: [Equation]

Original: i ≥ w ( k ) i .
Revised: i ≥ w ( k ) i .

Original: By (28) we have
Revised: By (28) we have

Original: [Equation]
Revised: [Equation]

Original: Let ˜ w ∈ R d with ˜ w j = w ( k ) j for all j ̸
Revised: Let ˜ w ∈ R d with ˜ w j = w ( k ) j for all j ̸

Original: = i , and ˜ w i = q .
Revised: = i , and ˜ w i = q .

Original: By the definition of ˜ w and (24), we have
Revised: By the definition of ˜ w and (24), we have

Original: [Equation]
Revised: [Equation]

Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (30).
Revised: where the last inequality follows from (30).

Original: It follows by (28) and (31) thatf ( x ( k
Revised: It follows by (28) and (31) that

Original: [Equation]
Revised: [Equation]

Original: Noting that x ( k +1)
Revised: Noting that x ( k +1)

Original: i =
Revised: i =

Original: z ( k +1)
Revised: z ( k + [Equation]

Original: i completes the proof.
Revised: i completes the proof.

Original: B. P ROOF OF L EMMA 3. 
Revised: B. P ROOF OF L EMMA B.

Original: Proof.
Revised: Before presenting the proof, we introduce and prove the following lemma. Lemma B.4. Suppose Assumption 1 holds, (cid:12)(cid:12)(cid:0) ∇ g ( w ( k ) ) (cid:1) i (cid:12)(cid:12) ≤ G , and s < min (cid:110)

Revised: [Equation]

Revised: G + G 2 / 2 , s 0 G (cid:111) ,

Revised: P ( x ( k +1) i ) ̸

Revised: = P ( x ( k ) i )

Revised: = m i for some k ≥ 0 and some

Revised: i ∈ [ d ] , and f is not continuous at q ( w ( k ) i ) .

Revised: Then f ( q ( w ( k ) i )) <

Revised: lim y

Revised: → q ( w ( k ) i ) − f ( y ) if q ( w ( k ) i ) is the right endpoint of R

Revised: m i for 1 ≤ m i < M , and f ( q ( w ( k ) i ))

Revised: <

Original: Let q = q ( w ( k ) i ) , and we have x ( k +1) i 
Revised: lim y → q ( w ( k ) i ) + f ( y ) if q ( w ( k ) i ) is the left endpoint of R m i for 1 <

Revised: m i ≤ M .

Revised: Proof of Lemma B.4 .

Original: =
Revised: We prove the case when q ( w ( k ) i ) is the right endpoint of R m i . The case when q ( w ( k ) i ) is the left endpoint of R m i can be proved in a similar manner. Suppose the claimed result f ( q ( w ( k ) i )) <

Revised: lim y

Revised: → q ( w ( k ) i ) − f ( y ) does not hold, so we must have lim y

Revised: → q ( w ( k ) i )

Revised: + f ( y ) >

Revised: f ( q ( w ( k ) i )) .

Revised: In this case, x ( k +1) i =

Original: z ( k +1)
Revised: z ( k +1)

Revised: i , and z ( k +1) i ∈ ( q ( w ( k ) i ) , ∞ ) .

Revised: Define p ( v ) :

Original:  i  .
Revised: = 12 s (cid:0) v − (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i (cid:1) 2 + f m i ( v ) . We have z ( k +1) i = [Equation] (cid:16)(cid:104) w ( k ) − s ∇ g ( w ( k ) (cid:105) i (cid:17) .

Revised: The optimality of z ( k +1)

Revised: i by Fermat’s rule indicates that z ( k +1) i

Revised: = (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i , and p ( z ( k +1) i ) ≤ p ( w ( k ) i ) .

Revised: We have w ( k ) i

Revised: [Equation]

Revised: to

Revised: [Equation]

Revised: On the other hand, we have

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: (35) follows because .

Revised: It follows by (34) and (35) that

Revised: [Equation]

Revised: where 1 ⃝ follows from s <

Revised: [Equation]

Revised: This contradiction shows that we must have lim y

Revised: → q ( w ( k ) i )

Revised: + f ( y ) ≤ f ( q ( w ( k ) i )) .

Revised: Since f is not continuous at q ( w ( k ) i ) , we must have f ( q ( w ( k ) i ))

Revised: <

Revised: lim y

Revised: → q ( w ( k ) i ) − f ( y ) .

Revised: Proof of Lemma B.3 .

Revised: According to Lemma B.4, the following claim holds: when q ( w ( k ) i ) is a right endpoint of R

Revised: m i , then f ( q ( w ( k ) i )) <

Revised: lim y

Revised: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of

Revised: R m i , then f ( q ( w ( k ) i )) <

Revised: lim y

Original:  Without loss of generality, we assume x ( k +1)
Revised: → q ( w ( k ) i ) + f ( y ) . Let q = q ( w ( k ) i ) . Without loss of generality, we assume x ( k +1)

Original: i ∈
Revised: i ∈

Original: R
Revised: R

Original: + m i .
Revised: + m i .

Original: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.
Revised: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.

Original: We first consider the case that R m i  is not a single-point set, that is, R m i  ̸ = { q } .
Revised: We first consider the case that R m i +1 is not a single-point set, that is, R m i +1 ̸ = { q } .

Original: According to the definition of surrogate function (5), we havef
Revised: According to the definition of surrogate function (5), we have

Original: [Equation]
Revised: [Equation]

Original: where v − =
Revised: where v − =

Original: lim x → q − f ′ ( x ) .
Revised: lim x → q − f ′ ( x ) .

Original: If x ( k +1) i ̸
Revised: If x ( k +1) i ̸

Original: = q , applying the argument in (25) in the proof of
Revised: = q , applying the argument in (25) in the proof of

Original: Lemma 3.2, s < s 0 G + F  guarantees that x ( k +1) i ∈( q [Equation]
Revised: Lemma B.2, s < s 0 G + F 0 guarantees that x ( k +1) i ∈

Original:  q [Equation] s 0 ) .
Revised: ( q, q + s 0 ) .

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: when x ∈ ( q, x ( k +1) i ) .
Revised: when x ∈ ( q, x ( k +1) i ) .

Original: Letting x
Revised: Letting x

Original: → q in (34), we have
Revised: → q in (38), we have

Original: [Equation]
Revised: [Equation]

Original: where v + = lim x → q + f ′ ( x ) .
Revised: where v + = lim x → q + f ′ ( x ) .

Original: In addition,
Revised: In addition,

Original: It follows by (35) and (25) in the proof of Lemma 3.2 that
Revised: It follows by (39) and (25) in the proof of Lemma B.2 that

Original: [Equation]
Revised: [Equation]

Original: We note that (36) holds for all x ( k +1) i ∈
Revised: We note that (40) holds for all x ( k +1) i ∈

Original: [ q, q + s 0 )
Revised: [ q, q + s 0 )

Original: Moreover, it follows by (33) and (25) in the proof of Lemma 3.2 that 
Revised: Moreover, it follows by (37) and (25) in the proof of Lemma B.2 that f m i

Original: [Equation]
Revised: [Equation]

Original: Combining (36) and (37), we have [Equation] − [Equation]
Revised: i

Original: sF

Original: [Equation]
Revised: [Equation]

Original: G

Original: [Equation] ( z ( [Equation] +
Revised: Combining (40) and (41), we have f ( x ( k +1)

Original: [Equation] ) − ( J [Equation] G
Revised: i

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: If R m i  = { q } , then x ( k +1) i = q and f ( x ( k +1) i ) = f ( q ) .
Revised: If R P ( z ( k +1) i ) = { q } , then x ( k +1) i = q and f ( x ( k +1) i ) = f ( q ) .

Original: By the fact thatf
Revised: By the fact that

Original: [Equation]
Revised: [Equation]

Original: i [Equation] z [Equation]
Revised: and (cid:12)(cid:12)(cid:12) z ( k +1)

Original: k +
Revised: i −

Original: [Equation]
Revised: [Equation]

Original: − q ) and (cid:12)(cid:12)(cid:12) z ( k +1) i − q (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) z
Revised: ≤ (cid:12)(cid:12)(cid:12) z ( k +1) i − w ( k ) i (cid:12)(cid:12)(cid:12) , following the argument similar to (41) we have f m i

Original: (  k +1) i
Revised: ( z ( k +

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]

Original: , following the argument similar to (37) we have

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by combining (38) and (39).
Revised: The proof is completed by combining (42) and (43).

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF L EMMA 3.

Original: Proof.
Revised: Proof of Lemma 3.1 .

Original: We split [ d ] into three disjoint subsets, [ d ] = S 1 ∪ S 2 ∪ S 3 , and the three subsets are defined by 
Revised: We split [ d ] into three disjoint subsets, [ d ] = S 1 ∪ S 2 ∪ S 3 , and the three subsets are defined by S 1 := (cid:110) i ∈ [ d ]: P ( x ( k +1) i ) ̸

Revised: = P ( x ( k ) i ) , f is continuous at q ( w ( k ) i ) (cid:111)

Revised: [Equation]

Revised: is not

Original: [Equation]
Revised: [Equation]

Original: Let m i
Revised: Let m i

Original: =
Revised: =

Original: P ( x ( k ) i ) .
Revised: P ( x ( k ) i ) .

Original: According to Lemma 3.2, for all i ∈ S 1 such that d i, 1 ≥ w 0 d
Revised: According to Lemma B.2, for all i ∈ S 1 such that d i, 1 ≥ w 0 d

Original: i, 0 , we have
Revised: i, 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In addition, for all i ∈ S 1 , we have
Revised: In addition, for all i ∈ S 1 , we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 3.1, S 2 satisfies the condition that for all i ∈ S 2 , when q ( w ( k ) i ) is a right endpoint of R m i , then f ( q ( w ( k ) i )) <
Revised: According to Lemma B.4, S 2 satisfies the condition that for all i ∈ S 2 , when q ( w ( k ) i ) is a right endpoint of R m i , then f ( q ( w ( k ) i )) <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of R
Revised: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of R

Original: m i , then f ( q ( w ( k ) i )) <
Revised: m i , then f ( q ( w ( k ) i )) <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i )
Revised: → q ( w ( k ) i )

Original: + f ( y ) .
Revised: + f ( y ) .

Original: By Lemma 3.3, f
Revised: By Lemma B.3,

Original: [Equation]
Revised: [Equation]

Original: ≤ f m

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation] − κ [Equation] 44) for all i ∈ S 2 .
Revised: for all i ∈ S 2 .

Original: For all i ∈ S 3 , we have
Revised: For all i ∈ S 3 , we have

Original: [Equation] 
Revised: [Equation] [Equation]

Original: The Negative-Curvature-Exploitation algorithm described in Algorithm 3 guarantees that when
Revised: The Negative-Curvature-Exploitation algorithm described in Algorithm 2 guarantees that when

Original: P ( x ( k +1) )
Revised: P ( x ( k +1) )

Original: ̸
Revised: ̸

Original: = P ( x ( k ) ) , there exists at lease one i ∈ S 1 ∪ S 2 such that (42) or (44) holds.
Revised: = P ( x ( k ) ) , there exists at lease one i ∈ S 1 ∪ S 2 such that (45) or (47) holds.

Original: It follows by (42)-(45) and the above argument that
Revised: It follows by (45)-(48) and the above argument that

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from x ( k +1)

Original: =

Original: z ( k +1) .

Original: It follows from (46) that
Revised: It follows from (49) that

Original: [Equation]
Revised: [Equation]

Original: The PPGD algorithm guarantees that
Revised: The PPGD algorithm guarantees that

Original: [Equation]
Revised: [Equation]

Original: Therefore,  F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ  which completes the proof.
Revised: Therefore, we have F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ , which completes the proof.

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF L EMMA 3.

Revised: Proof of Lemma 3.2 .

Revised: The PPGD algorithm described in Algorithm 1 ensures that F ( x ( k ) ) ≤

Revised: F ( x ( k − 1) ) for k = 1 , and (cid:13)(cid:13) ∇ g ( w ( k ) )

Revised: (cid:13)(cid:13) 2 ≤ G for k = 1 .

Revised: Suppose that F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) ) (cid:13)(cid:13) 2 ≤ G hold for all 1 ≤ k ≤ k ′ with k ′ ≥ 1 .

Revised: With the chosen step size and the proof of Lemma 3.1, we have F ( x ( k ′ +1) )

Revised: ≤ F ( x ( k ′ ) ) .

Revised: This indicates that x ( k ′ +1)

Revised: ∈

Revised: L and w ( k ′ +1)

Revised: ∈

Original: Proof.
Revised: L R 0 , so (cid:13)(cid:13)(cid:13) w ( k ′ +1) (cid:13)(cid:13)(cid:13) 2 ≤ G . It follows by induction that F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) ) (cid:13)(cid:13) 2 ≤ G hold for all k ≥ 1 .

Revised: B. P ROOF OF T HEOREM 3.

Revised: Proof of Theorem 3.3 .

Revised: By Lemma 3.2, (cid:13)(cid:13) ∇ g ( w ( k ) )

Revised: (cid:13)(cid:13) 2 ≤ G holds for all k ≥ 1 .

Revised: Then the conclusion of this theorem directly follows from Lemma 3.1.

Revised: B. P ROOF OF T HEOREM 3.

Original: According to Theorem 3.4, when P ( x ( k+1) ) ̸
Revised: The following lemma is crucial in the proof of Theorem 3.4. It shows that after sufficient iterations, all the coordinates of x ( k ) belong to the same the convex pieces indexed by m ∗ , that is, P ( x ( k ) )

Revised: = m ∗ .

Revised: Moreover, the objective value F ( x ( k ) ) is not greater than the surrogate objective value F m ∗ ( z ( k ) )

Revised: =

Revised: g ( z ( k ) )

Revised: + (cid:80) di =1 f m ∗ i ( z ( k ) i ) .

Revised: Lemma B.5.

Original: = P ( x ( k ) ) , F ( x{+ε 
 L g+} ( k +1) ) ≤ F ( x ( k ) ) − κ .
Revised: Suppose Assumption 1 and Assumption 2 hold, and s < min (cid:110) s 1 , {+ε 
 L g+} ( G + √ dF 0 ) (cid:111) . Then there must exists a finite k ∈ N such that P ( x ( k ) ) =

Original: Because inf x ∈ R d  F  ( x  ) >
Revised: m ∗ ∈ N d and F ( x ( k ) ) ≤ F m ∗ ( z ( k ) ) for all k ≥ k . Proof of Lemma B.5 . According to Theorem 3.3, when P ( x ( k +1) ) ̸

Original:  −∞ , we can only have finite number of k ’s such thatF ( x ( k +1) )
Revised: = P ( x ( k ) ) , F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ . Because inf x ∈ R d F ( x ) > −∞ , we can only have finite number of k ’s such that

Original:  ≤ F ( x ( k ) ) − κ .
Revised: F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ .

Original: As a result, there must exists a finite k 0 ∈ N such that P ( x ( k ) )
Revised: As a result, there must exists a finite k 1 ∈ N such that P ( x ( k ) )

Original: =
Revised: =

Original: m ∗ ∈ N d for all k ≥ k 0 . 
Revised: m ∗ ∈ N d for all k ≥ k 1 . We now prove that there exists a finite k 2 > k 1 such that F ( x ( k ) ) ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 . Suppose this is not the case and there are infinitely many k ’s such that F ( x ( k ) )

Original:  It follows that  x ∗ i ∈ R m ∗ i .
Revised: > F m ∗ ( z ( k ) ) and k ≥ k 1 . It follows that there exists a sequence { m k } k ≥ 1 such that F ( x ( m k ) ) > F m ∗ ( z ( m k ) ) with m k > k 1 for all k ≥ 1 , and lim k →∞ m k = ∞ .

Revised: According to the PPGD algorithm and the Negative-CurvatureExploitation algorithm described in Algorithm 1 and Algorithm 2, this is possible only if for all k ≥ 1 , there exists

Revised: i ∈

Revised: [ d ] such that P ( z ( m k ) i

Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: ( m k ) i − q (

Revised: [Equation]

Revised: m k

Revised: [Equation]

Revised: 1) i ) (cid:12)(cid:12)(cid:12) <

Revised: w 0 (cid:12)(cid:12)(cid:12) z ( m k ) i − w ( m k − 1) i (cid:12)(cid:12)(cid:12) , x ( m k ) =

Revised: x ( m k − 1) , and f is continuous at q ( w ( m k − 1) i ) .

Revised: We consider the case that q ( w ( m k − 1) i ) =

Revised: q m ∗ i is the right endpoint of R m ∗ i .

Revised: The case that q ( w ( m k − 1) i ) =

Revised: q m ∗ i − 1 is the left endpoint of R m ∗ i can be proved in a similar manner.

Revised: In this case, P ( z ( m k ) i )

Revised: = m

Revised: ∗ i + 1 .

Revised: Let ˜ w ∈ R d with ˜ w j = w ( m k − 1) j for all j ̸

Revised: = i , and ˜ w i = q m ∗ i .

Revised: By the definition of ˜ w and the same argument as (30) in the proof of Lemma B.2, we have z ( m k

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: Because (cid:13)(cid:13) ∇ g ( w ( m k − 1) )

Revised: − ∇ g ( ˜ w ) (cid:13)(cid:13) 2 ≤ sL g (1 − w 0 )

Revised: ( G + F 0 ) and s <

Original: {+ε 
+}
Revised: {+ε 
+}

Revised: L g (1 − w 0 )( G + F 0 ) , we must have [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 due to Assumption 2 and the fact that z ( m k ) i − w

Revised: [Equation]

Revised: 1) i > 0 .

Revised: With k → ∞ , we have t mk − 1 t mk → 1 and

Revised: [Equation]

Revised: 51) and it follows that w ( m k )

Revised: [Equation]

Revised: P

Revised: [Equation]

Revised: ,R 0 ( z ( m k ) ) , and we have (cid:104)

Revised: P x ( mk ) ,R 0 ( z ( m k ) )

Revised: (cid:105) i = (cid:104) P x ( mk − 1) ,R 0 ( z ( m k ) )

Revised: (cid:105) i =

Revised: q m ∗ i .

Revised: By the updating rule (9) in the PPGD algorithm and the first equality in (50), we have

Revised: [Equation]

Revised: It follows from (52) that

Revised: [Equation]

Revised: [Equation]

Revised: Here 1 ⃝ follows from (cid:104) P x ( mk ) ,R 0 ( z ( m k ) )

Revised: (cid:105) i = ˜

Revised: w i = q m ∗ i , w ( m k − 1) =

Revised: P x ( mk ) ,R 0 ( w ( m k − 1) ) , and x ( m k )

Revised: =

Revised: x ( m k − 1) .

Revised: 2 ⃝ follows from the contraction property of projection onto a closed convex set.

Revised: Combining (53) and the fact that [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 , we have

Revised: [Equation]

Revised: due to s <

Revised: [Equation]

Revised: Now noting that w ( m k ) =

Revised: P x ( mk ) ,R 0 ( u ( m k ) )

Revised: k →∞ → P x ( mk ) ,R 0 ( z ( m k ) ) , it follows from (54) and the smoothness of ∇ g that there exists a large enough k ′ such that when k ≥ k ′ ,

Revised: [Equation]

Revised: We now analyze the next iterate z ( m k +1) i .

Revised: By the updating rule z ( m k +1) i

Revised: = prox sf P ( x ( mk ) i )

Revised: (cid:0)(cid:2) w ( m k ) − s ∇ g ( w ( m k ) ) (cid:3) i (cid:1) , we must have

Revised: [Equation]

Revised: In the iteration m k + 1 , we have

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: and w ( m k ) i

Revised: k →∞ → q ( w ( m k ) i ) .

Revised: Therefore, with sufficiently large k ′ , we have d i, 1 ≥ w 0 d

Revised: i

Revised: 0 due to d i, 1 k →∞ → d i, 0 .

Revised: It follows from the Negative-Curvature-Exploitation algorithm described in Algorithm 2 that P ( x ( m k +1) i )

Revised: = m

Revised: ∗ i + 1 , which contradict the fact that P ( x ( m k ) )

Revised: =

Revised: m ∗ ∈ N d for all k ≥ k 1 .

Revised: This contradiction shows that there exists a finite k 2 > k 1 such that F ( x ( k ) ) ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 .

Revised: Setting k = k 2 completes the proof.

Revised: Proof of Theorem 3.4 .

Revised: According to Lemma B.5, there exists a finite k 1 > 1 such that P ( x ( k ) )

Revised: =

Revised: m ∗ ∈ N d and F ( x ( k ) )

Revised: ≤ F m ∗ ( z ( k ) ) for all k ≥ k 1 .

Revised: Furthermore, the proof of Lemma 3.2 shows that the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by PPGD satisfies (cid:8) x ( k ) (cid:9) k ≥ 1 ⊆

Revised: L which is a compact set, so there exists at least one limit point for (cid:8) x

Revised: ( k ) (cid:9) , and Ω ̸ = ∅ .

Original: Define h m ( x ) :=
Revised: Define h m ( x ) :=

Original: (cid:80) di
Revised: (cid:80) di

Original: =1 f m i ( x i ) for m ∈ N d and m i ∈
Revised: =1 f m i ( x i ) for m ∈ N d and m i ∈

Original: [ M ] for all i ∈ [ d ] , and F m := g + h m .
Revised: [ M ] for all i ∈ [ d ] , and F m := g + h m .

Original: Note that for all i ∈ [ d ] , f m ∗ i is convex except for the third case in (5) or (6).
Revised: Note that for all i ∈ [ d ] , f m ∗ i is convex except for the third case in (5) or (6).

Original: In such a case, either event 1 ⃝ : f m ∗ i ( x ) =
Revised: In such a case, either event 1 ⃝ : f m ∗ i ( x ) =

Original: lim y
Revised: lim y

Original: → q + m ∗ i f ( y ) for x > q m ∗ i and f ( q m ∗ i ) <
Revised: → q + m ∗ i f ( y ) for x > q m ∗ i and f ( q m ∗ i ) <

Original: lim y
Revised: lim y

Original: → q + m ∗ i f ( y ) , or event 2 ⃝ : f m ∗ i ( x ) =
Revised: → q + m ∗ i f ( y ) , or event 2 ⃝ : f m ∗ i ( x ) =

Original: lim y
Revised: lim y

Original: → q − m ∗ i − 1 f ( y ) for x < q m ∗ i − 1 and f ( q m ∗ i − 1 ) <
Revised: → q − m ∗ i − 1 f ( y ) for x < q m ∗ i − 1 and f ( q m ∗ i − 1 ) <

Original: lim y
Revised: lim y

Original: → q − m ∗ i − 1 f ( y ) .
Revised: → q − m ∗ i − 1 f ( y ) .

Original: It follows by the proof of Lemma 3.1 that all the sequences (cid:110) x ( k ) i (cid:111) k ≥ k 0 and (cid:110)
Revised: It follows by the proof of Lemma B.4 that all the sequences (cid:110) x ( k ) i (cid:111) k ≥ k 0 and (cid:110)

Original: z ( k ) i (cid:111)
Revised: z ( k ) i (cid:111)

Original: k ≥ k 

Revised: k ≥ k

Original: +1 satisfies (cid:110) x ( k ) i (cid:111)
Revised: +1 satisfies (cid:110) x ( k ) i (cid:111)

Original: k ≥ k 0 ⊆
Revised: k ≥ k 0 ⊆

Original: ( −∞ , q m ∗ i ] and (cid:110)
Revised: ( −∞ , q m ∗ i ] and (cid:110)

Original: z ( k ) i (cid:111)
Revised: z ( k ) i (cid:111)

Original: k ≥ k 0 +1 ⊆
Revised: k ≥ k 0 +1 ⊆

Original: ( −∞ , q m ∗ i ] if event 1 ⃝ happens, and (cid:110) x
Revised: ( −∞ , q m ∗ i ] if event 1 ⃝ happens, and (cid:110) x

Original: ( k ) i (cid:111)
Revised: ( k ) i (cid:111)

Original: k ≥ k 0 ⊆
Revised: k ≥ k 0 ⊆

Original: [ q m ∗ i − 1 , + ∞ ) and (cid:110)
Revised: [ q m ∗ i − 1 , + ∞ ) and (cid:110)

Original: z ( k ) i (cid:111)
Revised: z ( k ) i (cid:111)

Original: k ≥ k 0 +1 ⊆
Revised: k ≥ k 0 +1 ⊆

Original: [ q m ∗ i − 1 , + ∞ ) if event 2 ⃝ happens.
Revised: [ q m ∗ i − 1 , + ∞ ) if event 2 ⃝ happens.

Original: For all i ∈ [ d ] , define R i as the region over which f m ∗ i is convex.
Revised: For all i ∈ [ d ] , define R i as the region over which f m ∗ i is convex.

Original: It is clear that R i
Revised: It is clear that R i

Original: =
Revised: =

Original: R if event 1 ⃝ and 2 ⃝ do not happen for f m ∗ i .
Revised: R if event 1 ⃝ and 2 ⃝ do not happen for f m ∗ i .

Original: If only event 1 ⃝ happens for f m ∗ i , then R i =
Revised: If only event 1 ⃝ happens for f m ∗ i , then R i =

Original: ( −∞ , q m ∗ i ] .
Revised: ( −∞ , q m ∗ i ] .

Original: If only event 2 ⃝ happens, R i =
Revised: If only event 2 ⃝ happens, R i =

Original: [ q m ∗ i − 1 , + ∞ ) .
Revised: [ q m ∗ i − 1 , + ∞ ) .

Original: If both event 1 ⃝ and 2 ⃝ happen, then R i =
Revised: If both event 1 ⃝ and 2 ⃝ happen, then R i =

Original: [ q m ∗ i − 1 , q m ∗ i ] .
Revised: [ q m ∗ i − 1 , q m ∗ i ] .

Original: As a result, we have
Revised: Let ¯ x ∈ R d be an optimal solution to [Equation] Note that the existence of ¯ x is guaranteed by the fact that F m ∗ is convex on a convex set (cid:8) x

Original: [Equation]
Revised: [Equation]

Revised: We now consider k ≥ k 1 in the sequel.

Original:  for all i ∈ [ d ] , v ∈ R i , and all p ∈ ˜
Revised: We have [Equation] for all i ∈ [ d ] , v ∈ R i , and all p ∈ ˜

Original: ∂f ( z ( k +1) i ) .
Revised: ∂f ( z ( k +1) i ) .

Original: It follows that if v ∈ R d and v i ∈ R i for all
Revised: It follows that if v ∈ R d and v i ∈ R i for all

Original: i ∈
Revised: i ∈

Original: [ d ] , then
Revised: [ d ] , then

Original: [Equation] 
Revised: [Equation] [Equation]

Original: ) for all p ∈ ˜
Revised: for all p ∈ ˜

Original: ∂h m ∗ i .
Revised: ∂h m ∗ i .

Original: Because z ( k +1)
Revised: Because z ( k +1)

Original: i = proxsf P ( x ( k ) i ) (cid:0)(cid:2) w ( k ) − s ∇ g [Equation] w
Revised: i = prox

Original: [Equation]
Revised: [Equation]

Original: k ) [Equation] i (cid:1) in (9) of Algorithm 2, it follows by the optimality of z ( k +1) i that
Revised: in (9) of Algorithm 1, it follows by the optimality of z ( k +1) i that

Original: [Equation]
Revised: [Equation]

Original: It follows by (47) and (48) that
Revised: It follows by (58) and (59) that

Original: [Equation]
Revised: [Equation]

Original: for any v ∈ R d such that v i ∈ R i for all i ∈ [ d ] .
Revised: for any v ∈ R d such that v i ∈ R i for all i ∈ [ d ] .

Original: For such v
Revised: For such v

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Here 1 ⃝ follows from (49).
Revised: Here 1 ⃝ follows from (60).

Original: Let v = x ( k ) and v =  x∗ in (50), we haveF
Revised: Let v = x ( k ) and v = ¯ x in (61), we have

Original: [Equation]
Revised: [Equation]

Original: (51) andF m
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (52) (51) ×( t k − 1)+ (52), we have t k F m ∗ ( z ( k +1) ) −
Revised: 62) ×

Original: [Equation]
Revised: [Equation]

Original: t
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 1) F m
Revised: 1)+

Original: [Equation]
Revised: [Equation]

Original: x [Equation] [Equation] F m
Revised: 63), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Multiplying both sides of (54) by t k , since t 2 k − t k
Revised: Multiplying both sides of (65) by t k , since t 2 k − t k

Original: = t 2 k − 
Revised: = t 2 k − 1 , we have

Original: [Equation]
Revised: [Equation]

Original:  we have [Equation] [Equation] Note that w ( k ) = 
Revised: Since t k ≥ k +12 for k ≥ 1 , we have t k k →∞ → = ∞ and lim k →∞ (cid:16) 1 − 1 t k (cid:17) x

Original: P x ( k ) ( u ( k ) ) by (8) in Algorithm 2. Because t k > 1 for all k > 1 , (1 − 1 /t k ) x ( k ) + 1 /t k x ∗ is a convex combination of x ( k ) and x ∗ with P ( x ( k ) )
Revised: ( k ) + 1 t

Original:  =P ( x ∗ ) 
Revised: k ¯ x = x ( k ) .

Original: = m ∗ , it follows that P ((1 − 1 /t k ) x ( k ) + 1 /t k x ∗ ) = m ∗ We have
Revised: It follows that there exists a finite k 2 such that

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: for all k ≥ k 2 and all i ∈ [ d ] .

Original: [Equation] [Table] [Equation] [Table]
Revised: It follows from (67) that

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Now let k ≥ k 0 :

Original: [Table]
Revised: = max { k 1 , k 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: where the second last inequality follows from the contraction property of projection onto  closed convex sets in Hilbert spaces
Revised: where 1 ⃝ follows from (68), 2 ⃝ follows from the contraction property of projection onto a closed convex set.

Original: [Equation] It follows by (55) and (56) [Equation] that
Revised: It follows by (66) and (69), that

Original: [Equation]
Revised: [Equation]

Original: Define Q ( k +1)
Revised: Define Q ( k +1)

Original: =
Revised: =

Original: ( t k − 1) x ( k ) − t k z
Revised: ( t k − 1) x ( k ) − t k z

Original: ( k +1) 
Revised: ( k +1) +

Original: + x∗ , then Q ( k ) =
Revised: ¯ x , then Q ( k ) =

Original: ( t k − 1 − 1) x ( k − 1) − t k − 1 z
Revised: ( t k − 1 − 1) x ( k − 1) − t k − 1 z

Original: ( k ) +  x∗ .
Revised: ( k ) + ¯ x .

Original: It can be verified that Q ( k ) =
Revised: It can be verified that Q ( k ) =

Original: ( t k − 1) x ( k ) − t k u ( k ) +  x∗ .
Revised: ( t k − 1) x ( k ) − t k u ( k ) + ¯ x .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Revised: Noting that F ( x ( k +1) ) ≤ F m ∗ ( z ( k +1) ) for k ≥ k 0 , it follows from the above inequality that

Original:  Summing (58) over k = k 0 , . ..
Revised: [Equation] Summing (71) over k = k 0 , . .

Original: , m for m ≥ k 0 , we have t
Revised: , m for m ≥ k 0 , we have t

Original: [Equation]
Revised: [Equation]

Revised: Since t k ≥ k +12 for k ≥ 1 , it follows from (72) that

Revised: [Equation]

Revised: Noting that F ( x ( m +1) )

Revised: =

Revised: F m ∗ ( x ( m +1) ) , we have F ( x ( m +1) )

Revised: − F m ∗ (¯ x ) ≤

Revised: [Equation]

Revised: 2 U

Revised: [Equation]

Original: m
Revised: k

Revised: [Equation]

Revised: Replacing m + 1 with k , we have

Revised: [Equation]

Revised: for all k > k 0 .

Revised: Let x ′ ∈

Revised: Ω be an arbitrary limit point of (cid:8) x

Revised: ( k ) (cid:9) k ≥ 1 .

Revised: By Lemma 3.2, we have F ( x ( k ) ) ↓ F ( x ′ ) as k → ∞ .

Revised: In addition, (74) indicates that F ( x ( k ) ) ↓ F m ∗ (¯ x ) as k → ∞ .

Revised: As a result,

Revised: [Equation]

Revised: for any x ′ ∈ Ω .

Revised: That is, F has constant value on Ω .

Revised: It is noted that F m ∗ =

Revised: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) , so

Revised: [Equation]

Revised: On the other hand, since P (¯ x ) = m ∗ , we have F ( x ′ ) ≥ inf { x ∈ R d | P ( x )

Revised: = m ∗ } F ( x ) .

Revised: Combining this inequality and (76), we have F ( x ′ )

Revised: =

Revised: inf { x ∈ R d

Revised: | P ( x )= m ∗ } F ( x ) .

Revised: Now we consider the case that f m ∗ i does not take the third case in (5) or (6) for all i ∈ [ d ] .

Revised: As explained in the beginning of this proof, R i

Revised: =

Revised: R for all i ∈ [ d ] .

Revised: In this case, because F ( x ′ ) =

Revised: F m ∗ (¯ x ) for any x ′ ∈ Ω , x ′ is an optimal solution to min x ∈ R d F m ∗ ( x ) , and F m ∗ is convex over R d .

Revised: The optimality of x ′ for this convex programming problem indicates that 0 ∈ ˜ ∂F m ∗ ( x ′ ) .

Revised: Because F m ∗ =

Revised: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) and x ′ ∈ (cid:8) x ∈ R d

Revised: (cid:12)(cid:12) P ( x )

Revised: = m ∗ (cid:9) , we have 0 ∈ ∂F ( x ′ ) .

Revised: This can be verified using the definition of limitingsubdifferential by considering a constant sequence x k = x ′ for all k ≥ 1 .

Revised: We have with 0 ∈ ˜ ∂F ( x k ) because 0 ∈ ˜ ∂F m ∗ ( x ′ ) and P ( x ′ )

Revised: = m

Revised: ∗ .

Revised: C I NEXACT PPGD

Revised: We propose Inexact PPGD where the proximal mapping (9) in PPGD described in Algorithm 1 is inexact, that is,

Original: [Equation]
Revised: [Equation]

Revised: where e ( k ) represents the inexactness of computation of ∇ g ( w ( k ) ) , and ε k captures the inexactness of evaluation of the proximal map given by

Original: Since t k ≥ k +12 for k ≥
Revised: [Equation] where u = w ( k ) − s (cid:0) ∇ g ( w ( k [Equation] + e ( k ) (cid:1) .

Revised: It is noted that prox ε,Rsf m in (78) can be computed by projected subgradient descent, that is, every iterate by subgradient descent is projected onto [ w ( k ) i −

Revised: R, w ( k ) i + R ] , and we set R = s ( G + F 0 ) .

Revised: Theorem C.1.

Revised: Suppose the proximal mapping (9) in PPGD is replaced by its inexact version (77) with R =

Revised: s ( G + F 0 ) .

Original: {+Suppose s < min (cid:110) s 1 , ε 
+}
Revised: {+Suppose s < min (cid:110) s 1 , ε 
+}

Revised: L g ( G + √ dF 0 ) , 1 L g (cid:111) with s 1 defined by (16), Assumption 1 and Assumption 2 hold.

Revised: Then there exists a finite k 0 ≥ 1 such that all the statements of Theorem 3.4 remain true except for the convergence rate in (18).

Revised: We have

Original: [Equation]
Revised: [Equation]

Original: it follows from (59) that{+where A k 0 ,k := (cid:80) ki = k 
 i (cid:0) s (cid:13)(cid:13)+}
Revised: {+where A k 0 ,k := (cid:80) ki = k 
 i (cid:0) s (cid:13)(cid:13)+}

Revised: e ( i ) (cid:13)(cid:13) 2 + √ 2 sε i (cid:1) , B k 0 ,k := (cid:80) ki

Revised: = k 0 si 2 ε i , and a = O ( b ) means that a ≤ Cb for some constant C > 0 .

Revised: Sketch of the Proof.

Revised: With R =

Revised: s ( G + F 0 ) , we have (cid:12)(cid:12)(cid:12) z

Revised: ( k +1)

Revised: i − w ( k ) i (cid:12)(cid:12)(cid:12) ≤ R =

Revised: s ( G + F 0 ) .

Revised: Note that the results of Lemma 3.1, Lemma 3.2, and Lemma B.5 essentially depend on the bound on (cid:12)(cid:12)(cid:12)

Revised: z ( k +1)

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: With the s in the given condition and the bound R = s ( G + F 0 ) for (cid:12)(cid:12) (cid:12) z ( k +1)

Revised: i − w ( k ) i (cid:12)(cid:12) (cid:12) , one can repeat the arguments in the proof of Lemma 3.1, Lemma 3.2, Lemma B.5, and Theorem 3.3 to prove that there exists a finite k 0 such that it can be verified that there exists a finite k 0 ≥ 1 such that all P ( x ( k ) )

Revised: = m ∗ for some m ∗ ∈ N d for all k > k 0 .

Revised: By (Schmidt et al., 2011, Lemma 2),

Revised: [Equation]

Revised: where v ( k ) ∈

Revised: R d with (cid:13)(cid:13) v ( k ) (cid:13)(cid:13) 2 ≤ √ 2 sε k .

Revised: ˜ ∂ ε denotes the ε -subdifferential.

Revised: For a convex function q : R d → R , its ε -subdifferential at a point x ∈ R d is defined by ˜ ∂ ε q ( x ) := (cid:8) u (cid:12)(cid:12) q ( y ) ≥ q ( x ) + ⟨ u , y − x ⟩ − ε, ∀ y ∈ R d (cid:9) .

Revised: Using (80) and the argument in the proof of Theorem 3.4, the convergence rate (79) can be proved.

Revised: Similar to (Schmidt et al., 2011), the RHS of (79) matches the rate of O ( 1 k 2 ) with sufficiently small (cid:8) e ( k ) (cid:9) and ε k .