Original: P ROJECTIVE P ROXIMAL G RADIENT D ESCENT FOR A C LASS OF N ONCONVEX N ONSMOOTH O PTIMIZATION P ROBLEMS : F AST C ONVERGENCE W ITHOUT K URDYKA - Ł OJASIEWICZ (KŁ) P ROPERTY
Revised: P ROJECTIVE P ROXIMAL G RADIENT D ESCENT FOR A C LASS OF N ONCONVEX N ONSMOOTH O PTIMIZATION P ROBLEMS : F AST C ONVERGENCE W ITHOUT K URDYKA -Ł OJASIEWICZ (KŁ) P ROPERTY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Nonconvex and nonsmooth optimization problems are important and challenging for statistics and machine learning.
Revised: Nonconvex and nonsmooth optimization problems are important and challenging for statistics and machine learning.

Original: In this paper, we propose Projected Proximal Gradient Descent (PPGD) which solves a class of nonconvex and nonsmooth optimization problems, where the nonconvexity and nonsmoothness come from a nonsmooth regularization term which is nonconvex but piecewise convex.
Revised: In this paper, we propose Projected Proximal Gradient Descent (PPGD) which solves a class of nonconvex and nonsmooth optimization problems, where the nonconvexity and nonsmoothness come from a nonsmooth regularization term which is nonconvex but piecewise convex.

Original: In contrast with existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems based on the Kurdyka-Łojasiewicz (KŁ) property, we provide a new theoretical analysis showing local fast convergence of PPGD.
Revised: In contrast with existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems based on the Kurdyka-Łojasiewicz (KŁ) property, we provide a new theoretical analysis showing local fast convergence of PPGD.

Original: It is proved that PPGD achieves a fast convergence rate of O (1 /k 2 ) when the iteration number k ≥ k 0 for a finite k 0 on a class of nonconvex and nonsmooth problems under mild assumptions, which is locally the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with
Revised: It is proved that PPGD achieves a fast convergence rate of O (1 /k 2 ) when the iteration number k ≥ k 0 for a finite k 0 on a class of nonconvex and nonsmooth problems under mild assumptions, which is locally the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function with

Original: Lipschitz continuous gradient.
Revised: Lipschitz continuous gradient.

Original: Experimental results demonstrate the effectiveness of PPGD.
Revised: Experimental results demonstrate the effectiveness of PPGD.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Nonconvex and nonsmooth optimization problems are challenging ones which have received a lot of attention in statistics and machine learning (Bolte et al., 2014; Ochs et al., 2015).
Revised: Nonconvex and nonsmooth optimization problems are challenging ones which have received a lot of attention in statistics and machine learning (Bolte et al., 2014; Ochs et al., 2015).

Original: In this paper, we consider fast optimization algorithms for a class of nonconvex and nonsmooth problems presented as
Revised: In this paper, we consider fast optimization algorithms for a class of nonconvex and nonsmooth problems presented as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where g is convex, h ( x )
Revised: where g is convex, h ( x )

Original: =
Revised: =

Original: d (cid:80) j
Revised: d (cid:80) j

Original: =1 h j ( x j ) is a separable regularizer, each h j is piecewise convex.
Revised: =1 h j ( x j ) is a separable regularizer, each h j is piecewise convex.

Original: A piecewise convex function is defined in Definition 1.1.
Revised: A piecewise convex function is defined in Definition 1.1.

Original: For simplicity of analysis we let h j = f for all j ∈ [ d ] , and f is a piecewise convex function.
Revised: For simplicity of analysis we let h j = f for all j ∈ [ d ] , and f is a piecewise convex function.

Original: Here [ d ] is the set of natural numbers between 1 and n inclusively.
Revised: Here [ d ] is the set of natural numbers between 1 and n inclusively.

Original: f can be either nonconvex or convex, and all the results in this paper can be straightforwardly extended to the case when { h j } are different.
Revised: f can be either nonconvex or convex, and all the results in this paper can be straightforwardly extended to the case when { h j } are different.

Original: Definition 1.1.
Revised: Definition 1.1.

Original: A univariate function f : R → R is piecewise convex if f is lower semicontinuous and there exist intervals {R m } Mm =1 such that R =
Revised: A univariate function f : R → R is piecewise convex if f is lower semicontinuous and there exist intervals {R m } Mm =1 such that R =

Original: (cid:83) Mm =1 R m , and f restricted on R m is convex for each m ∈
Revised: (cid:83) Mm =1 R m , and f restricted on R m is convex for each m ∈

Original: [ M ] .
Revised: [ M ] .

Original: The left and right endpoints of R m are denoted by q m − 1 and q m for all m ∈
Revised: The left and right endpoints of R m are denoted by q m − 1 and q m for all m ∈

Original: [ M ] , where { q m } Mm =0 are the endpoints such that q 0 =
Revised: [ M ] , where { q m } M m =0 are the endpoints such that q 0 =

Original: −∞ ≤ q 1 < q 2 < . . .
Revised: −∞ ≤ q 1 < q 2 < . . .

Original: <
Revised: <

Original: q M = + ∞ .
Revised: q M = + ∞ .

Original: Furthermore, f is either left continuous or right continuous at each endpoint q m for m ∈ [ M − 1] .
Revised: Furthermore, f is either left continuous or right continuous at each endpoint q m for m ∈ [ M − 1] .

Original: {R m } M m =1 are also referred to as convex pieces throughout this paper.
Revised: {R m } Mm =1 are also referred to as convex pieces throughout this paper.

Original: It is important to note that for all m ∈
Revised: It is important to note that for all m ∈

Original: [ M − 1] , when f is continuous at the endpoint q m or f is only left continuous at q m , q m ∈ R m and q m / ∈ R m +1 .
Revised: [ M − 1] , when f is continuous at the endpoint q m or f is only left continuous at q m , q m ∈ R m and q m / ∈ R m +1 .

Original: If f is only right continuous at q m , q m / ∈ R m and q m ∈ R m +1 .
Revised: If f is only right continuous at q m , q m / ∈ R m and q m ∈ R m +1 .

Original: This ensures that any point in R lies in only one convex piece.
Revised: This ensures that any point in R lies in only one convex piece.

Original: When M = 1 , f becomes a convex function on R , and problem (1) is a convex problem.
Revised: When M = 1 , f becomes a convex function on R , and problem (1) is a convex problem.

Original: We consider M > 1 throughout this paper, and our proposed algorithm trivially extends to the case when M = 1 .
Revised: We consider M > 1 throughout this paper, and our proposed algorithm trivially extends to the case when M = 1 .

Original: We allow a special case that an interval R m = { q m } for m ∈ [ M − 1] is a singlepoint set, in this case q m − 1 = q m .
Revised: We allow a special case that an interval R m = { q m } for m ∈ [ M − 1] is a singlepoint set, in this case q m − 1 = q m .

Original: When there are no single-point intervals in {R m } Mm =1 , the minimum interval length is defined by R 0 = min m ∈
Revised: When there are no single-point intervals in {R m } Mm =1 , the minimum interval length is defined by R 0 = min m ∈

Original: [ M ] |R m | .
Revised: [ M ] |R m | .

Original: Otherwise,the minimum interval length is R 0 = min m ∈
Revised: Otherwise,the minimum interval length is R 0 = min m ∈

Original: [ M ]: |R m |̸ =

Revised: [ M ]: |R m |̸ =

Original: |R m | , where |·| denotes the length of an interval.
Revised: |R m | , where |·| denotes the length of an interval.

Original: It is worthwhile to emphasize that the nonconvexity and nonsmoothness of many popular optimization problems come from piecewise convex regularizers.
Revised: It is worthwhile to emphasize that the nonconvexity and nonsmoothness of many popular optimization problems come from piecewise convex regularizers.

Original: Below are three examples of piecewise convex functions with the corresponding regularizers which have been widely used in constrained optimization and sparse learning problems.
Revised: Below are three examples of piecewise convex functions with the corresponding regularizers which have been widely used in constrained optimization and sparse learning problems.

Original: [Figure]
Revised: [Figure]

Original: Example 1.1.
Revised: Example 1.1.

Original: The indicator penalty function f ( x ) =
Revised: The indicator penalty function f ( x ) =

Original: λ 1I { x<τ } is piecewise convex with R 1 = ( −∞ , τ ) , R 2 =
Revised: λ 1I { x<τ } is piecewise convex with R 1 = ( −∞ , τ ) , R 2 =

Original: [ τ, ∞ ) .
Revised: [ τ, ∞ ) .

Original: The capped- ℓ 1 penalty function f ( x ) =
Revised: The capped- ℓ 1 penalty function f ( x ) =

Original: f
Revised: f

Original: ( x ; λ, b ) =
Revised: ( x ; λ, b ) =

Original: λ min {| x | , b } is piecewise convex with R 1 = ( −∞ , − b ] , R 2 =
Revised: λ min {| x | , b } is piecewise convex with R 1 = ( −∞ , − b ] , R 2 =

Original: [ − b, b ] , R 3 =
Revised: [ − b, b ] , R 3 =

Original: [ b, ∞ ) .
Revised: [ b, ∞ ) .

Original: The leaky cappedℓ 1 penalty function (Wangni & Lin, 2017) f
Revised: The leaky cappedℓ 1 penalty function (Wangni & Lin, 2017) f

Original: = λ min {| x | , b } + β 1I {| x |≥ b } | x − b | with R 1 = ( −∞ , − b ] , R 2 =
Revised: = λ min {| x | , b } + β 1I {| x |≥ b } | x − b | with R 1 = ( −∞ , − b ] , R 2 =

Original: [ − b, b ] , R 3 =
Revised: [ − b, b ] , R 3 =

Original: [ b, ∞ ) .
Revised: [ b, ∞ ) .

Original: The three functions are illustrated in Figure 1.
Revised: The three functions are illustrated in Figure 1.

Original: While not illustrated, f ( x ) = 1I { x ̸ =0 } for the ℓ 0 -norm with h ( x ) =
Revised: While not illustrated, f ( x ) = 1I { x ̸ =0 } for the ℓ 0 -norm with h ( x ) =

Original: ∥ x ∥ 0 is also piecewise convex.
Revised: ∥ x ∥ 0 is also piecewise convex.

Original: [Figure] A SSUMPTION
Revised: 1. M AIN A SSUMPTION

Original: The main assumption of this paper is that g and h satisfy the following conditions.
Revised: The main assumption of this paper is that g and h satisfy the following conditions.

Original: Assumption 1 (Main Assumption) .
Revised: Assumption 1 (Main Assumption) .

Original: (a) g 
Revised: a) g is

Revised: convex with

Revised: L -smooth

Original: is convex with L -smooth gradient, that is, ∥∇ g ( x ) − ∇ g ( y ) ∥ 2 ≤ L g ∥ x − y ∥ 2 .
Revised: gradient that is, ∥∇ g ( x ) − ∇ g ( y ) ∥ 2 ≤ L g ∥ x − y ∥ 2 .

Original: F is coercive, that is, F ( x ) → ∞ when ∥ x ∥ 2 → ∞ , and inf x ∈ R d F ( x ) >
Revised: F is coercive, that is, F ( x ) → ∞ when ∥ x ∥ 2 → ∞ , and inf x ∈ R d F ( x ) >

Original: −∞ .
Revised: −∞ .

Original: (b) f : R → R is a piecewise convex function and lower semicontinuous.
Revised: (b) f : R → R is a piecewise convex function and lower semicontinuous.

Original: Furthermore, there exists a small positive constant s 0 < R 0 such that f is differentiable on ( q m − s 0 , q m ) and ( q m , q m + s 0 ) for all m ∈
Revised: Furthermore, there exists a small positive constant s 0 < R 0 such that f is differentiable on ( q m − s 0 , q m ) and ( q m , q m + s 0 ) for all m ∈

Original: [ M − 1] .
Revised: [ M − 1] .

Original: (c)
Revised: (c)

Original: The proximal mapping prox sf i has a closed-form solution for all m ∈[ − M, M ] , where prox
Revised: The proximal mapping prox sf m has a closed-form solution for all m ∈

Original: [Equation]
Revised: [ M ] , where prox sf m ( x ) := argmin v ∈ R 1 2 s ( v − x ) 2 + f m ( v ) .

Original: (d) f has “negative curvature” at each endpoint q m where f is continuous for all m ∈
Revised: (d) f has “negative curvature” at each endpoint q m where f is continuous for all m ∈

Original: [ M − 1] , that is, lim x → q − m f ′
Revised: [ M − 1] , that is, lim x → q − m f ′

Original: ( x ) >
Revised: ( x ) >

Original: lim x → q + m f ′ ( x ) .
Revised: lim x → q + m f ′ ( x ) .

Original: We define
Revised: We define

Original: [Equation]
Revised: [Equation]

Original: In addition, f has bounded Fr´echet subdifferential, that is, sup x ∈
Revised: In addition, f has bounded Fr´echet subdifferential, that is, sup x ∈

Original: [Equation]
Revised: [Equation]

Original: Mm [Equation] R om sup v ∈ ˜ ∂f
Revised: Mm =1 R om sup v ∈ ˜ ∂f

Original: [Equation]

Original:  x ) ∥ v ∥ 2 ≤
Revised: ( x ) ∥ v ∥ 2 ≤

Original: F 0 for some absolute constant F 0 > 0 , where R o denotes the interior of an interval.
Revised: F 0 for some absolute constant F 0 > 0 , where R o denotes the interior of an interval.

Original: Fr´echet subdifferential is formally defined in Section 4.
Revised: Fr´echet subdifferential is formally defined in Section 4.

Original: It is noted that on each R om , the convex differential of f coincides with its Fr´echet subdifferential.
Revised: It is noted that on each R om , the convex differential of f coincides with its Fr ´ echet subdifferential.

Original: We define the minimum jump value of f at noncontinuous endpoints by
Revised: We define the minimum jump value of f at noncontinuous endpoints by

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Assumption 1 is mild and explained as follows.
Revised: Assumption 1 is mild and explained as follows.

Original: Smooth gradient in Assumption 1(a) is a commonly required in the standard analysis of proximal methods for non-convex problems, such as (Bolte et al., 2014; Ghadimi & Lan, 2016a).
Revised: Smooth gradient in Assumption 1(a) is a commonly required in the standard analysis of proximal methods for non-convex problems, such as (Bolte et al., 2014; Ghadimi & Lan, 2016).

Original: The objective F is widely assumed to be coercive in the nonconvex and nonsmooth optimization literature, such as (Li & Lin, 2015; Li et al., 2017).
Revised: The objective F is widely assumed to be coercive in the nonconvex and nonsmooth optimization literature, such as (Li & Lin, 2015; Li et al., 2017).

Original: In addition, Assumption 1(b)-(d) are mild, and they hold for all the three piecewise convex functions in Example 1.1 as well as the ℓ 0 -norm.
Revised: In addition, Assumption 1(b)-(d) are mild, and they hold for all the three piecewise convex functions in Example 1.1 as well as the ℓ 0 -norm.

Original: It is noted that (1) covers a broad range of optimization problems in machine learning and statistics.
Revised: It is noted that (1) covers a broad range of optimization problems in machine learning and statistics.

Original: The nonnegative programming problem min x ∈ R d : x i ≥ 0 ,i ∈
Revised: The nonnegative programming problem min x ∈ R d : x i ≥ 0 ,i ∈

Original: [ d ] g ( x ) can be reduced to the regularized problem g ( x ) + h ( x ) with f being the indicator penalty f
Revised: [ d ] g ( x ) can be reduced to the regularized problem g ( x ) + h ( x ) with f being the indicator penalty f

Original: ( x ) = λ 1I { x<τ } for a properly large λ .
Revised: ( x ) = λ 1I { x<τ } for a properly large λ .

Original: When g is the squared loss, that is, g ( x ) =
Revised: When g is the squared loss, that is, g ( x ) =

Original: ∥
Revised: ∥

Original: y − Dx ∥ 22 where y ∈ R n and D ∈ R n × d is the design matrix, (1) is the well-known regularized sparse linear regression problems with convex or nonconvex regularizers such as the capped- ℓ 1 or ℓ 0 -norm penalty.
Revised: y − Dx ∥ 22 where y ∈ R n and D ∈ R n × d is the design matrix, (1) is the well-known regularized sparse linear regression problems with convex or nonconvex regularizers such as the capped- ℓ 1 or ℓ 0 -norm penalty.

Original: 1. M AIN R ESULTS AND C ONTRIBUTIONS
Revised: 1. M AIN R ESULTS AND C ONTRIBUTIONS

Original: We propose a novel method termed Projective Proximal Gradient Descent (PPGD), which extends the existing Accelerated Proximal Gradient descent method (APG) (Beck & Teboulle, 2009b; Nesterov, 2013) to solve problem (1).
Revised: We propose a novel method termed Projective Proximal Gradient Descent (PPGD), which extends the existing Accelerated Proximal Gradient descent method (APG) (Beck & Teboulle, 2009b; Nesterov, 2013) to solve problem (1).

Original: PPGD enjoys fast convergence rate by a novel projection operator and a new negative curvature exploitation procedure.
Revised: PPGD enjoys fast convergence rate by a novel projection operator and a new negative curvature exploitation procedure.

Original: Our main results are summarized below.
Revised: Our main results are summarized below.

Original: Using a novel and carefully designed projection operator and the Negative-Curvature-Exploitation algorithm (Algorithm 2), PPGD achieves a fast convergence rate for the nonconvex and nonsmooth optimization problem (1) which locally matches the Nesterov’s optimal convergence rate of firstorder methods on smooth and convex objective function with Lipschitz continuous gradient.
Revised: Using a novel and carefully designed projection operator and the Negative-Curvature-Exploitation algorithm (Algorithm 2), PPGD achieves a fast convergence rate for the nonconvex and nonsmooth optimization problem (1) which locally matches the Nesterov’s optimal convergence rate of firstorder methods on smooth and convex objective function with Lipschitz continuous gradient

Original: In particular, it is proved that under two mild assumptions, Assumption 1 and Assumption 2 to be introduced in Section 4, there exists a finite k 0 ≥ 1 such that for all k > 
Revised: In particular, it is proved that under two mild assumptions, Assumption 1 and Assumption 2 to be introduced in Section 4, there exists a finite k 0 ≥ 1 such that for all k > k 0 ,

Original: [Equation]
Revised: [Equation]

Original: where x ∗ is any limit point of (cid:8) x
Revised: where x ∗ is any limit point of (cid:8) x

Original: ( k ) (cid:9) k ≥ 1 lying on the same convex pieces as (cid:8) x
Revised: ( k ) (cid:9) k ≥ 1 lying on the same convex pieces as (cid:8) x

Original: ( k ) (cid:9) k>k 0 .
Revised: ( k ) (cid:9) k>k 0 .

Original: Details are deferred to Theorem 4.4 in Section 4.
Revised: Details are deferred to Theorem 4.4 in Section 4.

Original: It should be emphasized that this is the same convergence rate as that of regular APG on convex problems (Beck & Teboulle, 2009a;b).
Revised: It should be emphasized that this is the same convergence rate as that of regular APG on convex problems (Beck & Teboulle, 2009b;a).

Original: Our analysis provides insights into accelerated PGD methods for a class of challenging nonconvex and nonsmooth problems.
Revised: Our analysis provides insights into accelerated PGD methods for a class of challenging nonconvex and nonsmooth problems.

Original: In contrast to most existing accelerated PGD methods (Li & Lin, 2015;
Revised: In contrast to most existing accelerated PGD methods (Li & Lin, 2015;

Original: Li et al., 2017) which employ the Kurdyka-Łojasiewicz (KŁ) property to analyze the convergence rates, PPGD provides a new perspective of convergence analysis without the KŁ property while locally matching the Nesterov’s optimal convergence rate.
Revised: Li et al., 2017) which employ the Kurdyka-Łojasiewicz (KŁ) property to analyze the convergence rates, PPGD provides a new perspective of convergence analysis without the KŁ property while locally matching the Nesterov’s optimal convergence rate.

Original: Our analysis reveals that the objective function makes progress, that is, its value is decreased by a positive amount, when the iterate sequence generated by PPGD transits from one convex piece to another.
Revised: Our analysis reveals that the objective function makes progress, that is, its value is decreased by a positive amount, when the iterate sequence generated by PPGD transits from one convex piece to another.

Original: Such observation opens the door for future research in analyzing convergence rate of accelerated proximal gradient methods without the KŁ property.
Revised: Such observation opens the door for future research in analyzing convergence rate of accelerated proximal gradient methods without the KŁ property.

Original: It should be emphasized that the conditions in Assumption 1 can be relaxed while PPGD still enjoys the same order of convergence rate.
Revised: It should be emphasized that the conditions in Assumption 1 can be relaxed while PPGD still enjoys the same order of convergence rate.

Original: First, the assumption that f is piecewise convex can be relaxed to a weaker one to be detailed in Remark 4.6.
Revised: First, the assumption that f is piecewise convex can be relaxed to a weaker one to be detailed in Remark 4.6.

Original: Second, the proximal mapping in Assumption 1 does not need to have a closed-form solution.
Revised: Second, the proximal mapping in Assumption 1 does not need to have a closed-form solution.

Original: In Section C of the supplementary, the fast convergence rate of PPGD is proved with inexact proximal mapping.

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Assuming that both g and h satisfy the KŁ Property defined in Definition A.1 in Section A.1 of the supplementary, the accelerated PGD algorithms in (Li & Lin, 2015; Li et al., 2017) have linear convergence rate with θ ∈
Revised: Assuming that both g and h satisfy the KŁ Property defined in Definition A.1 in Section A.1 of the supplementary, the accelerated PGD algorithms in (Li & Lin, 2015; Li et al., 2017) have linear convergence rate with θ ∈

Original: [1 / 2 , 1) , and sublinear rate O ( k − 1 1 − 2 θ ) with θ ∈ (0 , 1 / 2) , where θ is the
Revised: [1 / 2 , 1) , and sublinear rate O ( k − 1 1 − 2 θ ) with θ ∈ (0 , 1 / 2) , where θ is the

Original: Kurdyka-Lojasiewicz (KŁ) exponent and both rates are for objective values.
Revised: Kurdyka-Lojasiewicz (KŁ) exponent and both rates are for objective values.

Original: To the best of our knowledge, most existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems, where the nonconvexity and nonsmoothness are due to the regularizer h , are based on the KŁ property.
Revised: To the best of our knowledge, most existing convergence analysis of accelerated PGD methods for nonconvex and nonsmooth problems, where the nonconvexity and nonsmoothness are due to the regularizer h , are based on the KŁ property.

Original: While (Ghadimi & Lan, 2016b) provides analysis of an accelerated PGD method for nonconvex and nonsmooth problems, the nonconvexity comes from the smooth part of the objective function ( g in our objective function), so it cannot handle the problems discussed in this paper.
Revised: While (Ghadimi & Lan, 2016) provides analysis of an accelerated PGD method for nonconvex and nonsmooth problems, the nonconvexity comes from the smooth part of the objective function ( g in our objective function), so it cannot handle the problems discussed in this paper.

Original: Furthermore, the fast PGD method by (Yang & Yu, 2020) is restricted to ℓ 0regularized problems.
Revised: Furthermore, the fast PGD method by (Yang & Yu, 2019) is restricted to ℓ 0regularized problems.

Original: As a result, it remains an interesting and important open problem regarding the convergence rate of accelerated PGD methods for problem (1).
Revised: As a result, it remains an interesting and important open problem regarding the convergence rate of accelerated PGD methods for problem (1).

Original: Another line of related works focuses on accelerated gradient methods for smooth objective functions.
Revised: Another line of related works focuses on accelerated gradient methods for smooth objective functions.

Original: For example, (Jin et al., 2018) proposes perturbed accelerated gradient method which achieves faster convergence by decreasing a quantity named Hamiltonian, which is the sum of the current objective value and the scaled squared distance between consecutive iterates, by exploiting the negative curvature.
Revised: For example, Jin et al. (2018) proposes perturbed accelerated gradient method which achieves faster convergence by decreasing a quantity named Hamiltonian, which is the sum of the current objective value and the scaled squared distance between consecutive iterates, by exploiting the negative curvature.

Original: The very recent work (Li & Lin, 2022) further removes the polylogarithmic factor in the time complexity of (Jin et al., 2018) by a restarting strategy.
Revised: The very recent work Li & Lin (2022) further removes the polylogarithmic factor in the time complexity of Jin et al. (2018) by a restarting strategy.

Original: Because our objective is nonsmooth, the Hamiltonian cannot be decreased by exploiting the negative curvature of the objective in the same way as (Jin et al., 2018).
Revised: Because our objective is nonsmooth, the Hamiltonian cannot be decreased by exploiting the negative curvature of the objective in the same way as Jin et al.

Original: However, we still manage to design a “Negative-Curvature-Exploitation” algorithm which decreases the objective value by an absolute positive amount when the iterates of PPGD cross endpoints.
Revised: However, we still manage to design a “Negative-Curvature-Exploitation” algorithm which decreases the objective value by an absolute positive amount when the iterates of PPGD cross endpoints.

Original: Our results are among the very few results in the optimization literature about fast optimization methods for nonconvex and nonsmooth problems which are not limited to ℓ 0 -regularized problems.
Revised: Our results are among the very few results in the optimization literature about fast optimization methods for nonconvex and nonsmooth problems which are not limited to ℓ 0 -regularized problems.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: Throughout this paper, we use bold letters for matrices and vectors, regular lower letters for scalars.
Revised: Throughout this paper, we use bold letters for matrices and vectors, regular lower letters for scalars.

Original: The bold letter with subscript indicates the corresponding element of a matrix or vector.
Revised: The bold letter with subscript indicates the corresponding element of a matrix or vector.

Original: ∥·∥ p denotes the ℓ p -norm of a vector, or the p -norm of a matrix.
Revised: ∥·∥ p denotes the ℓ p -norm of a vector, or the p -norm of a matrix.

Original: | A | denotes the cardinality of a set A .
Revised: | A | denotes the cardinality of a set A .

Original: When
Revised: When

Original: R ⊆ R is an interval, R is the closure of R , R + denotes the set of all the points in R to the right of
Revised: R ⊆ R is an interval, R is the closure of R , R + denotes the set of all the points in R to the right of

Original: R while not in R , and R − is defined similarly denoting the set of all the points to the left of R .
Revised: R while not in R , and R − is defined similarly denoting the set of all the points to the left of R .

Original: The domain of any function u is denoted by dom u .
Revised: The domain of any function u is denoted by dom u .

Original: lim x → a + and lim x
Revised: lim x → a + and lim x

Original: → a − denote left limit and right limit at a point a ∈ R .
Revised: → a − denote left limit and right limit at a point a ∈ R .

Original: N is the set of all natural numbers.
Revised: N is the set of all natural numbers.

Original:  [Equation] TO F AST C ONVERGENCE BY PPGD
Revised:  R OADMAP TO F AST C ONVERGENCE BY PPGD

Original: Two essential components of PPGD contribute to its fast convergence rate.
Revised: Two essential components of PPGD contribute to its fast convergence rate.

Original: The first component, a combination of a carefully designed Negative-Curvature-Exploitation algorithm and a new projection operator, decreases the objective function by a positive amount when the iterates generated by
Revised: The first component, a combination of a carefully designed Negative-Curvature-Exploitation algorithm and a new projection operator, decreases the objective function by a positive amount when the iterates generated by

Original: PPGD transit from one convex piece to another.
Revised: PPGD transit from one convex piece to another.

Original: As a result, there can be only a finite number of such transitions.
Revised: As a result, there can be only a finite number of such transitions.

Original: After finite iterations, all iterates must stay on the same convex pieces.
Revised: After finite iterations, all iterates must stay on the same convex pieces.

Original: Restricted on these convex pieces, problem (1) is convex.
Revised: Restricted on these convex pieces, problem (1) is convex.

Original: The second component, which comprises M surrogate functions, naturally enables that after iterates reach their final convex pieces, they are operated in the same way as a regular APG does so that the convergence rate of PPGD locally matches
Revised: The second component, which comprises M surrogate functions, naturally enables that after iterates reach their final convex pieces, they are operated in the same way as a regular APG does so that the convergence rate of PPGD locally matches

Original: the
Revised: the

Original: Nesterov’s optimal rate achieved by APG.
Revised: Nesterov’s optimal rate achieved by APG.

Original: Restricted on each convex piece, the piecewise convex function f is convex.
Revised: Restricted on each convex piece, the piecewise convex function f is convex.

Original: Every surrogate function is designed to be an extension of this restricted function to the entire R , and PPGD performs proximal mapping only on the surrogate functions.
Revised: Every surrogate function is designed to be an extension of this restricted function to the entire R , and PPGD performs proximal mapping only on the surrogate functions.

Original:  ALGORITHMS
Revised:  ALGORITHMS

Original: Before presenting the proposed algorithm, we first define proximal mapping, and then describe how to build surrogate functions for each convex piece.
Revised: Before presenting the proposed algorithm, we first define proximal mapping, and then describe how to build surrogate functions for each convex piece.

Original: Definition 3.1 (Proximal Mapping) .
Revised: Definition 3.1 (Proximal Mapping) .

Original: The proximal mapping associated with function u is defined as prox u ( x ) :
Revised: The proximal mapping associated with function u is defined as prox u ( x ) :

Original: = argmin v ∈ R d u ( v ) + 12 ∥ v − x ∥ 22 for x ∈ R d , and prox u ( · ) is called the proximal mapping associated with u .
Revised: = argmin v ∈ R d u ( v ) + 12 ∥ v − x ∥ 22 for x ∈ R d , and prox u ( · ) is called the proximal mapping associated with u .

Original: 3. C ONSTRUCTION OF THE S URROGATE F UNCTIONS { f m } Mm = 
Revised: 3. C ONSTRUCTION OF THE S URROGATE F UNCTIONS { f m } Mm =

Original: Given that f is convex on each convex piece R m , we describe how to construct a surrogate function f m : R → R such that f m ( x ) =
Revised: Given that f is convex on each convex piece R m , we describe how to construct a surrogate function f m : R → R such that f m ( x ) =

Original: f
Revised: f

Original: ( x ) for all x ∈ R m .
Revised: ( x ) for all x ∈ R m .

Original: The key idea is to extend the domain of f m from R m to R with the simplest structure, that is, f m is linear outside of R m .
Revised: The key idea is to extend the domain of f m from R m to R with the simplest structure, that is, f m is linear outside of R m .

Original: More concretely, if the right endpoint q = q m is not + ∞ and f is continuous at q , then f
Revised: More concretely, if the right endpoint q = q m is not + ∞ and f is continuous at q , then f

Original: m extends f | R m such that f m on ( q, + ∞ ) is linear.
Revised: m extends f | R m such that f m on ( q, + ∞ ) is linear.

Original: Similar extension applies to the case when q = q m − 1 is the left endpoint of
Revised: Similar extension applies to the case when q = q m − 1 is the left endpoint of

Original: R m .
Revised: R m .

Original: Formally, we let
Revised: Formally, we let

Original: [Equation]
Revised: [Equation]

Original: 5) for all x ∈ R + m if q m ̸ = + ∞ , where v − =
Revised: ) for all x ∈ R + m if q m ̸ = + ∞ , where v − =

Original: lim x → q − m f ′ ( x ) .
Revised: lim x → q − m f ′ ( x ) .

Original: The surrogate function f m is defined similarly on R − m by
Revised: The surrogate function f m is defined similarly on R − m by

Original: [Equation]
Revised: [Equation]

Original: (6) for all x ∈ R − m if q m − 1 ̸ =
Revised: for all x ∈ R − m if q m − 1 ̸ =

Original: −∞ , where v + = lim x
Revised: −∞ , where v + = lim x

Original: → q + m − 1 f ′ ( x ) .
Revised: → q + m − 1 f ′ ( x ) .

Original: Figure 2 illustrates the surrogate function f m ( x ) with x ∈ R + m for the three different cases in (5).
Revised: Figure 2 illustrates the surrogate function f m ( x ) with x ∈ R + m for the three different cases in (5).

Original: [Equation]
Revised: [Equation]

Original: iscontinuous at m f q ( ) ( [Equation] ) m [Equation]
Revised: is

Original: [Equation]

Original: m

Original: [Equation]
Revised: [Equation]

Original: m

Original: [Equation]

Original: q f

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Example 3.1 explains how to build surrogate functions for two popular piecewise convex functions in the optimization literature.
Revised: Example 3.1 explains how to build surrogate functions for two popular piecewise convex functions in the optimization literature.

Original: It can be observed from the definition of surrogate functions and
Revised: It can be observed from the definition of surrogate functions and

Original: Figure 2 that surrogate functions always have a simpler geometric structure than the original f , so it is usually the case that proximal mapping associated with the surrogate functions have closedform expressions, given that proximal mapping associated with f has a closed-form solution.
Revised: Figure 2 that surrogate functions always have a simpler geometric structure than the original f , so it is usually the case that proximal mapping associated with the surrogate functions have closedform expressions, given that proximal mapping associated with f has a closed-form solution.

Original: The proximal mappings associated with the surrogate functions are also given in Example 3.1.
Revised: The proximal mappings associated with the surrogate functions are also given in Example 3.1.

Original: Example 3.1 (Examples of Surrogate Functions) .
Revised: Example 3.1 (Examples of Surrogate Functions) .

Original: The capped- ℓ 1 penalty function f ( x ) =
Revised: The capped- ℓ 1 penalty function f ( x ) =

Original: λ min {| x | , b } has three surrogate functions as f 1 = f 3 ≡ λb , and
Revised: λ min {| x | , b } has three surrogate functions as f 1 = f 3 ≡ λb , and

Original: f 2 = λ | x | .
Revised: f 2 = λ | x | .

Original: The proximal mappings for all { f i } 3 i =1 has closed-form expressions, with prox sf 1 ( x ) =
Revised: The proximal mappings for all { f i } 3 i =1 has closed-form expressions, with prox sf 1 ( x ) =

Original: prox sf 3 ( x )
Revised: prox sf 3 ( x )

Original: = x , and prox sf 2 ( x ) =
Revised: = x , and prox sf 2 ( x ) =

Original: (1 − λs/ | x | ) + x being the soft thresholding operator, where ( a ) + = a when a ≥and ( a ) + = 0 when a < 0 .
Revised: (1 − λs/ | x | ) + x being the soft thresholding operator, where ( a ) + = a when a ≥and ( a ) + = 0 when a < 0 .

Original: The indicator penalty function f ( x ) =
Revised: The indicator penalty function f ( x ) =

Original: λ 1I { x<τ } has surrogate functions f 1 ≡ λ , f 2 = f .
Revised: λ 1I { x<τ } has surrogate functions f 1 ≡ λ , f 2 = f .

Original: The proximal mappings are prox sf 1 ( x ) = x , prox sf 2 ( x ) =
Revised: The proximal mappings are prox sf 1 ( x ) = x , prox sf 2 ( x ) =

Original: τ when τ − √ 2 λs < x < τ , and prox sf 2 ( x ) =
Revised: τ when τ − √ 2 λs < x < τ , and prox sf 2 ( x ) =

Original: x otherwise except for the point x = τ − √ 2 λs .
Revised: x otherwise except for the point x = τ − √ 2 λs .

Original: While prox sf 2 ( x ) has two values at x = τ − √ 2 λs , it will not be evaluated at x = τ − √ 2 λs in our PPGD algorithm to be introduced.
Revised: While prox sf 2 ( x ) has two values at x = τ − √ 2 λs , it will not be evaluated at x = τ − √ 2 λs in our PPGD algorithm to be introduced.

Original: 3. P ROJECTIVE P ROXIMAL G RADIENT D ESCENT
Revised: 3. P ROJECTIVE P ROXIMAL G RADIENT D ESCENT

Original: We now define a basic operator P which returns the index of the convex piece that the input lies in, that is, P ( x ) = m for x ∈ R m .
Revised: We now define a basic operator P which returns the index of the convex piece that the input lies in, that is, P ( x ) = m for x ∈ R m .

Original: It is noted that P ( x ) is a single-valued function, so P ( x ) ̸ = P ( y ) indicates x, y are on different convex pieces, that is, there is no convex piece which contain both x and y .
Revised: It is noted that P ( x ) is a single-valued function, so P ( x ) ̸ = P ( y ) indicates x, y are on different convex pieces, that is, there is no convex piece which contain both x and y .

Original: Given a vector x ∈ R d , we define P ( x ) ∈ R d such that [ P ( x )] i = P ( x i ) for all
Revised: Given a vector x ∈ R d , we define P ( x ) ∈ R d such that [ P ( x )] i = P ( x i ) for all

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: We then introduce a novel projection operator P x ,R 0 ( · ) for any x ∈ R d .
Revised: We then introduce a novel projection operator P x ,R 0 ( · ) for any x ∈ R d .

Original: That is, P x ,R 0 : R d → R d , and
Revised: That is, P x ,R 0 : R d → R d , and

Original: P x ,R 0 ( u ) for all u ∈ R d is defined as
Revised: P x ,R 0 ( u ) for all u ∈ R d is defined as

Original: [Equation]
Revised: [Equation]

Original:  1: z (1) =
Revised: (0) (cid:1) 1: z (1) =

Original: x (1) =
Revised: x (1) =

Original: x (0) , t 0 = 0 , t 1 = 1 , endpoint { q m } M − 1 m =1 , step size s , constant w 0 ∈ (0 , 1] .
Revised: x (0) , t 0 = 0 , t 1 = 1 , endpoint { q m } M − 1 m =1 , step size s , constant w 0 ∈ (0 , 1] .

Original: 2: for, do u ( k) = x ( [Equation]
Revised: 2: for k = 1 , . .

Original: + [Equation]
Revised: , do u ( k ) =

Original:  k − 1 t k ( [Equation] [Equation] [Equation]
Revised: x ( k ) + t k − 1 t k ( z ( k ) − x ( k ) )

Original: [Equation] t k [Equation] 1  − 
Revised: + t k − 1 − 1 t k ( x ( k ) − x ( k − 1) )

Original: [Equation] k ( x ( k ) [Equation] x ( k − 1) ) (7) w ( k ) =
Revised: (7) w ( k ) =

Original: P x ( k ) ,R 0 ( u ( k ) )
Revised: P x ( k ) ,R 0 ( u ( k ) )

Original: (8) for i = 1 , . . .
Revised: (8) for i = 1 , . . .

Original: , d, do z ( k +1)
Revised: , d, do z ( k +1)

Original: i =
Revised: i =

Original: prox sf P ( x ( k ) i ) (cid:16)(cid:104) w ( k ) − s ∇ g ( w ( k ) ) (cid:105) i (cid:17) (9)
Revised: prox sf P ( x ( k ) i ) (cid:16)(cid:104) w ( k ) − s ∇ g ( w ( k ) ) (cid:105) i (cid:17) (9)

Original: t k +1 =
Revised: t k +1 =

Original: (cid:112) 1 + 4 t 2 k
Revised: (cid:112) 1 + 4 t 2 k

Original: + .
Revised: + .

Original: if F P ( x ( k ) )
Revised: if F P ( x ( k ) )

Original: ( z ( k +1) ) ≤ F ( x ( k ) )
Revised: ( z ( k +1) ) ≤ F ( x ( k ) )

Original: then x ( k +1) =
Revised: then x ( k +1) =

Original: Negative-Curvature-Exploitation (cid:16) x ( k ) , z ( k +1) , w 0 (cid:17) (10) else x ( k +1)
Revised: Negative-Curvature-Exploitation (cid:16) x ( k ) , z ( k +1) , w 0 (cid:17) (10) else x ( k +1)

Original: =
Revised: =

Original: x ( k ) for all i ∈ [ d ] , where B ( x i , R 0 ) :=
Revised: x ( k ) for all i ∈ [ d ] , where B ( x i , R 0 ) :=

Original: [ x i − R 0 , x i + R 0 ] .
Revised: [ x i − R 0 , x i + R 0 ] .

Original: Since R P ( x i ) ∩ B ( x i , R 0 ) is a closed convex set, (11) is the projection of u i onto the closed convex set R P ( x i )
Revised: Since R P ( x i ) ∩ B ( x i , R 0 ) is a closed convex set, (11) is the projection of u i onto the closed convex set R P ( x i )

Original: ∩ B ( x i , R 0 ) which is well-defined.
Revised: ∩ B ( x i , R 0 ) which is well-defined.

Original: P x ,R 0 ( u ) can be computed very efficiently by setting the endpoint of R P ( x i )
Revised: P x ,R 0 ( u ) can be computed very efficiently by setting the endpoint of R P ( x i )

Original: ∩ B ( x i , R 0 ) closer to u i to [ P x ,R 0 ( u )] i .
Revised: ∩ B ( x i , R 0 ) closer to u i to [ P x ,R 0 ( u )] i .

Original: Algorithm 2 Negative-Curvature-Exploitation (cid:0) x ( k ) , z ( k +1) , w 0 (cid:1) 1: if P ( z ( k +1) ) =
Revised: -Curvature-Exploitation (cid:0) x ( k ) , z ( k +1) , w 0 (cid:1) 1: if P ( z ( k +1) ) =

Original: P ( x ( k ) )
Revised: P ( x ( k ) )

Original: then 2: x ( k +1)
Revised: then 2: x ( k +1)

Original: =
Revised: =

Original: z ( k +1) 3: return x ( k +1) 4: Flag = false 5: z ′ =
Revised: z ( k +1) 3: return x ( k +1) 4: Flag = false 5: z ′ =

Original: z ( k +1) 6: for i = 1 , . . .
Revised: z ( k +1) 6: for i = 1 , . . .

Original: , d do 7: if P ( z ( k +1) i ) ̸
Revised: , d do 7: if P ( z ( k +1) i ) ̸

Original: = P ( x ( k ) i )
Revised: = P ( x ( k ) i )

Original: =
Revised: =

Original: m i then 8: if f is continuous at q ( w ( k ) i ) and d i, 1 ≥ w 0 d
Revised: m i then 8: if f is continuous at q ( w ( k ) i ) and d i, 1 ≥ w 0 d

Original: i, 0 then 9:
Revised: i, 0 then 9:

Original: Flag = true 10: if f is not continuous at q ( w ( k ) i ) then 11:
Revised: Flag = true 10: if f is not continuous at q ( w ( k ) i ) then 11:

Original: Flag = true 12: if R P ( z ( k +1) i ) = (cid:110) q ( w ( k ) i ) (cid:111) then 13: z ′ i =
Revised: Flag = true 12: if R P ( z ( k +1) i ) = (cid:110) q ( w ( k ) i ) (cid:111) then 13: z ′ i =

Original: q ( w ( k ) i ). 14: if Flag = false then 15: x ( k +1)=
Revised: q ( w ( k ) i ) 14: if Flag = false then 15: x ( k +1)

Original:  x ( k ) 16:
Revised: = x ( k ) 16:

Original: else 17: x ( k +1)
Revised: else 17: x ( k +1)

Original: =
Revised: =

Original: z ′ 18: return x ( k +1)
Revised: z ′ 18: return x ( k +1)

Original: The Projected Proximal Gradient Descent (PPGD) algorithm is described in Algorithm 1.
Revised: The Projected Proximal Gradient Descent (PPGD) algorithm is described in Algorithm 1.

Original: The following functions and quantities are defined which are useful for PPGD.
Revised: The following functions and quantities are defined which are useful for PPGD.

Original: Let q ( w ( k ) i ) be the endpoint closest to w ( k ) i which lies in [ w ( k ) i , z ( k +1) i ] or [ z ( k +1)
Revised: Let q ( w ( k ) i ) be the endpoint closest to w ( k ) i which lies in [ w ( k ) i , z ( k +1) i ] or [ z ( k +1)

Original: i , w ( k ) i ] when P ( z ( k +1) i ) ̸
Revised: i , w ( k ) i ] when P ( z ( k +1) i ) ̸

Original: = P ( x ( k ) i ) at iteration k of Algorithm 1.
Revised: = P ( x ( k ) i ) at iteration k of Algorithm 1.

Original: We define F P ( x ) ( v ) :
Revised: We define F P ( x ) ( v ) :

Original: = g ( v ) + (cid:80) di
Revised: = g ( v ) + (cid:80) di

Original: =1 f P ( x i ) ( v i ) , d i, 0 :=
Revised: =1 f P ( x i ) ( v i ) , d i, 0 :=

Original: (cid:12)(cid:12)(cid:12) z ( k +1)
Revised: (cid:12)(cid:12)(cid:12) z ( k +1)

Original: i − w ( k ) i (cid:12)(cid:12)(cid:12) ,d i, 1 :
Revised: i − w ( k ) i (cid:12)(cid:12)(cid:12) ,

Original: [Equation]
Revised: [Equation]

Original:  The idea of PPGD is to use z ( k +1) as a probe for the next convex pieces that the current iterate x ( k ) should transit to.
Revised: i [Equation] [Equation] The idea of PPGD is to use z ( k +1) as a probe for the next convex pieces that the current iterate x ( k ) should transit to.

Original: Compared to the regular APG described in Algorithm 4 in Section A.2 of the supplementary, the projection of u ( k ) onto a closed convex set is used to compute z ( k +1) .
Revised: Compared to the regular APG described in Algorithm 4 in Section A.2 of the supplementary, the projection of u ( k ) onto a closed convex set is used to compute z ( k +1) .

Original: This is to make sure that u ( k ) is “dragged” back to the convex pieces of x ( k ) , and the only variable that can explore new convex pieces is z ( k +1) .
Revised: This is to make sure that u ( k ) is “dragged” back to the convex pieces of x ( k ) , and the only variable that can explore new convex pieces is z ( k +1) .

Original: We have a novel Negative-Curvature-Exploitation (NCE) algorithm described in Algorithm 2 which decides if PPGD should update the next iterate x ( k +1) .
Revised: We have a novel Negative-Curvature-Exploitation (NCE) algorithm described in Algorithm 2 which decides if PPGD should update the next iterate x ( k +1) .

Original: In particular, if z ( k +1) is on the same convex pieces as x ( k ) , then x ( k +1) is updated to z ( k +1)
Revised: In particular, if z ( k +1) is on the same convex pieces as x ( k ) , then x ( k +1) is updated to z ( k +1)

Original: only if z ( k +1) has smaller objective value.
Revised: only if z ( k +1) has smaller objective value.

Original: Otherwise, the NCE algorithm carefully checks if any one of the two sufficient conditions can be met so that the objective value can be decreased.
Revised: Otherwise, the NCE algorithm carefully checks if any one of the two sufficient conditions can be met so that the objective value can be decreased.

Original: If this is the case, then Flag is set to True and x ( k +1) is updated by z ( k +1) , which indicates that x ( k +1) transits to convex pieces different from that of x ( k ) .
Revised: If this is the case, then Flag is set to True and x ( k +1) is updated by z ( k +1) , which indicates that x ( k +1) transits to convex pieces different from that of x ( k ) .

Original: Otherwise, x ( k +1) is set to the previous value x ( k ) .
Revised: Otherwise, x ( k +1) is set to the previous value x ( k ) .

Original: The two sufficient conditions are checked in line 8 and 10 of Algorithm 2.
Revised: The two sufficient conditions are checked in line 8 and 10 of Algorithm 2.

Original: Such properties of NCE along with the convergence of PPGD will be proved in the next section.
Revised: Such properties of NCE along with the convergence of PPGD will be proved in the next section.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, we present the analysis for the convergence rate of PPGD in Algorithm 1.
Revised: In this section, we present the analysis for the convergence rate of PPGD in Algorithm 1.

Original: Before that, we present the definition of Fr ´ echet subdifferential which is used to define critical points of the objective function.
Revised: Before that, we present the definition of Fr ´ echet subdifferential which is used to define critical points of the objective function.

Original: Definition
Revised: Definition

Original: (Fr ´ echet subdifferential and critical points (Rockafellar & Wets, 2009))
Revised: (Fr ´ echet subdifferential and critical points (Rockafellar & Wets, 2009))

Original: Given a non-convex function u : R d → R ∪ { + ∞} which is a proper and lower semicontinuous function.
Revised: Given a non-convex function u : R d → R ∪ { + ∞} which is a proper and lower semicontinuous function.

Original: • For a given x ∈ dom u , its Fr ´ echet subdifferential of u at x , 
Revised: • For a given x ∈ dom u , its Fr ´ echet subdifferential of u at x , denoted

Original: denoted by ˜ ∂u( x ) , is the set of all vectors u ∈ R
Revised: by ˜ ∂u x ) , is the set of all vectors u ∈ R

Original: d which satisfy
Revised: d which satisfy

Original: [Equation]
Revised: [Equation]

Original: • The limiting-subdifferential of u at x ∈ R d , denoted by written ∂u ( x ) , is defined by
Revised: • The limiting-subdifferential of u at x ∈ R d , denoted by written ∂u ( x ) , is defined by

Original: [Equation]
Revised: [Equation]

Original: The point x is a critical point of u if 0 ∈ ∂u ( x ) .
Revised: The point x is a critical point of u if 0 ∈ ∂u ( x ) .

Original: The following assumption is useful for our analysis when the convex pieces have one or more endpoints at which f is continuous.
Revised: The following assumption is useful for our analysis when the convex pieces have one or more endpoints at which f is continuous.

Original: Assumption 2 (Nonvanishing Gradient at Continuous Endpoints) .
Revised: Assumption 2 (Nonvanishing Gradient at Continuous Endpoints) .

Original: Let q m with m ∈ [ M − 1] be an endpoint at which f is continuous.
Revised: Let q m with m ∈ [ M − 1] be an endpoint at which f is continuous.

Original: There exists a positive constant ε 0 such that the following two conditions hold.
Revised: There exists a positive constant ε 0 such that the following two conditions hold.

Original: If q m is a left endpoint, then inf x ∈ R d , x i =
Revised: If q m is a left endpoint, then inf x ∈ R d , x i =

Original: q m | [ ∇ g ( x )]
Revised: q m | [ ∇ g ( x )]

Original: i + p + | ≥ ε 0 for all
Revised: i + p + | ≥ ε 0 for all

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: If q m is a right endpoint, then inf x ∈ R d , x i =
Revised: If q m is a right endpoint, then inf x ∈ R d , x i =

Original: q m | [ ∇ g ( x )]
Revised: q m | [ ∇ g ( x )]

Original: i + p − | ≥ ε 0 .
Revised: i + p − | ≥ ε 0 .

Original: Here p + := lim y
Revised: Here p + := lim y

Original: → q + m f ′ ( y ) , and p − :=
Revised: → q + m f ′ ( y ) , and p − :=

Original: lim y
Revised: lim y

Original: → q − m f ′ ( y ) .
Revised: → q − m f ′ ( y ) .

Original: We note that Assumption 2 is rather mild.
Revised: We note that Assumption 2 is rather mild.

Original: For example, when f ( x ) = λ min {| x | , b } is the cappedℓ 1 penalty, then Assumption 2 holds when λ > G and ε 0 can be set to λ − G .
Revised: For example, when f ( x ) = λ min {| x | , b } is the cappedℓ 1 penalty, then Assumption 2 holds when λ > G and ε 0 can be set to λ − G .

Original: Such requirement for large λ is commonly used for analysis of consistency of sparse linear models, such as (Loh, 2017).
Revised: Such requirement for large λ is commonly used for analysis of consistency of sparse linear models, such as (Loh, 2017).

Original: In addition, when there are no endpoints at which f is continuous, then Assumption 2 is not required for the provable convergence rate of PPGD.
Revised: In addition, when there are no endpoints at which f is continuous, then Assumption 2 is not required for the provable convergence rate of PPGD.

Original: Details are deferred to Remark 4.5.
Revised: Details are deferred to Remark 4.5.

Original: We then have the important Theorem 4.3, which directly follows from Lemma 4.1 and Lemma 4.
Revised: We then have the important Theorem 4.3, which directly follows from Lemma 4.1 and Lemma 4.

Original: below.
Revised: below.

Original: Theorem 4.3 states that the Negative-Curvature-Exploitation algorithm guarantees that, if convex pieces change across consecutive iterates x ( k ) and x ( k +1) , then the objective value is decreased by a positive amount.
Revised: Theorem 4.3 states that the Negative-Curvature-Exploitation algorithm guarantees that, if convex pieces change across consecutive iterates x ( k ) and x ( k +1) , then the objective value is decreased by a positive amount.

Original: Before presenting these results, we note that the level set
Revised: Before presenting these results, we note that the level set

Original: L := (cid:8) x (cid:12) (cid:12) F ( x ) ≤ F ( x (0) ) (cid:9) is bounded because F is coercive.
Revised: L := (cid:8) x (cid:12) (cid:12) F ( x ) ≤ F ( x (0) ) (cid:9) is bounded because F is coercive.

Original: Since ∇ g is smooth, we define the supremum of ∥∇ g ∥ 2 over an enlarged version of L as G :
Revised: Since ∇ g is smooth, we define the supremum of ∥∇ g ∥ 2 over an enlarged version of L as G :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: G is finite due to the fact that L R 0 is compact, and it will serve as the upper bound for (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: G is finite due to the fact that L R 0 is compact, and it will serve as the upper bound for (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 .
Revised: (cid:13)(cid:13) 2 .

Original: Lemma 4.1 (Decrease of the objective function when convex pieces change) . 
Revised: Lemma 4.1 (Decrease of the objective function when convex pieces change) . Define A :=

Revised: L g ( G + √ dF 0 ) , κ 0 :

Revised: = min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε 0 − sL g (1 − w 0 )

Revised: ( G + F 0 )) , κ 2 :=

Original:  Let the step size 
Revised: J − 2 sF 0 ( G + F 0 ) , and κ := κ 0 − s (cid:16) sL g ( G + √ dF 0 ) + G (cid:17) ( G + √ dF 0 ) . Let the step size satisfy s < s 1 := min

Original: [Equation]
Revised: [Equation]

Original: and suppose (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: and suppose (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 ≤ G .
Revised: (cid:13)(cid:13) 2 ≤ G .

Original: If P ( x ( k +1) ) ̸
Revised: If P ( x ( k +1) ) ̸

Original: = P ( x ( k ) )
Revised: = P ( x ( k ) )

Original: for some k ≥ 0 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm, then under Assumption 1 and Assumption 2,
Revised: for some k ≥ 1 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm, then under Assumption 1 and Assumption 2,

Original: [Equation]
Revised: [Equation]

Original: where κ := min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε 0 − sL g (1 − w

Original: [Equation]

Original: )( G + F 0 )) , κ

Original: [Equation]

Original: = J − 2 sF 0 ( G + F

Original: [Equation]

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: The sequences (cid:8) x ( k ) (cid:9) k [Equation] and (cid:8) w ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm described in Algorithm 1 satisfy
Revised: The sequences (cid:8) x ( k ) (cid:9) k ≥ 1 and (cid:8) w ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm described in Algorithm 1 satisfy

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Suppose the step size satisfies [Equation]
Revised: Suppose the step size s < s 1 where s 1 is defined in (13).

Original: If P ( x ( k +1) ) ̸
Revised: If P ( x ( k +1) ) ̸

Original: = P ( x ( k ) )
Revised: = P ( x ( k ) )

Original: for some k ≥ 0 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm, then under Assumption 1 and Assumption 2,
Revised: for some k ≥ 1 in the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by the PPGD algorithm, then under Assumptionand Assumption 2,

Original: [Equation]
Revised: [Equation]

Original: where κ := min { κ 1 , κ 2 } with κ 1 := sCw 0 ( ε 0 − sL g (1 − w 0 ) ( G + F 0 )) , κ 2 := J − 2 sF 0 ( G + F 0 ) .
Revised: where κ is defined in Lemma 4.1.

Original: Due to Theorem 4.3, there can only be a finite number of k ’s such that the convex pieces change across consecutive iterates x ( k ) and x ( k +1) .
Revised: Due to Theorem 4.3, there can only be a finite number of k ’s such that the convex pieces change across consecutive iterates x ( k ) and x ( k +1) .

Original: As a result, after finite iterations all the iterates generated by PPGD lie on the same convex pieces, so the nice properties of convex optimization apply to these iterates.
Revised: As a result, after finite iterations all the iterates generated by PPGD lie on the same convex pieces, so the nice properties of convex optimization apply to these iterates.

Original: Formally, we have the following theorem stating the convergence rate of PPGD.
Revised: Formally, we have the following theorem stating the convergence rate of PPGD.

Original: Theorem 4.4 (Convergence rate of PPGD) .
Revised: Theorem 4.4 (Convergence rate of PPGD) .

Original: Suppose the step size s < min (cid:110) s 1 ,
Revised: Suppose the step size s < min (cid:110) s [Equation]

Original: [Equation]
Revised: [Equation]

Original: with s 1 defined by (16), Assumption 1 and Assumption 2 hold.
Revised: with s 1 defined by (13), Assumption 1 and Assumption 2 hold.

Original: Then there exists a finite k 0 ≥such that the following statements hold.
Revised: Then there exists a finite k 0 ≥such that the following statements hold.

Original: (1) P ( x ( k ) )
Revised: (1) P ( x ( k ) )

Original: = m ∗ for some m ∗ ∈ N d for all k > k 0 .
Revised: = m ∗ for some m ∗ ∈ N d for all k > k 0 .

Original: (2) Let Ω :=
Revised: (2) Let Ω :=

Original: (cid:110) x
Revised: (cid:110) x

Original: (cid:12)(cid:12)(cid:12) x is a limit point of (cid:8) x
Revised: (cid:12)(cid:12)(cid:12) x is a limit point of (cid:8) x

Original: ( k ) (cid:9) k ≥ 1 , P ( x )
Revised: ( k ) (cid:9) k ≥ 1 , P ( x )

Original: = m ∗ (cid:111) be the set of all limit points of the sequence (cid:8) x
Revised: = m ∗ (cid:111) be the set of all limit points of the sequence (cid:8) x

Original: ( k ) (cid:9) k ≥ 1 generated by Algorithm 1 lying on the convex pieces indexed by m ∗ .
Revised: ( k ) (cid:9) k ≥ 1 generated by Algorithm 1 lying on the convex pieces indexed by m ∗ .

Original: Then for any x ∗ ∈ Ω , F ( x ∗ ) = inf { x ∈ R d
Revised: Then for any x ∗ ∈ Ω , F ( x ∗ ) = inf { x ∈ R d

Original: | P ( x )= m ∗ } F ( x ) , and
Revised: [Equation] F [Equation]

Original: F ( x ( k
Revised: x ) , and

Original: [Equation]
Revised: [Equation]

Original: 18) for all k > k 0 ,
Revised: 17) for all k > k 0 ,

Original: where U ( k 0 ) :
Revised: where U ( k 0 ) :

Original: = (cid:0) 12 s ∥ ( t k 0 − 1 − 1) x ( k 0 − 1) − t k 0 − 1 z ( k 0 )
Revised: = (cid:0) 12 s ∥ ( t k 0 − 1 − 1) x ( k 0 − 1) − t k 0 − 1 z ( k 0 )

Original: + x ∗ ∥ 2 2 + t 2 k 0 − 1 ( F ( x ( k 0 ) )
Revised: + x ∗ ∥ 22 + t 2 k 0 − 1 ( F ( x ( k 0 ) )

Original: −
Revised: −

Original: F ( x ∗ ))
Revised: F ( x ∗ ))

Original: (cid:1) .
Revised: (cid:1) .

Original: Moreover, if f m ∗ i does not take the third case in (5) or (6) for all i ∈
Revised: Moreover, if f m ∗ i does not take the third case in (5) or (6) for all i ∈

Original: [ d ] , then x ∗ is a critical point of F , that is, 0 ∈ ∂F ( x ∗ ) .
Revised: [ d ] , then x ∗ is a critical point of F , that is, 0 ∈ ∂F ( x ∗ ) .

Original: Remark 4.5.
Revised: Remark 4.5.

Original: If the convex pieces {R m } Mm =1 has no endpoints at which f is continuous, then Theorem 4.4 holds without requiring Assumption 2.
Revised: If the convex pieces {R m } Mm =1 has no endpoints at which f is continuous, then Theorem 4.4 holds without requiring Assumption 2.

Original: In particular, Theorem 4.4 holds without Assumption 2 for problem (1) with indicator penalty or ℓ 0 -norm regularizer.
Revised: In particular, Theorem 4.4 holds without Assumption 2 for problem (1) with indicator penalty or ℓ 0 -norm regularizer.

Original: Remark 4.6.
Revised: Remark 4.6.

Original: All the statements of Theorem 4.4 still hold if f is not piecewise convex while satisfying all the other conditions of Assumption 1 and Assumption 2, if f restricted on the final convex piece R m ∗
Revised: All the statements of Theorem 4.4 still hold if f is not piecewise convex while satisfying all the other conditions of Assumption 1 and Assumption 2, if f restricted on the final convex piece R m ∗

Original: i is convex for all
Revised: i is convex for all

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: The reason is that we only need the convexity of f on the final convex pieces to prove the Nesterov’s optimal convergence rate in (18).
Revised: The reason is that we only need the convexity of f on the final convex pieces to prove the Nesterov’s optimal convergence rate in (17).

Original: EXPERIMENTAL RESULTS
Revised: Roadmap of the proof of Theorem 4.4. Proof of Theorem 4.4 is presented in Section B.6 of the supplementary, and the proof consists of three steps. In step 1 , it is proved that there exists a finite k 0 ≥ k 1 such that for all k > k 0 , [Equation]

Original: We conduct empirical study using the proposed PPGD methods in this section.
Revised: Noting that F ( x ( k ) ) = F m ∗ ( x ( k ) ) ≥ F m ∗ (¯ x ) by the optimality of ¯ x , the above inequality combined with the monotone nonincreasing of (cid:8) F ( x ( k ) ) (cid:9) indicate that F ( x ( k ) ) ↓ F m ∗ (¯ x ) .

Original: PPGD and the baseline optimization methods are used to solve the regularized logistic regression problem, min x 1 n n (cid:80) i =1 log(1 + exp( − y i x ′ x i )) + h ( x )  , where g ( x  ) = 1 n n (cid:80) i
Revised: In step 2 , it is proved that F ( x ′ ) = F m ∗ (¯ x ) for any limit point x ′ ∈ Ω . According to the definition of F m ∗ and the optimality of ¯ x , it follows that F ( x ′ ) is a local minimum of F and a global minimum of F m ∗ over R ∗ .

Original:  =
Revised: In step 3 , it is proved that any limit point x ′ ∈ Ω is a critical point of F under a mild condition, following the argument in step 2 .  EXPERIMENTAL RESULTS In this section, PPGD and the baseline optimization methods are used to solve the capped- ℓ regularized logistic regression problem, min x 1 n n (cid:80) i =

Original: 1 log(1 + exp( − y i x ′ x i )), and
Revised: 1 log(1 + exp( − y i x ⊤ x i ))

Original: [Figure] [Figure] h ( x ) =
Revised: + h ( x ) , where { x i , y i } ni

Original: (cid:80) di =1 f ( x i ) for three choices of f ( x ) : the indicator penalty, f ( x ) = 
Revised: = are training data, g ( x ) = 1 n n (cid:80) i

Original: λ 1I { x< 0 } ; the capped- ℓ penalty, f ( x ) =
Revised: =1 log(1 + exp( − y i

Original: λ min {| x | , b } ; the ℓ 0 -norm, f ( x ) = 
Revised: x ⊤ x i )) , and h ( x ) = (cid:80) di

Original: λ 1I { x ̸ =0 } .
Revised: =1 f ( x i ) with f ( x ) =

Original: Three three choices for f correspond to nonnegative logistic regression, capped- ℓ regularized logistic regression, and ℓ 0 -regularized logistic regression.
Revised: λ min {| x | , b } .

Original: We set λ = 0 .
Revised: We set λ = 0 .

Original: 2 and conduct experiments on the MNIST handwritten digits dataset (LeCun et al., 2010) and the CIFAR-10 dataset (Krizhevsky, 2009).
Revised: 2 and conduct experiments on the MNIST handwritten digits dataset (LeCun et al., 1998). [Figure] logistic regression

Original: The MNIST data set contains 70 , 000 examples of handwritten digit images of 10 classes.
Revised: The MNIST data set contains 70 ,examples of handwritten digit images of 10 classes.

Original: Each image is of size 28 × 28 and represented by a 784 -dimensional vector.
Revised: Each image is of size 28 × 28 and represented by a 784 -dimensional vector.

Original: The CIFAR-10 dataset contains 60000 images of size 32 × 32 with 10 classes.
Revised: The CIFAR10 dataset contains 60000 images of size 32 × 32 with 10 classes.

Original: For each dataset, we randomly sample two classes with 5000 images in each class, which form the training data { x i , y i } 10000 i = with { x i } 10000 i = being the sampled images and { y i } 10000 i = being the corresponding class labels ( ± 1 ).
Revised: For each dataset, we randomly sample two classes with 5000 images in each class, which form the training data { x i , y i } 10000 i = with { x i } 10000 i = being the sampled images and { y i } 10000 i = being the corresponding class labels ( ± 1 ).

Original: We compare PPGD toProximal Gradient Descent (PGD), the regular monotone Accelerated Proximal Gradient (APG) described in Algorithm 4, and the monotone Accelerated Proximal Gradient (mAPG) algorithm
Revised: We compare PPGD to the regular monotone Accelerated Proximal Gradient (APG) described in Algorithm 4 and the monotone Accelerated Proximal Gradient (mAPG) algorithm

Original: (Li & Lin, 2015).
Revised: (Li & Lin, 2015).

Original: Due to the observation that monotone version of accelerated gradient methods usually converge faster than its nonmonotone counterparts, the regular nonmonotone APG and the nonmonotone Accelerated Proximal Gradient (nmAPG) method (Li & Lin, 2015) are not included in the baselines.PPGD is also compared to the fast PGD method (Yang & Yu, 2020), referred to as APGD- ℓ 0 , in our experiments for ℓ 0 -regularized logistic regression.
Revised: Due to the observation that monotone version of accelerated gradient methods usually converge faster than its nonmonotone counterparts, the regular nonmonotone APG and the nonmonotone Accelerated Proximal Gradient (nmAPG) method (Li & Lin, 2015) are not included in the baselines.

Original: Figure 3and Figure 4 illustrate the decrease of objective value with respect to the iteration number on the MNIST dataset and the
Revised: Figure 3 illustrate the decrease of objective value with respect to the iteration number on the MNIST dataset.

Original: CIFAR-10 dataset respectively.

Original: It can be observed that PPGD always converges faster than all the competing baselines, evidencing the proved fast convergence results in this paper.
Revised: It can be observed that PPGD always converges faster than all the competing baselines, evidencing the proved fast convergence results in this paper.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We present Projective Proximal Gradient Descent (PPGD) to solve a class of challenging nonconvex and nonsmooth optimization problems.
Revised: We present Projective Proximal Gradient Descent (PPGD) to solve a class of challenging nonconvex and nonsmooth optimization problems.

Original: Using a novel projection operator and a carefully designed
Revised: Using a novel projection operator and a carefully designed

Original: Negative-Curvature-Exploitation algorithm, PPGD locally matches the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function.
Revised: Negative-Curvature-Exploitation algorithm, PPGD locally matches the Nesterov’s optimal convergence rate of first-order methods on smooth and convex objective function.

Original: The effectiveness of
Revised: The effectiveness of

Original: PPGD is evidenced by experiments.
Revised: PPGD is evidenced by experiments.

Original: A M ORE DETAILS ABOUT THIS PAPER A. K URDYKA -Ł OJASIEWICZ 
Revised: A M ORE DETAILS ABOUT THIS PAPER A. K URDYKA -Ł OJASIEWICZ (KŁ) P ROPERTY

Original: KŁ P ROPERTY

Original: Definition A.1 (Kurdyka-Łojasiewicz (KŁ) Property) .
Revised: Definition A.1 (Kurdyka-Łojasiewicz (KŁ) Property) .

Original: Let u : R d → ( −∞ , ∞ ] be proper and lower semicontinuous.
Revised: Let u : R d → ( −∞ , ∞ ] be proper and lower semicontinuous.

Original: u is said to satisfy the Kurdyka-Lojasiewicz (KŁ) property at u ∈ dom ∂u := { u ′ ∈ R d : ∂u ( u ′ ) ̸ = ∅} , if there exist η ∈ (0 , ∞ ] , a neighborhood U of u and a function ϕ ∈
Revised: u is said to satisfy the Kurdyka-Lojasiewicz (KŁ) property at u ∈ dom ∂u := { u ′ ∈ R d : ∂u ( u ′ ) ̸ = ∅} , if there exist η ∈ (0 , ∞ ] , a neighborhood U of u and a function ϕ ∈

Original: Φ η , such that for all ¯ u ∈ U ∪ [ u ( u ) <
Revised: Φ η , such that for all ¯ u ∈ U ∪ [ u ( u ) <

Original: u (¯ u ) < u ( u ) + η ] , the following inequality holds:
Revised: u (¯ u ) < u ( u ) + η ] , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Here η ∈ (0 , ∞ ] , Φ η is defined to be the class of all concave and continuous functions ϕ : [0 , η ) → R + which satisfy the following conditions:
Revised: Here η ∈ (0 , ∞ ] , Φ η is defined to be the class of all concave and continuous functions ϕ : [0 , η ) → R + which satisfy the following conditions:

Original: ϕ (0) = 0 ; 2.
Revised: ϕ (0) = 0 ; 2.

Original: ϕ is continuously differentiable on (0 , η ) and continuous at 0 ; 3.
Revised: ϕ is continuously differentiable on (0 , η ) and continuous at 0 ; 3.

Original: ϕ ′ ( r ) > 0 for all r ∈ (0 , η ) .
Revised: ϕ ′ ( r ) > 0 for all r ∈ (0 , η ) .

Original: If u satisfies the KŁ property at each point of dom ∂u , then u is called a KŁ function.
Revised: If u satisfies the KŁ property at each point of dom ∂u , then u is called a KŁ function.

Original: A. P ROXIMAL G RADIENT D ESCENT (PGD) AND MONOTONE A CCELERATED P ROXIMAL G RADIENT (APG) D ESCENT
Revised: A. P ROXIMAL G RADIENT D ESCENT (PGD) AND MONOTONE A CCELERATED P ROXIMAL G RADIENT (APG) D ESCENT

Original: We describe PGD and the regular monotone APG in this subsection.
Revised: We describe PGD and the regular monotone APG in this subsection.

Original: In the k -th iteration of PGD for k ≥ 1 , gradient descent is performed on the squared loss term g ( x ) to obtain an intermediate variable as the result of gradient descent, i.e. x ( k ) − s ∇ g ( x ( k ) ) , where s > 0 is the step size, and 1 s is usually chosen to be larger than L g .
Revised: In the k -th iteration of PGD for k ≥ 1 , gradient descent is performed on the squared loss term g ( x ) to obtain an intermediate variable as the result of gradient descent, i.e. x ( k ) − s ∇ g ( x ( k ) ) , where s > 0 is the step size, and 1 s is usually chosen to be larger than L g .

Original: x ( k +1) is computed by proximal mapping on x ( k ) − 
Revised: x ( k +1) is computed by proximal mapping on x ( k ) − s ∇ g ( x ( k ) ) . 1: for k=1,..., do

Original: [Equation]
Revised: [Equation]

Original: g ( x ( k ) )

Original: [Equation]
Revised: [Equation]

Original: 1: for k=1,..., do
Revised: 1: z (1) = x

Original: [Equation]
Revised: [Equation]

Original: 1: z (1) = x (1) =
Revised: =

Original: x (0) , t 0 = 0.
Revised: x (0) , t 0 = 0.

Original: 2: for k = 1 , . . .
Revised: 2: for k = 1 , . . .

Original: , do
Revised: , do

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Revised: k

Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: The iterations start from k = 1 and continue until the sequence { F ( x ( k ) )
Revised: The iterations start from k = 1 and continue until the sequence { F ( x ( k ) )

Original: } k or { x ( k ) } k converges or maximum iteration number is achieved.
Revised: } k or { x ( k ) } k converges or maximum iteration number is achieved.

Original: The optimization algorithm for the ℓ 0 sparse approximation problem (1) by PGD is described in Algorithm 3.
Revised: The optimization algorithm for the ℓ 0 sparse approximation problem (1) by PGD is described in Algorithm 3.

Original: In practice, the time complexity of optimization by PGD is O ( Mdn ) where M is the number of iterations (or maximum number of iterations) for
Revised: In practice, the time complexity of optimization by PGD is O ( Mdn ) where M is the number of iterations (or maximum number of iterations) for

Original: PGD.
Revised: PGD.

Original: The regular monotone Accelerated Proximal Gradient (APG) descent algorithm is described in Algorithm 4.
Revised: The regular monotone Accelerated Proximal Gradient (APG) descent algorithm is described in Algorithm 4.

Original: B P ROOFS
Revised: B P ROOFS

Original: The following lemma shows that the Fermat’s rule still applies to functions in the sense of Fr´echet subdifferential, that is, local mininum has 0 in its Fr´echet subdifferential and also its (limiting- )subdifferential.
Revised: The following lemma shows that the Fermat’s rule still applies to functions in the sense of Fr´echet subdifferential, that is, local mininum has 0 in its Fr´echet subdifferential and also its (limiting- )subdifferential.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: (Fermat’s rule for Fr´echet subdifferential and (limiting-)subdifferential, also in (Rockafellar & Wets, 2009, Theorem 10.1))
Revised: (Fermat’s rule for Fr´echet subdifferential and (limiting-)subdifferential, also in (Rockafellar & Wets, 2009, Theorem 10.1))

Original: Let u be a real-valued function defined on Ω ⊆ R d , u :=
Revised: Let u be a real-valued function defined on Ω ⊆ R d , u :=

Original: Ω → R , has a local minimum point at x ∈ Ω .
Revised: Ω → R , has a local minimum point at x ∈ Ω .

Original: Then 0 ∈ ˜
Revised: Then 0 ∈ ˜

Original: ∂u ( x ) ⊆
Revised: ∂u ( x ) ⊆

Original: ∂u ( x ) .
Revised: ∂u ( x ) .

Original: Proof.
Revised: Proof.

Original: Since x ∈ Ω is a local minimum point, then there exisits a neighborhood x ∈ U ( x ) such that u ( y ) ≥ u ( x ) for all y ∈ U ( x ) .
Revised: Since x ∈ Ω is a local minimum point, then there exisits a neighborhood x ∈ U ( x ) such that u ( y ) ≥ u ( x ) for all y ∈ U ( x ) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: By (20), 0 ∈ ˜ ∂u ( x ) .
Revised: By (19), 0 ∈ ˜ ∂u ( x ) .

Original: Because ˜ ∂u ( x ) ⊆
Revised: Because ˜ ∂u ( x ) ⊆

Original: ∂u ( x ) , 0 ∈ ˜ ∂u ( x ) ⊆
Revised: ∂u ( x ) , 0 ∈ ˜ ∂u ( x ) ⊆

Original: ∂u ( x ) .
Revised: ∂u ( x ) .

Original: We need to have the following two lemmas before proving Lemma 4.1.
Revised: We need to have the following two lemmas before proving Lemma 4.1.

Original: Lemma B.2 (Decrease of f at continuous endpoint) .
Revised: Lemma B.2 (Decrease of f at continuous endpoint) .

Original: Suppose Assumption 1 and Assumptionhold, s < min (cid:110) s 0 G + F 0 , ε 

Revised: Suppose Assumption 1 and Assumptionhold, s < min (cid:110) s 0 G + F 0 , ε

Original: L g (1 − w 0 )( G + F 0 ) (cid:111) , (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: L g (1 − w 0 )( G + F 0 ) (cid:111) , (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 ≤ G , P ( x ( k +1) i )
Revised: (cid:13)(cid:13) 2 ≤ G , P ( x ( k +1) i )

Original: ̸
Revised: ̸

Original: = P ( x ( k ) i )
Revised: = P ( x ( k ) i )

Original: = m i for some k ≥ 0 and some
Revised: = m i for some k ≥ 0 and some

Original: i ∈ [ d ] , and f is continuous at q ( w ( k ) i ) .
Revised: i ∈ [ d ] , and f is continuous at q ( w ( k ) i ) .

Original: Then f ( x ( k +1) i ) ≤ f m i ( z ( k +1) i ) .
Revised: Then f ( x ( k +1) i ) ≤ f m i ( z ( k +1) i ) .

Original: Furthermore, if d i, 1 ≥ w 0 d
Revised: Furthermore, if d i, 1 ≥ w 0 d

Original: i, 0 , then
Revised: i, 0 , then

Original: [Equation]
Revised: [Equation]

Original: where κ 1 :=
Revised: where κ 1 :=

Original: sCw 0 ( ε 0 − sL g (1 − w 0 )
Revised: sCw 0 ( ε 0 − sL g (1 − w 0 )

Original: ( G + F 0 )) .
Revised: ( G + F 0 )) .

Original: Lemma B.3 (Decrease of f at discontinuous endpoints) .
Revised: Lemma B.3 (Decrease of f at discontinuous endpoints) .

Original: Suppose Assumption 1 holds, s < min (cid:110) s 0 G + F 0 ,
Revised: Suppose Assumption 1 holds, s < min (cid:110) s 0 G + F 0 ,

Original: [Equation]
Revised: [Equation]

Original: and some i ∈ [ d ] .
Revised: and some i ∈ [ d ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where κ 2 :
Revised: where κ 2 :

Original: =
Revised: =

Original: J − 2 sF 0 ( G + F 0 ) .
Revised: J − 2 sF 0 ( G + F 0 ) .

Original: B. P ROOF OF L EMMA B.
Revised: B

Revised: P ROOF OF L EMMA B.

Original: Proof of Lemma B.2 .
Revised: Proof of Lemma B.2 .

Original: Let q = q ( w ( k ) i ) .
Revised: Let q = q ( w ( k ) i ) .

Original: We must have x ( k +1) i =
Revised: We must have x ( k +1) i =

Original: z [Equation] k +1)
Revised: z ( k +1)

Original: i with the PPGD algorithm and the Negative-Curvature-Exploitation algorithm described in Algorithm 1 and Algorithm 2.
Revised: i with the PPGD algorithm and the Negative-Curvature-Exploitation algorithm described in Algorithm 1 and Algorithm 2.

Original: Without loss of generality, we assume x ( k +1)
Revised: Without loss of generality, we assume x ( k +1)

Original: i ∈
Revised: i ∈

Original: R
Revised: R

Original: + m i .
Revised: + m i .

Original: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.
Revised: The case that x ( k +1) i ∈ R − m i can be proved in a similar manner.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: (23) by definition (5), where v − =
Revised: (22) by definition (5), where v − =

Original: lim x → q − f ′ ( x ) .
Revised: lim x → q − f ′ ( x ) .

Original: On the other hand, we have z ( k +
Revised: On the other hand, we have z ( k +

Original: [Equation]
Revised: [Equation]

Original: It follows by (24) that
Revised: It follows by (23) that

Original: [Equation]
Revised: [Equation]

Original: As a result, s < s 0 G + F 0 guarantees that x ( k +1) i ∈
Revised: As a result, s < s 0 G + F 0 guarantees that x ( k +1) i ∈

Original: ( q, q + s 0 ) .
Revised: ( q, q + s 0 ) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: when x ∈ ( q, x ( k +1) i ) .
Revised: when x ∈ ( q, x ( k +1) i ) .

Original: Letting x
Revised: Letting x

Original: → q in (26), we have
Revised: → q in (25), we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where v + = lim x → q + f ′ ( x ) .
Revised: where v + = lim x → q + f ′ ( x ) .

Original: By the negative curvature condition in Assumption 1(d), we have v − − v + ≥ C > 0 , therefore,
Revised: By the negative curvature condition in Assumption 1(d), we have v − − v + ≥ C > 0 , therefore,

Original: [Equation]
Revised: [Equation]

Original: Because z ( k +1) i ∈ R
Revised: Because z ( k +1) i ∈ R

Original: + m i , z ( k 
Revised: + m i , z ( k +

Original: [Equation]
Revised: [Equation]

Original:  i ≥ w ( k ) i .
Revised: ) i ≥ w ( k ) i .

Original: By (28) we have
Revised: By (27) we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let ˜ w ∈ R d with ˜ w j = w ( k ) j for all j ̸
Revised: Let ˜ w ∈ R d with ˜ w j = w ( k ) j for all j ̸

Original: = i , and ˜ w i = q .
Revised: = i , and ˜ w i = q .

Original: By the definition of ˜ w and (24), we have
Revised: By the definition of ˜ w and (23), we have

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (30).
Revised: where the last inequality follows from (29).

Original: It follows by (28) and (31) that
Revised: It follows by (27) and (30) that

Original: [Equation]
Revised: [Equation]

Original: Noting that x ( k +1)
Revised: Noting that x ( k +1)

Original: i =
Revised: i =

Original: z ( k +
Revised: z ( k +

Original: [Equation]
Revised: [Equation]

Original: i  completes the proof.
Revised: i in (31) completes the proof.

Original: B. P ROOF OF L EMMA B. 
Revised: B. P ROOF OF L EMMA B.

Original: Before presenting the proof, we introduce and prove the following lemma.
Revised: Before presenting the proof, we introduce and prove the following lemma.

Original: Lemma B.4.
Revised: Lemma B.4.

Original: Suppose Assumption 1 holds, (cid:12)(cid:12)(cid:0) ∇ g ( w ( k ) ) (cid:1) i (cid:12)(cid:12) ≤ G , and s < min (cid:110) 
Revised: Suppose Assumption 1 holds, (cid:13)(cid:13) w ( k ) (cid:13)(cid:13) 2 ≤ G , and s < min (cid:110) [Equation] F

Original: [Equation]
Revised: [Equation]

Original: G + G 2 / 2 , s 0 G (cid:111) ,
Revised: G + G 2 / 2 , s 0 G (cid:111) ,

Original: P ( x ( k +1) i ) ̸
Revised: P ( x ( k +1) i ) ̸

Original: = P ( x ( k ) i )
Revised: = P ( x ( k ) i )

Original: = m i for some k ≥ 0 and some
Revised: = m i for some k ≥ 0 and some

Original: i ∈ [ d ] , and f is not continuous at q ( w ( k ) i ) .
Revised: i ∈ [ d ] , and f is not continuous at q ( w ( k ) i ) .

Original: Then f ( q ( w ( k ) i )) <
Revised: Then f ( q ( w ( k ) i )) <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i ) − f ( y ) if q ( w ( k ) i ) is the right endpoint of R
Revised: → q ( w ( k ) i ) − f ( y ) if q ( w ( k ) i ) is the right endpoint of R

Original: m i for 1 ≤ m i < M , and f ( q ( w ( k ) i ))
Revised: m i for 1 ≤ m i < M , and f ( q ( w ( k ) i ))

Original: <
Revised: <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i )
Revised: → q ( w ( k ) i )

Original: + f ( y ) if q ( w ( k ) i ) is the left endpoint of R
Revised: + f ( y ) if q ( w ( k ) i ) is the left endpoint of R

Original: m i for 1 <
Revised: m i for 1 <

Original: m i ≤ M .
Revised: m i ≤ M .

Original: Proof of Lemma B.4 .
Revised: Proof of Lemma B.4 .

Original: We prove the case when q ( w ( k ) i ) is the right endpoint of R m i .
Revised: We prove the case when q ( w ( k ) i ) is the right endpoint of R m i .

Original: The case when q ( w ( k ) i ) is the left endpoint of R m i can be proved in a similar manner.
Revised: The case when q ( w ( k ) i ) is the left endpoint of R m i can be proved in a similar manner.

Original: Suppose the claimed result f ( q ( w ( k ) i ))
Revised: Suppose the claimed result f ( q ( w ( k ) i ))

Original: <
Revised: <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i ) − f ( y ) does not hold, so we must have lim y
Revised: → q ( w ( k ) i ) − f ( y ) does not hold, so we must have lim y

Original: → q ( w ( k ) i )
Revised: → q ( w ( k ) i )

Original: + f ( y ) >
Revised: + f ( y ) >

Original: f ( q ( w ( k ) i )) .
Revised: f ( q ( w ( k ) i )) .

Original: In this case, x ( k +1) i =
Revised: In this case, x ( k +1) i =

Original: z ( k +1)
Revised: z ( k +1)

Original: i , and z ( k +1) i ∈ ( q ( w ( k ) i ) , ∞ ) .
Revised: i , and z ( k +1) i ∈ ( q ( w ( k ) i ) , ∞ ) .

Original: Define p ( v ) :
Revised: Define p ( v ) :

Original: = 12 s (cid:0) v − (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i (cid:1) 2 + f m i ( v ) .
Revised: = 12 s (cid:0) v − (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i (cid:1) 2 + f m i ( v ) .

Original: We have z ( k +1)
Revised: We have z ( k +1)

Original: i = prox
Revised: i = prox

Original: [Equation]
Revised: [Equation]

Original: (cid:16)(cid:104) w ( k ) − s ∇ g ( w ( k ) (cid:105) i (cid:17) .
Revised: (cid:16)(cid:104) w ( k ) [Equation] s ∇ g ( w ( k ) (cid:105) i (cid:17) .

Original: The optimality of z ( k +1)
Revised: The optimality of z ( k +1)

Original: i by Fermat’s rule indicates that z ( k +1) i
Revised: i by Fermat’s rule indicates that z ( k +1) i

Original: = (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i , and p ( z ( k +1) i ) ≤ p ( w ( k ) i ) .
Revised: = (cid:2) w ( k ) − s ∇ g ( w ( k ) (cid:3) i , and p ( z ( k +1) i ) ≤ p ( w ( k ) i ) .

Original: We have w ( k ) i 
Revised: We have w ( k ) i ∈

Original: [Equation]
Revised: [Equation]

Original:  to 
Revised: due to s <

Original: [Equation]
Revised: [Equation]

Revised: G .

Original: On the other hand, we have
Revised: On the other hand, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (35) follows because .
Revised: (34) follows because .

Original: It follows by (34) and (35) that
Revised: It follows by (33) and (34) that

Original: [Equation]
Revised: [Equation]

Original: where 1 ⃝ follows from s <
Revised: where 1 ⃝ follows from s <

Revised: J F 0 G

Original: [Equation]
Revised: + G 2 / 2 .

Original: This contradiction shows that we must have lim y
Revised: This contradiction shows that we must have lim y

Original: → q ( w ( k ) i )
Revised: → q ( w ( k ) i )

Original: + f ( y ) ≤ f ( q ( w ( k ) i )) .
Revised: + f ( y ) ≤ f ( q ( w ( k ) i )) .

Original: Since f is not continuous at q ( w ( k ) i ) , we must have f ( q ( w ( k ) i ))
Revised: Since f is not continuous at q ( w ( k ) i ) , we must have f ( q ( w ( k ) i ))

Original: <
Revised: <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i ) − f ( y ) .
Revised: → q ( w ( k ) i ) − f ( y ) .

Original: Proof of Lemma B.3 .
Revised: Proof of Lemma B.3 .

Original: According to Lemma B.4, the following claim holds: when q ( w ( k ) i ) is a right endpoint of R
Revised: According to Lemma B.4, the following claim holds: when q ( w ( k ) i ) is a right endpoint of R

Original: m i , then f ( q ( w ( k ) i )) <
Revised: m i , then f ( q ( w ( k ) i )) <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of
Revised: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of

Original: R m i , then f ( q ( w ( k ) i )) <
Revised: R m i , then f ( q ( w ( k ) i )) <

Original: lim y
Revised: lim y

Original: → q ( w ( k ) i )
Revised: → q ( w ( k ) i )

Original: + f ( y ) .
Revised: + f ( y ) .

Original: Let q = q ( w ( k ) i ) .
Revised: Let q = q ( w ( k ) i ) .

Original: Without loss of generality, we assume x ( k +1)
Revised: Without loss of generality, we assume x ( k +1)

Original: i ∈
Revised: i ∈

Original: R
Revised: R

Original: + m i .
Revised: + m i .

Original: The case that x ( k +1) i ∈
Revised: The case that x ( k +1) i ∈

Original: R − m i can be proved in a similar manner.
Revised: R − m i can be proved in a similar manner.

Original: We first consider the case that R m i +1 is not a single-point set, that is, R m i +1 ̸ = { q } .
Revised: We first consider the case that R m i +1 is not a single-point set, that is, R m i +1 ̸ = { q } .

Original: According to the definition of surrogate function (5), we have
Revised: According to the definition of surrogate function (5), we have

Original: [Equation]
Revised: [Equation]

Original: where v − =
Revised: where v − =

Original: lim x → q − f ′ ( x ) .
Revised: lim x → q − f ′ ( x ) .

Original: If x ( k +1) i ̸
Revised: If x ( k +1) i ̸

Original: = q , applying the argument in (25) in the proof of
Revised: = q , applying the argument in (24) in the proof of

Original: Lemma B.2, s <
Revised: Lemma B.2, s <

Original: s 0 G + F 0 guarantees that x ( k +1) i ∈
Revised: s 0 G + F 0 guarantees that x ( k +1) i ∈

Original: ( q, q + s 0 ) .
Revised: ( q, q + s 0 ) .

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: when x ∈ ( q, x ( k +1) i ) .
Revised: when x ∈ ( q, x ( k +1) i ) .

Original: Letting x
Revised: Letting x

Original: → q in (38), we have
Revised: → q in (37), we have

Original: [Equation]
Revised: [Equation]

Original: where v + = lim x → q + f ′ ( x ) .
Revised: where v + = lim x → q + f ′ ( x ) .

Original: In addition,
Revised: In addition,

Original: It follows by (39) and (25) in the proof of Lemma B.2 that
Revised: It follows by (38) and (24) in the proof of Lemma B.2 that

Original: [Equation]
Revised: [Equation]

Original: We note that (40) holds for all x ( k +1) i ∈
Revised: We note that (39) holds for all x ( k +1) i ∈

Original: [ q, q + s 0 )
Revised: [ q, q + s 0 )

Original: Moreover, it follows by (37) and (25) in the proof of Lemma B.2 that f m i 
Revised: Moreover, it follows by (36) and (24) in the proof of Lemma B.2 that f m i [Equation] x

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Combining (40) and (41), we have f ( x ( k +1)
Revised: Combining (39) and (40), we have f ( x ( k + [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: If R P ( z ( k +1) i ) = { q } , then x ( k +1) i = q and f ( x ( k +1) i ) = f ( q ) .
Revised: If R P ( z ( k +1) i ) = { q } , then x ( k +1) i = q and f ( x ( k +1) i ) = f ( q ) .

Original: By the fact that
Revised: By the fact that

Original: [Equation]
Revised: [Equation]

Original: and (cid:12)(cid:12)(cid:12) z ( k +1)
Revised: and (cid:12)(cid:12)(cid:12) z ( k +1)

Original: i−
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)  z ( k +1)
Revised: (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) z ( k +1)

Revised: i − w ( k ) i (cid:12)(cid:12)(cid:12) , following the argument similar to (40) we have f

Original: [Equation]
Revised: [Equation]

Original: − w ( k ) i (cid:12)(cid:12)(cid:12) , following the argument similar to (41) we have f m i ( z( k +
Revised: z

Original: [Equation]
Revised: [Equation]

Original: i
Revised: k

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by combining (42) and (43).
Revised: The proof is completed by combining (41) and (42).

Original: B. P ROOF OF L EMMA 4. 
Revised: B. P ROOF OF L EMMA 4.

Original: Proof of Lemma 4.1 .
Revised: Proof of Lemma 4.1 .

Original: We split [ d ] into three disjoint subsets, [ d ] = S 1 ∪ S 2 ∪ S 3 , and the three subsets are defined by
Revised: We split [ d ] into three disjoint subsets, [ d ] = S 1 ∪ S 2 ∪ S 3 , and the three subsets are defined by

Original: S 1 := (cid:110) i ∈
Revised: [Equation]

Original: [ d ]: P ( x ( k +1) i ) ̸
Revised: [Equation] [Equation]

Original: = P ( x ( k ) i ) , f is continuous at q ( w ( k ) i ) (cid:111)
Revised: ) , f is continuous at q [Equation] ( k ) [Equation]

Original: [Equation]
Revised: [Equation]

Original:  is not 
Revised: f is not continuous at

Original: [Equation]
Revised: [Equation]

Original: Let m i
Revised: Let m i

Original: =
Revised: =

Original: P ( x ( k ) i ) .
Revised: P ( x ( k ) i ) .

Original: According to Lemma B.2, for all i ∈ S 1 such that d i, 1 ≥ w 0 d
Revised: According to Lemma B.2, for all i ∈ S 1 such that d i, 1 ≥ w 0 d

Original: i, 0 , we have
Revised: i, 0 , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In addition, for all i ∈ S 1 , we have
Revised: In addition, for all i ∈ S 1 , we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma B.4, S 2 satisfies the condition that for all i ∈ S 2 , when q ( w ( k ) i ) is a right endpoint of R m i , then f ( q ( w ( k ) i )) <

Original: lim y

Original: → q ( w ( k ) i ) − f ( y ) ; when q ( w ( k ) i ) is a left endpoint of R

Original: m i , then f ( q ( w ( k ) i )) <

Original: lim y

Original: → q ( w ( k ) i )

Original: + f ( y ) .

Original: By Lemma B.3,[Equation]
Revised: According to Lemma B.3,

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ S 2 .
Revised: for all i ∈ S 2 .

Original: For all i ∈ S 3 , we have
Revised: For all i ∈ S 3 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The Negative-Curvature-Exploitation algorithm described in Algorithm 2 guarantees that when
Revised: The Negative-Curvature-Exploitation algorithm described in Algorithm 2 guarantees that when

Original: P ( x ( k +1) )
Revised: P ( x ( k +1) )

Original: ̸
Revised: ̸

Original: = P ( x ( k ) ) , there exists at lease one i ∈ S 1 ∪ S 2 such that (45) or (47) holds.
Revised: = P ( x ( k ) ) , there exists at lease one i ∈ S 1 ∪ S 2 such that (44) or (46) holds.

Original: It follows by (45)-(48) and the above argument that 
Revised: It follows by (44)-(47) and the above argument that [Equation] i [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: − κ

Revised: [Equation]

Revised: Let ¯ S

Revised: k

Revised: +1 :

Revised: = (cid:110) i ∈ [ d ]: x ( k +1)

Revised: i ̸

Revised: = z ( k +1)

Revised: i (cid:111) .

Revised: It can be verified by Algorithm 2 that ¯ S k +1 ⊆

Revised: S 2 .

Revised: If ¯ S k +1 = ∅ , then x ( k +1)

Revised: =

Revised: z ( k +1) .

Original: It follows from (49) that
Revised: It follows from (48) that

Original: [Equation]
Revised: [Equation]

Revised: If ¯ S k +1 ̸ = ∅ , then it is possible that x ( k +1) ̸

Revised: = z ( k +1) .

Revised: To handle the case that x ( k +1) ̸

Revised: = z ( k +1) , we first bound (cid:13)(cid:13) x

Revised: ( k +1)

Revised: − z ( k +1)

Revised: (cid:13)(cid:13) 2 .

Revised: Define h m ( x ) :=

Revised: (cid:80) di

Revised: =1 f m i ( x i ) for m ∈ N d and m i ∈

Revised: m i =

Revised: P ( x ( k ) i ) for all i ∈ [ d ] .

Revised: By the optimality of z ( k +1) , we have

Revised: [Equation]

Revised: It follows from (50) that (cid:13)(cid:13) w ( k ) − z

Revised: ( k +1)

Revised: (cid:13)(cid:13) 2 ≤ s ( G + √ dF 0 ) , and (cid:13)(cid:13) x ( k +1)

Original: {+− z ( k +1) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13) w ( k ) − z k +1 (cid:13)(cid:13) 
+}
Revised: {+− z ( k +1) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13) w ( k ) − z k +1 (cid:13)(cid:13) 
+}

Revised: ≤ s ( G + √ dF 0 ) .

Revised: We then bound (cid:12)(cid:12) g

Revised: ( x ( k +1) )

Revised: − g ( z ( k +1) ) (cid:12)(cid:12) by (cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: k +1) )

Revised: [Equation]

Revised: z

Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: ≤

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: Here ζ in 1 ⃝ lies in the line segment between x ( k +1) and z ( k +1) by the mean value theorem of differentiable functions.

Revised: 2 ⃝ follows from the fact that (cid:13)(cid:13) ζ − w ( k ) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13) w ( k ) − z

Revised: ( k +1)

Revised: (cid:13)(cid:13) 2 ≤ s ( G + √ dF 0 ) .

Revised: We then have

Revised: [Equation]

Original: The PPGD algorithm guarantees that
Revised: The PPGD algorithm guarantees that

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have F ( x ( k +1) ){+It follows from (49), (52), and (53) that [Equation] Since s < − G + (cid:114) G 
 + 4 Aκ 0 G + √+}
Revised: {+It follows from (49), (52), and (53) that [Equation] Since s < − G + (cid:114) G 
 + 4 Aκ 0 G + √+}

Original: ≤ F ( x ( k ) ) −{+dF
 A , we have+} κ  , which completes the proof.
Revised: {+dF
 A , we have+} κ > 0 , which completes the proof.

Original: B. P ROOF OF L EMMA 4. 
Revised: B. P ROOF OF L EMMA 4.

Original: Proof of Lemma 4.2 .
Revised: Proof of Lemma 4.2 .

Original: The PPGD algorithm described in Algorithm 1 ensures that F ( x ( k ) ) ≤
Revised: The PPGD algorithm described in Algorithm 1 ensures that F ( x ( k ) ) ≤

Original: F ( x ( k − 1) ) for k = 1 , and (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: F ( x ( k − 1) ) for k = 1 , and (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 ≤ G for k = 1 .
Revised: (cid:13)(cid:13) 2 ≤ G for k = 1 .

Original: Suppose that F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) ) (cid:13)(cid:13) 2 ≤ G hold for all 1 ≤ k ≤ k ′ with k ′ ≥ 1 .
Revised: Suppose that F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) ) (cid:13)(cid:13) 2 ≤ G hold for all 1 ≤ k ≤ k ′ with k ′ ≥ 1 .

Original: With the chosen step size and the proof of Lemma 4.1, we have F ( x ( k ′ +1) )
Revised: With the chosen step size and the proof of Lemma 4.1, we have F ( x ( k ′ +1) )

Original: ≤ F ( x ( k ′ ) ) .
Revised: ≤ F ( x ( k ′ ) ) .

Original: This indicates that x ( k ′ +1)
Revised: This indicates that x ( k ′ +1)

Original: ∈
Revised: ∈

Original: L and w ( k ′ +1)
Revised: L and w ( k ′ +1)

Original: ∈
Revised: ∈

Original: L R 0 , so (cid:13)(cid:13)(cid:13) w ( k ′ +1) (cid:13)(cid:13)(cid:13) 2 ≤ G .
Revised: L R 0 , so (cid:13)(cid:13)(cid:13) w ( k ′ +1) (cid:13)(cid:13)(cid:13) 2 ≤ G .

Original: It follows by induction that
Revised: It follows by induction that

Original: F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: F ( x ( k ) ) ≤ F ( x ( k − 1) ) and (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 ≤ G hold for all k ≥ 1 .
Revised: (cid:13)(cid:13) 2 ≤ G hold for all k ≥ 1 .

Original: B. P ROOF OF T HEOREM4.
Revised: B. P ROOF OF T HEOREM

Original: Proof of Theorem 4.3 .
Revised: Proof of Theorem 4.3 .

Original: By Lemma 4.2, (cid:13)(cid:13) ∇ g ( w ( k ) )
Revised: By Lemma 4.2, (cid:13)(cid:13) ∇ g ( w ( k ) )

Original: (cid:13)(cid:13) 2 ≤ G holds for all k ≥ 1 .
Revised: (cid:13)(cid:13) 2 ≤ G holds for all k ≥ 1 .

Original: Then the conclusion of this theorem directly follows from Lemma 4.1.
Revised: Then the conclusion of this theorem directly follows from Lemma 4.1.

Original: B. P ROOF OF T HEOREM 4. 
Revised: B. P ROOF OF T HEOREM 4.

Original: The following lemma is crucial in the proof of Theorem 4.4.
Revised: The following lemma is crucial in the proof of Theorem 4.4.

Original: It shows that after sufficient iterations, all the coordinates of x ( k ) belong to the same
Revised: It shows that after sufficient iterations, all the coordinates of x ( k ) belong to the same

Original: the convex pieces indexed by m ∗ , that is,
Revised: the convex pieces indexed by m ∗ , that is,

Original: P ( x ( k ) )
Revised: P ( x ( k ) )

Original: = m ∗ .
Revised: = m ∗ .

Original: Moreover, the objective value F ( x ( k ) ) is not greater than the surrogate objective value F m ∗ ( z ( k ) )
Revised: Moreover, the objective value F ( x ( k ) ) is not greater than the surrogate objective value F m ∗ ( z ( k ) )

Original: =
Revised: =

Original: g ( z ( k ) )
Revised: g ( z ( k ) )

Original: + (cid:80) di =1 f m ∗ i ( z ( k ) i ) .
Revised: + (cid:80) di =1 f m ∗ i ( z ( k ) i ) .

Original: Lemma B.5.
Revised: Lemma B.5.

Original: Suppose Assumption 1 and Assumption 2 hold, and s < min (cid:110) s 1 , ε 

Revised: Suppose Assumption 1 and Assumption 2 hold, and s < min (cid:110) s 1 , ε

Original: L g ( G + √ dF 0 ) (cid:111) .
Revised: L g ( G + √ dF 0 ) (cid:111) .

Original: Then there must exists a finite  k ∈ N such that P ( x ( k ) )
Revised: Then there must exists a finite ¯ k ∈ N such that P ( x ( k ) )

Original: =
Revised: =

Original: m ∗ ∈ N d and F ( x ( k ) )
Revised: m ∗ ∈ N d and F ( x ( k ) )

Original: ≤ F m ∗ ( z ( k ) ) for all k ≥  k .
Revised: ≤ F m ∗ ( z ( k ) ) for all k ≥ ¯ k .

Original: Proof of Lemma B.5 .
Revised: Proof of Lemma B.5 .

Original: According to Theorem 4.3, when P ( x ( k +1) ) ̸
Revised: According to Theorem 4.3, when P ( x ( k +1) ) ̸

Original: = P ( x ( k ) ) , F ( x ( k +1) ) ≤
Revised: = P ( x ( k ) ) , F ( x ( k +1) ) ≤

Original: F ( x ( k ) ) − κ .
Revised: F ( x ( k ) ) − κ .

Original: Because inf x ∈ R d F ( x ) >
Revised: Because inf x ∈ R d F ( x ) >

Original: −∞ , we can only have finite number of k ’s such that
Revised: −∞ , we can only have finite number of k ’s such that

Original: F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ .
Revised: F ( x ( k +1) ) ≤ F ( x ( k ) ) − κ .

Original: As a result, there must exists a finite k 1 ∈ N such that P ( x ( k ) )
Revised: As a result, there must exists a finite k 1 ∈ N such that P ( x ( k ) )

Original: =
Revised: =

Original: m ∗ ∈ N d for all k ≥ k 1 .
Revised: m ∗ ∈ N d for all k ≥ k 1 .

Original: We now prove that there exists a finite k 2 > k 1 such that F ( x ( k ) )
Revised: We now prove that there exists a finite k 2 > k 1 such that F ( x ( k ) )

Original: ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 .
Revised: ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 .

Original: Suppose this is not the case and there are infinitely many k ’s such that F ( x ( k ) )
Revised: Suppose this is not the case and there are infinitely many k ’s such that F ( x ( k ) )

Original: >
Revised: >

Original: F m ∗ ( z ( k ) ) and k ≥ k 1 .
Revised: F m ∗ ( z ( k ) ) and k ≥ k 1 .

Original: It follows that there exists a sequence { m k } k ≥ 1 such that F ( x ( m k ) ) >
Revised: It follows that there exists a sequence { m k } k ≥ 1 such that F ( x ( m k ) ) >

Original: F m ∗ ( z ( m k ) ) with m k > k 1 for all k ≥ 1 , and lim k →∞ m k = ∞ .
Revised: F m ∗ ( z ( m k ) ) with m k > k 1 for all k ≥ 1 , and lim k →∞ m k = ∞ .

Original: According to the PPGD algorithm and the Negative-CurvatureExploitation algorithm described in Algorithm 1 and Algorithm 2, this is possible only if for all k ≥ 1 , there exists
Revised: According to the PPGD algorithm and the Negative-CurvatureExploitation algorithm described in Algorithm 1 and Algorithm 2, this is possible only if for all k ≥ 1 , there exists

Original: i ∈
Revised: i ∈

Original: [ d ] such that P ( z ( m k ) i
Revised: [ d ] such that P ( z ( m k ) i

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: ( m k ) i − q (

Original: [Equation]
Revised: [Equation]

Original: m k 
Revised: k )

Original: [Equation] 1) i ) (cid:12)(cid:12)(cid:12) < w 0 (cid:12)(cid:12)(cid:12) z ( m k ) i − w ( m k − 1) i (cid:12)(cid:12)(cid:12) , x ( m k ) =
Revised: =

Original: x ( m k − 1) , and f is continuous at q ( w ( m k − 1) i ) .
Revised: x ( m k − 1) , and f is continuous at q ( w ( m k − 1) i ) .

Original: We consider the case that q ( w ( m k − 1) i ) =
Revised: We consider the case that q ( w ( m k − 1) i ) =

Original: q m ∗ i is the right endpoint of R m ∗ i .
Revised: q m ∗ i is the right endpoint of R m ∗ i .

Original: The case that q ( w ( m k − 1) i ) =
Revised: The case that q ( w ( m k − 1) i ) =

Original: q m ∗ i − 1 is the left endpoint of R m ∗ i can be proved in a similar manner.
Revised: q m ∗ i − 1 is the left endpoint of R m ∗ i can be proved in a similar manner.

Original: In this case, P ( z ( m k ) i )
Revised: In this case, P ( z ( m k ) i )

Original: = m
Revised: = m

Original: ∗ i + 1 .
Revised: ∗ i + 1 .

Original: Let ˜ w ∈ R d with ˜ w j = w ( m k − 1) j for all j ̸
Revised: Let ˜ w ∈ R d with ˜ w j = w ( m k − 1) j for all j ̸

Original: = i , and ˜ w i = q m ∗ i .
Revised: = i , and ˜ w i = q m ∗ i .

Original: By the definition of ˜ w and the same argument as (30) in the proof of Lemma B.2, we have z ( m k [Equation]
Revised: By the definition of ˜ w and the same argument as (29) in the proof of Lemma B.2, we have z ( m k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Because (cid:13)(cid:13) ∇ g ( w ( m k − 1) )
Revised: Because (cid:13)(cid:13) ∇ g ( w ( m k − 1) )

Original: − ∇ g ( ˜ w ) (cid:13)(cid:13) 2 ≤ sL g (1 − w 0 )
Revised: − ∇ g ( ˜ w ) (cid:13)(cid:13) 2 ≤ sL g (1 − w 0 )

Original: ( G + F 0 ) and s <
Revised: ( G + F 0 ) and s <

Original: ε 

Revised: ε

Original: L g (1 − w 0 )( G + F 0 ) , we must have [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 due to Assumption 2 and the fact that z ( m k ) i − w ( m k − 1) i > 0 .
Revised: L g (1 − w 0 )( G + F 0 ) , we must have [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 due to Assumption 2 and the fact that z ( m k ) i − w ( m k − 1) i > 0 .

Original: With k → ∞ , we have t mk − 1 t mk → 1 and
Revised: With k → ∞ , we have t mk − 1 t mk → 1 and

Original: [Equation]
Revised: [Equation]

Original: 51) and it follows that w ( m k ) 
Revised: and it follows that w ( m k ) =

Original: [Equation]
Revised: P x ( mk ) ,R 0 ( u ( m k ) )

Original: P
Revised: k

Original: [Equation]
Revised: [Equation]

Original:  ,R 0 ( z ( m k ) ) , and we have (cid:104)
Revised: → P x ( mk ) ,R 0 ( z ( m k ) ) , and we have (cid:104)

Original: P x ( mk ) ,R 0 ( z ( m k ) )
Revised: P x ( mk ) ,R 0 ( z ( m k ) )

Original: (cid:105) i = (cid:104) P x ( mk − 1) ,R 0 ( z ( m k ) )
Revised: (cid:105) i = (cid:104) P x ( mk − 1) ,R 0 ( z ( m k ) )

Original: (cid:105) i =
Revised: (cid:105) i =

Original: q m ∗ i .
Revised: q m ∗ i .

Original: By the updating rule (9) in the PPGD algorithm and the first equality in (50), we have
Revised: By the updating rule (9) in the PPGD algorithm and the first equality in (54), we have

Original: [Equation]
Revised: [Equation]

Original: It follows from (52) that
Revised: It follows from (56) that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here 1 ⃝ follows from (cid:104) P x ( mk ) ,R 0 ( z ( m k ) )
Revised: Here 1 ⃝ follows from (cid:104) P x ( mk ) ,R 0 ( z ( m k ) )

Original: (cid:105) i = ˜
Revised: (cid:105) i = ˜

Original: w i = q m ∗ i , w ( m k − 1) =
Revised: w i = q m ∗ i , w ( m k − 1) =

Original: P x ( mk ) ,R 0 ( w ( m k − 1) ) , and x ( m k )
Revised: P x ( mk ) ,R 0 ( w ( m k − 1) ) , and x ( m k )

Original: =
Revised: =

Original: x ( m k − 1) .
Revised: x ( m k − 1) .

Original: 2 ⃝ follows from the contraction property of projection onto a closed convex set.
Revised: 2 ⃝ follows from the contraction property of projection onto a closed convex set.

Original: Combining (53) and the fact that [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 , we have
Revised: Combining (57) and the fact that [ ∇ g ( ˜ w )] i + v − ≤ − ε 0 , we have

Original: [Equation]
Revised: [Equation]

Original: due to s <
Revised: due to s <

Original: [Equation]
Revised: [Equation]

Original: Now noting that w ( m k ) =
Revised: Now noting that w ( m k ) =

Original: P x ( mk ) ,R 0 ( u ( m k ) )
Revised: P x ( mk ) ,R 0 ( u ( m k ) )

Original: k →∞ → P x ( mk ) ,R 0 ( z ( m k ) ) , it follows from (54) and the smoothness of ∇ g that there exists a large enough k ′ such that when k ≥ k ′ ,
Revised: k →∞ → P x ( mk ) ,R 0 ( z ( m k ) ) , it follows from (58) and the smoothness of ∇ g that there exists a large enough k ′ such that when k ≥ k ′ ,

Original: [Equation]
Revised: [Equation]

Original: We now analyze the next iterate z ( m k +1) i .
Revised: We now analyze the next iterate z ( m k +1) i .

Original: By the updating rule z ( m k +1) i
Revised: By the updating rule z ( m k +1) i

Original: = prox sf P ( x ( mk )i )
Revised: = prox sf P ( x ( mk )

Original: (cid:0)(cid:2) w ( m k ) − s ∇ g ( w ( m k ) ) (cid:3) i(cid:1) , we must have
Revised: i

Original: [Equation]
Revised: [Equation]

Original: In the iteration m k + 1 , we have
Revised: w

Original: [Equation]
Revised: [Equation]

Original: i
Revised: s ∇ g [Equation]

Original: [Equation]
Revised: [Equation]

Original: and w ( m k ) i
Revised: i (cid:1) , we must have [Equation] In the iteration m k + 1 , we have [Equation]

Original:  k →∞ → q ( w ( m k ) i ) .
Revised: [Equation] and w ( m k ) i k →∞ → q ( w ( m k ) i ) .

Original: Therefore, with sufficiently large k ′ , we have d i, 1 ≥ w 0 d
Revised: Therefore, with sufficiently large k ′ , we have d i, 1 ≥ w 0 d

Original: i
Revised: i

Original: 0 due to d i, 1 k →∞ → d i, 0 .
Revised: 0 due to d i, 1 k →∞ → d i, 0 .

Original: It follows from the Negative-Curvature-Exploitation algorithm described in Algorithm 2 that P ( x ( m k +1) i )
Revised: It follows from the Negative-Curvature-Exploitation algorithm described in Algorithm 2 that P ( x ( m k +1) i )

Original: = m
Revised: = m

Original: ∗ i + 1 , which contradict the fact that P ( x ( m k ) )
Revised: ∗ i + 1 , which contradict the fact that P ( x ( m k ) )

Original: =
Revised: =

Original: m ∗ ∈ N d for all k ≥ k 1 .
Revised: m ∗ ∈ N d for all k ≥ k 1 .

Original: This contradiction shows that there exists a finite k 2 > k 1 such that F ( x ( k ) ) ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 . 
Revised: This contradiction shows that there exists a finite k 2 > k 1 such that F ( x ( k ) ) ≤ F m ∗ ( z ( k ) ) for all k ≥ k 2 . Setting ¯

Original: Setting k = k 2 completes the proof.
Revised: k = k 2 completes the proof.

Original: Proof of Theorem 4.4 .
Revised: Proof of Theorem 4.4 .

Original: According to Lemma B.5, there exists a finite k 1 > 1 such that P ( x ( k ) )
Revised: According to Lemma B.5, there exists a finite k 1 > 1 such that P ( x ( k ) )

Original: =
Revised: =

Original: m ∗ ∈ N d and F ( x ( k ) )
Revised: m ∗ ∈ N d and F ( x ( k ) )

Original: ≤ F m ∗ ( z ( k ) ) for all k ≥ k 1 .
Revised: ≤ F m ∗ ( z ( k ) ) for all k ≥ k 1 .

Original: Furthermore, the proof of Lemma 4.2 shows that the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by PPGD satisfies (cid:8) x ( k ) (cid:9) k ≥ 1 ⊆
Revised: Furthermore, the proof of Lemma 4.2 shows that the sequence (cid:8) x ( k ) (cid:9) k ≥ 1 generated by PPGD satisfies (cid:8) x ( k ) (cid:9) k ≥ 1 ⊆

Original: L which is a compact set, so there exists at least one limit point for (cid:8) x
Revised: L which is a compact set, so there exists at least one limit point for (cid:8) x

Original: ( k ) (cid:9) , and Ω ̸ = ∅ .
Revised: ( k ) (cid:9) , and Ω ̸ = ∅ .

Original: Define h m ( x ) :=
Revised: Define h m ( x ) :=

Original: (cid:80) di
Revised: (cid:80) di

Original: =1 f m i ( x i ) for m ∈ N d and m i ∈
Revised: =1 f m i ( x i ) for m ∈ N d and m i ∈

Original: [ M ] for all i ∈ [ d ] , and F m := g + h m .
Revised: [ M ] for all i ∈ [ d ] , and F m := g + h m .

Original: Note that for all i ∈ [ d ] , f m ∗ i is convex except for the third case in (5) or (6).
Revised: Note that for all i ∈ [ d ] , f m ∗ i is convex except for the third case in (5) or (6).

Original: In such a case, either event 1 ⃝ : f m ∗ i ( x ) =
Revised: In such a case, either event 1 ⃝ : f m ∗ i ( x ) =

Original: lim y
Revised: lim y

Original: → q + m ∗ i f ( y ) for x > q m ∗ i and f ( q m ∗ i ) <
Revised: → q + m ∗ i f ( y ) for x > q m ∗ i and f ( q m ∗ i ) <

Original: lim y
Revised: lim y

Original: → q + m ∗ i f ( y ) , or event 2 ⃝ : f m ∗ i ( x ) =
Revised: → q + m ∗ i f ( y ) , or event 2 ⃝ : f m ∗ i ( x ) =

Original: lim y
Revised: lim y

Original: → q − m ∗ i − 1 f ( y ) for x < q m ∗ i − 1 and f ( q m ∗ i − 1 ) <
Revised: → q − m ∗ i − 1 f ( y ) for x < q m ∗ i − 1 and f ( q m ∗ i − 1 ) <

Original: lim y
Revised: lim y

Original: → q − m ∗ i − 1 f ( y ) .
Revised: → q − m ∗ i − 1 f ( y ) .

Original: It follows by the proof of Lemma B.4 that all the sequences (cid:110) x ( k ) i (cid:111) k ≥ k 0 and (cid:110)
Revised: It follows by the proof of Lemma B.4 that all the sequences (cid:110) x ( k ) i (cid:111) k ≥ k 1 and (cid:110)

Original: z ([-k ) i (cid:111) k ≥ k 
 +1 satisfies (cid:110) x ( k ) i (cid:111) k ≥ k 0 ⊆-]
Revised: z ([-k ) i (cid:111) k ≥ k 
 +1 satisfies (cid:110) x ( k ) i (cid:111) k ≥ k 0 ⊆-]

Original: ( −∞ , q m ∗ i ] and (cid:110) z ( k ) i (cid:111) k ≥ k 0 +1 ⊆ ( −∞ , q m ∗ i ] if event 1 ⃝ happens, and (cid:110) x
Revised: [Table]

Original:  ( k ) i (cid:111)
Revised: (cid:110) x ( k ) i (cid:111)

Original: k ≥ k 0 ⊆
Revised: k ≥ k 1 ⊆

Original: [ q m ∗ i − 1 , + ∞ ) and (cid:110)
Revised: [ q m ∗ i − 1 , + ∞ ) and (cid:110)

Original: z ( k ) i (cid:111)
Revised: z ( k ) i (cid:111)

Original: k ≥ k 0 +1 ⊆
Revised: k ≥ k 1 +1 ⊆

Original: [ q m ∗ i − 1 , + ∞ ) if event 2 ⃝ happens.
Revised: [ q m ∗ i − 1 , + ∞ ) if event 2 ⃝ happens.

Original: For all i ∈ [ d ] , define R  i as the region over which f m ∗ i is convex.
Revised: For all i ∈ [ d ] , define R ∗ i as the region over which f m ∗ i is convex.

Original: It is clear that R  i
Revised: It is clear that R ∗ i

Original: =
Revised: =

Original: R if event 1 ⃝ and 2 ⃝ do not happen for f m ∗ i .
Revised: R if event 1 ⃝ and 2 ⃝ do not happen for f m ∗ i .

Original: If only event 1 ⃝ happens for f m ∗ i , then R  i =
Revised: If only event 1 ⃝ happens for f m ∗ i , then R ∗ i =

Original: ( −∞ , q m ∗ i ] .
Revised: ( −∞ , q m ∗ i ] .

Original: If only event 2 ⃝ happens, R  i =
Revised: If only event 2 ⃝ happens, R ∗ i =

Original: [ q m ∗ i − 1 , + ∞ ) .
Revised: [ q m ∗ i − 1 , + ∞ ) .

Original: If both event 1 ⃝ and 2 ⃝ happen, then R  i =
Revised: If both event 1 ⃝ and 2 ⃝ happen, then R ∗ i =

Original: [ q m ∗ i − 1 , q m ∗ i ] .
Revised: [ q m ∗ i − 1 , q m ∗ i ] .

Original: Let ¯ x ∈ R
Revised: Let ¯ x ∈ R

Original: d be an optimal solution to
Revised: d be an optimal solution to

Original: [Equation]
Revised: [Equation]

Revised: The existence of ¯ x is proved as follows.

Revised: First, it can be verified that the convex surrogate function F m ∗ is continuous over the convex region R ∗ , and R ∗ is a closed set in the usual

Revised: Euclidean topology.

Revised: By the coercivity assumption and the continuity of F m ∗ , the set R 0 :=

Revised: (cid:8) x | F m ∗ ( x ) ≤ F m ∗ ( x (0) ) , x ∈ R ∗ (cid:9) ( x (0) is the initialization point of PPGD) is bounded and closed, so the set R 0 ∩ R ∗ is bounded and closed thus a compact set.

Revised: Therefore, the minimizer ¯ x is by its definition a minimizer of F m ∗ over R ∗ , which is also a minimizer of a continuous function F m ∗ over the compact set R 0 ∩ R ∗ .

Original: Note that the existenceof ¯ x is guaranteed by the fact that F m ∗ is convex on a convex set (cid:8) x
Revised: The existence ¯ x follows by the existence of a minimizer of a continuous function over a compact set. Roadmap of the proof. We prove this theorem in three steps. In step 1 , it is proved that there exists a finite k 0 ≥ k 1 such that for all k > k 0 ,

Original: [Equation]
Revised: [Equation]

Revised: Noting that F ( x ( k ) )

Revised: =

Revised: F m ∗ ( x ( k ) ) ≥ F m ∗ (¯ x ) by the optimality of ¯ x , the above inequality combined with the monotone nonincreasing of (cid:8) F ( x ( k ) ) (cid:9) indicate that F ( x ( k ) )

Revised: ↓ F m ∗ (¯ x ) .

Revised: In step 2 , we will prove that F ( x ′ )

Revised: =

Revised: F m ∗ (¯ x ) for any limit point x ′ ∈ Ω .

Revised: According to the definition of F m ∗ and the optimality of ¯ x , it follows that F ( x ′ ) is a local minimum of F and a global minimum of F m ∗ over R ∗ .

Revised: In step 3 , we will prove that any limit point x ′ ∈ Ω is a critical point of F under a mild condition, following the argument in step 2 .

Revised: Step 1.

Original: We now consider k ≥ k 1 in the sequel.
Revised: We now consider k ≥ k 1 in the sequel.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [ d ] , v ∈ R i , and all p ∈ ˜
Revised: for all i ∈ [ d ] , v ∈ R i , and all p ∈ ˜

Original: ∂f ( z ( k +1) i ) .
Revised: ∂f ( z ( k +1) i ) .

Original: It follows that if v ∈ R d and v i ∈ R  i for all i ∈
Revised: It follows that if v ∈ R d and v i ∈ R ∗ i for all i ∈

Original: [ d ] , then
Revised: [ d ] , then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for all p ∈ ˜
Revised: for all p ∈ ˜

Original: ∂h m ∗i .
Revised: ∂h m ∗ .

Original: Because z ( k +1)
Revised: Because z ( k +1)

Original: i = prox
Revised: i [Equation] g

Original: [Equation]
Revised: [Equation]

Original:  in (9) of Algorithm 1, it follows by the optimality of z ( k +1)
Revised: w [Equation] k ) (cid:3) i (cid:1) in (9) of Algorithm 1, it follows by the optimality of z ( k +1)

Original: i that
Revised: i that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It follows by (58) and (59) that
Revised: It follows by (62) and (63) that

Original: [Equation]
Revised: [Equation]

Original: for any v ∈ R d such that v i ∈ R  i for all i ∈[ d ] .
Revised: for any v ∈ R d such that v i ∈ R ∗ i for all i ∈

Original:  For such v [Equation] we have
Revised: [ d ] . For such v , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here 1 ⃝ follows from (60).
Revised: Here 1 ⃝ follows from (64).

Original: Let v = x ( k ) and v = ¯ x in (61), we have
Revised: Let v = x ( k ) and v = ¯ x in (65), we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: 62) × 
Revised: (66) × ( t k − 1)+ (67), we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: k
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: 1)+
Revised: x − w

Original: [Equation]
Revised: [Equation]

Original: 63), we have
Revised: k )

Original: [Equation]
Revised: [Equation]

Original: It follows that

Original:  [Equation]
Revised: s [Equation]

Revised: z

Original:  Multiplying both sides of (65) by t k , since t 2 k − t k
Revised: [Equation] [Equation] ( k ) (cid:13)(cid:13)(cid:13) 2 2 . Multiplying both sides of (69) by t k , since t 2 k − t k

Original: = t 2 k − 1 , we have 
Revised: = t 2 k − 1 , we have t 2 k (cid:16) [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since t k ≥ k +12 for k ≥ 1 , we have t k k →∞ → = ∞ and lim k →∞ (cid:16) 1 − 1 t k (cid:17) x
Revised: Since t k ≥ k +12 for k ≥ 1 , we have t k k →∞ → = ∞ and lim k →∞ (cid:16) 1 − 1 t k (cid:17) x

Original: ( k ) + 1 t
Revised: ( k ) + 1 t

Original: k ¯ x = x ( k ) .
Revised: k ¯ x = x ( k ) .

Original: It follows that there exists a finite k 2 such that
Revised: It follows that there exists a finite k 2 such that

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ k 2 and all i ∈ [ d ] .
Revised: for all k ≥ k 2 and all i ∈ [ d ] .

Original: It follows from (67) that
Revised: It follows from (71) that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now let k ≥ k 0 :
Revised: Now let k ≥ k 0 :

Original: = max { k 1 , k 2 } , we have
Revised: = max { k 1 , k 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: where 1 ⃝ follows from (68), 2 ⃝ follows from the contraction property of projection onto a closed convex set.
Revised: where 1 ⃝ follows from (72), 2 ⃝ follows from the contraction property of projection onto a closed convex set.

Original: It follows by (66) and (69), that
Revised: It follows by (70) and (73), that

Original: [Equation]
Revised: [Equation]

Original: Define Q ( k +1)
Revised: Define Q ( k +1)

Original: =
Revised: =

Original: ( t k − 1) x ( k ) − t k z
Revised: ( t k − 1) x ( k ) − t k z

Original: ( k +1)
Revised: ( k +1)

Original: +
Revised: +

Original: ¯ x , then Q ( k ) =
Revised: ¯ x , then Q ( k ) =

Original: ( t k − 1 − 1) x ( k − 1) − t k − 1 z
Revised: ( t k − 1 − 1) x ( k − 1) − t k − 1 z

Original: ( k )
Revised: ( k )

Original: +
Revised: +

Original: ¯ x .
Revised: ¯ x .

Original: It can be verified that Q ( k ) =
Revised: It can be verified that Q ( k ) =

Original: ( t k − 1) x ( k ) − t k u ( k ) + ¯ x .
Revised: ( t k − 1) x ( k ) − t k u ( k ) + ¯ x .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Noting that F ( x ( k +1) ) ≤ F m ∗ ( z ( k +1) ) for k ≥ k 0 , it follows from the above inequality that
Revised: Noting that F ( x ( k +1) ) ≤ F m ∗ ( z ( k +1) ) for k ≥ k 0 , it follows from the above inequality that

Original: [Equation]
Revised: [Equation]

Original: Summing (71) over k = k 0 , . .
Revised: Summing (75) over k = k 0 , . .

Original: , m for m ≥ k 0 , we havet
Revised: , m for m ≥ k 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Since t k ≥ k +12 for k ≥ 1 , it follows from (72) that
Revised: Since t k ≥ k +12 for k ≥ 1 , it follows from (76) that

Original: [Equation]
Revised: [Equation]

Original: Noting that F ( x ( m +1) )
Revised: Noting that F ( x ( m +1) )

Original: =
Revised: =

Original: F m ∗ ( x ( m +1) ) , we have F ( x ( m +1) )
Revised: F m ∗ ( x ( m +1) ) , we have F ( x ( m +1) )

Original: − F m ∗ (¯ x ) ≤ [Equation] 2 U [Equation]
Revised: − F m ∗ (¯ x ) ≤ 4 ( m +1) 2 U ( k 0 ) .

Original: k

Original: [Equation]

Original: Replacing m + 1 with k , we have
Revised: Replacing m + 1 with k , we have

Original: [Equation]
Revised: [Equation]

Original: for all k > k 0 .
Revised: for all k > k 0 .

Original: Let x ′ ∈
Revised: Because F ( x ( k ) ) = F m ∗ ( x ( k ) ) ≥ F m ∗ (¯ x ) due to the optimality of ¯ x , we have F ( x ( k ) ) ↓ F m ∗ (¯ x ) as k → ∞ based on (78).

Revised: Step 2.

Revised: We now prove that any limit point x ′ ∈ Ω achieves a local minimum of F .

Revised: Let x ′ ∈

Original: Ω be an arbitrary limit point of (cid:8) x
Revised: Ω be an arbitrary limit point of (cid:8) x

Original: ( k ) (cid:9) k ≥ 1 .
Revised: ( k ) (cid:9) k ≥ 1 .

Original: By Lemma 4.2, we have F ( x ( k ) ) ↓ F ( x ′ ) as k → ∞ .
Revised: By Lemma 4.2, we have F ( x ( k ) ) ↓ F ( x ′ ) as k → ∞ .

Revised: To see this, we first note that F m ∗ is continuous over the set R ∗ by the definition of R ∗ in the beginning of this proof.

Revised: It follows by the beginning of this proof that (cid:110) x

Revised: ( k ) i (cid:111)

Original:  In addition,(74) indicates that F ( x ( k ) ) ↓ F m ∗ (¯ x ) as k → ∞ .
Revised: k ≥ k 0 ⊆ R ∗ i for all k ≥ k 0 and all i ∈ [ d ] . In addition, x ′ i ∈ R m ∗ i ⊆ R ∗ i for all i ∈

Revised: [ d ] .

Revised: Therefore, (cid:8) x ( k ) (cid:9) k ≥ k 0 and x ′ belong to R ∗ on which F m ∗ is continuous, so F ( x ( k ) )

Revised: =

Revised: F m ∗ ( x ( k ) )

Revised: ↓ F m ∗ ( x ′ )

Revised: =

Revised: F ( x ′ ) .

Revised: We also have F ( x ( k ) )

Revised: ↓ F m ∗ (¯ x ) as k → ∞ due to step 1 .

Original: As a result,
Revised: As a result,

Original: [Equation]
Revised: [Equation]

Original: for any x ′ ∈ Ω .
Revised: for any x ′ ∈ Ω .

Original: That is, F has constant value on Ω .
Revised: That is, F has constant value on Ω .

Original: It is noted that F m ∗ =
Revised: It is noted that F m ∗ =

Original: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) , so
Revised: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) ⊆

Revised: R ∗ , so the optimality of ¯ x indicates that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, since P (¯ x ) = m ∗ , we have F ( x ′ ) ≥inf { x ∈ R d | P ( x )
Revised: On the other hand, since P (¯ x ) = m ∗ , we have F ( x ′ ) ≥

Original: = m ∗ } F ( x ) .
Revised: [Equation]

Original: Combining this inequality and (76), we have F ( x ′ )
Revised: Combining this inequality and (80), we have F ( x ′ )

Original: =
Revised: =

Original: inf { x ∈ Rd
Revised: inf { x [Equation] R

Original:  | P ( x )= m∗ } F ( x ) .
Revised: [Equation] | P ( [Equation] = m

Revised: [Equation]

Revised: F ( x )

Revised: [Equation]

Original: Now we consider the case that f m ∗ i does not take the third case in (5) or (6) for all i ∈ [ d ] .
Revised: Step 3. We now prove that any limit point x ′ ∈ Ω is a critical point of F under a mild condition that f m ∗ i does not take the third case in (5) or (6) for all i ∈ [ d ] .

Original: As explained in the beginning of this proof,  R  i
Revised: As explained in the beginning of this proof, under this condition, R ∗ i

Original: =
Revised: =

Original: R for all i ∈ [ d ] .
Revised: R for all i ∈ [ d ] .

Original: In this case, because F ( x ′ ) =
Revised: Because F ( x ′ ) =

Original: F m ∗ (¯ x ) for any x ′ ∈ Ω , x ′ is an optimal solution to min x ∈ R d F m ∗ ( x ) , and F m ∗ is convex over R d .
Revised: F m ∗ (¯ x ) for any x ′ ∈ Ω , x ′ is an optimal solution to min x ∈ R d F m ∗ ( x ) , and F m ∗ is convex over R d .

Original: The optimality of x ′ for this convex programming problem indicates that 0 ∈ ˜ ∂F m ∗ ( x ′ ) .
Revised: The optimality of x ′ for this convex programming problem indicates that 0 ∈ ˜ ∂F m ∗ ( x ′ ) .

Original: Because F m ∗ =
Revised: Because F m ∗ =

Original: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) and x ′ ∈ (cid:8) x ∈ R d
Revised: F on the set (cid:8) x ∈ R d (cid:12)(cid:12) P ( x ) = m ∗ (cid:9) and x ′ ∈ (cid:8) x ∈ R d

Original: (cid:12)(cid:12) P ( x )
Revised: (cid:12)(cid:12) P ( x )

Original: = m ∗ (cid:9) , we have 0 ∈ ∂F ( x ′ ) .
Revised: = m ∗ (cid:9) , we have 0 ∈ ∂F ( x ′ ) .

Original: This can be verified using the definition of limitingsubdifferential by considering a constant sequence x k = x ′ for all k ≥ 1 .
Revised: This can be verified using the definition of limitingsubdifferential by considering a constant sequence x k = x ′ for all k ≥ 1 .

Original: We have with 0 ∈ ˜ ∂F ( x k ) because 0 ∈ ˜ ∂F m ∗ ( x ′ ) and P ( x ′ )
Revised: We have with 0 ∈ ˜ ∂F ( x k ) because 0 ∈ ˜ ∂F m ∗ ( x ′ ) and P ( x ′ )

Original: = m
Revised: = m

Original: ∗ .
Revised: ∗ .

Original: C I NEXACT PPGD

Original: We propose Inexact PPGD where the proximal mapping (9) in PPGD described in Algorithm 1 is inexact, that is,

Original: [Equation]

Original: where e ( k ) represents the inexactness of computation of ∇ g ( w ( k ) ) , and ε k captures the inexactness of evaluation of the proximal map given by

Original: [Equation]

Original: where u = w ( k ) − s (cid:0) ∇ g ( w ( k

Original: [Equation]

Original: + e ( k ) (cid:1) .

Original: It is noted that prox ε,Rsf m in (78) can be computed by projected subgradient descent, that is, every iterate by subgradient descent is projected onto [ w ( k ) i −

Original: R, w ( k ) i + R ] , and we set R = s ( G + F 0 ) .

Original: Theorem C.1.

Original: Suppose the proximal mapping (9) in PPGD is replaced by its inexact version (77) with R =

Original: s ( G + F 0 ) .

Original: [-Suppose s < min (cid:110) s 1 , ε 
-]
Revised: [-Suppose s < min (cid:110) s 1 , ε 
-]

Original: L g ( G + √ dF 0 ) , 1 L g (cid:111) with s 1 defined by (16), Assumption 1 and Assumption 2 hold.

Original: Then there exists a finite k 0 ≥ 1 such that all the statements of Theorem 4.4 remain true except for the convergence rate in (18).

Original: We have

Original: [Equation]

Original: where A k 0 ,k := (cid:80)

Original: ki =

Original: [-k 
-]
Revised: [-k 
-]

Original: i (cid:0) s (cid:13)(cid:13)

Original: e ( i ) (cid:13)(cid:13) 2 + √ 2 sε i (cid:1) , B k 0 ,k := (cid:80) ki

Original: = k 0 si 2 ε i , and a = O ( b ) means that a ≤ Cb for some constant C > 0 .

Original: Sketch of the Proof.

Original: With R =

Original: s ( G + F 0 ) , we have (cid:12)(cid:12)(cid:12) z

Original: ( k +1)

Original: i − w ( k ) i (cid:12)(cid:12)(cid:12) ≤ R =

Original: s ( G + F 0 ) .

Original: Note that the results of Lemma 4.1, Lemma 4.2, and Lemma B.5 essentially depend on the bound on (cid:12)(cid:12)(cid:12)

Original: z ( k +1)

Original: i −

Original: [Equation]

Original: With the s in the given condition and the bound R = s ( G + F 0 ) for (cid:12)(cid:12) (cid:12) z ( k +1)

Original: i − w ( k ) i (cid:12)(cid:12) (cid:12) , one can repeat the arguments in the proof of Lemma 4.1, Lemma 4.2, Lemma B.5, and Theorem 4.3 to prove that there exists a finite k 0 such that it can be verified that there exists a finite k 0 ≥ 1 such that all P ( x ( k ) )

Original: = m ∗ for some m ∗ ∈ N d for all k > k 0 .

Original: By (Schmidt et al., 2011, Lemma 2),

Original: [Equation]

Original: where v ( k ) ∈

Original: R d with (cid:13)(cid:13) v ( k ) (cid:13)(cid:13) 2 ≤ √ 2 sε k .

Original: ˜ ∂ ε denotes the ε -subdifferential.

Original: For a convex function q : R d → R , its ε -subdifferential at a point x ∈ R d is defined by ˜ ∂ ε q ( x ) := (cid:8) u (cid:12) (cid:12) q ( y ) ≥ q ( x ) + ⟨ u , y − x ⟩ − ε, ∀ y ∈ R d (cid:9) .

Original: Using (80) and the argument in the proof of Theorem 4.4, the convergence rate (79) can be proved.

Original: Similar to (Schmidt et al., 2011), the RHS of (79) matches the local convergen rate of exact PPGD

Original: O ( 1 k 2 ) with sufficiently small (cid:8) e ( k ) (cid:9) and { ε k } .