Original: GAGA : Deciphering Age-path of Generalized Self-paced Regularizer
Revised: GAGA : Deciphering Age-path of Generalized Self-paced Regularizer

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Nowadays self-paced learning (SPL) is an important machine learning paradigmthat mimics the cognitive process of humans and animals.
Revised: Nowadays self-paced learning (SPL) is an important machine learning paradigm that mimics the cognitive process of humans and animals.

Original: The SPL regime involvesa self-paced regularizer and a gradually increasing age parameter, which plays akey role in SPL but where to optimally terminate this process is still non-trivialto determine.
Revised: The SPL regime involves a self-paced regularizer and a gradually increasing age parameter, which plays a key role in SPL but where to optimally terminate this process is still non-trivial to determine.

Original: A natural idea is to compute the solution path w.r.t.
Revised: A natural idea is to compute the solution path w.r.t.

Original: age parameter( i.e. , age-path).
Revised: age parameter ( i.e. , age-path).

Original: However, current age-path algorithms are either limited to the simplest regularizer, or lack solid theoretical understanding as well as computationalefficiency.
Revised: However, current age-path algorithms are either limited to the simplest regularizer, or lack solid theoretical understanding as well as computational efficiency.

Original: To address this challenge, we propose a novel Generalized Age-path
Revised: To address this challenge, we propose a novel Generalized Age-path

Original: Algorithm ( GAGA ) for SPL with various self-paced regularizers based on ordinarydifferential equations (ODEs) and sets control, which can learn the entire solutionspectrum w.r.t.
Revised: Algorithm ( GAGA ) for SPL with various self-paced regularizers based on ordinary differential equations (ODEs) and sets control, which can learn the entire solution spectrum w.r.t.

Original: a range of age parameters.
Revised: a range of age parameters.

Original: To the best of our knowledge, GAGA isthe first exact path-following algorithm tackling the age-path for general self-pacedregularizer.
Revised: To the best of our knowledge, GAGA is the first exact path-following algorithm tackling the age-path for general self-paced regularizer.

Original: Finally the algorithmic steps of classic SVM and Lasso are describedin detail.
Revised: Finally the algorithmic steps of classic SVM and Lasso are described in detail.

Original: We demonstrate the performance of GAGA on real-world datasets, and findconsiderable speedup between our algorithm and competing baselines.
Revised: We demonstrate the performance of GAGA on real-world datasets, and find considerable speedup between our algorithm and competing baselines.

Original:  Introduction
Revised:  Introduction

Original: The SPL.
Revised: The SPL.

Original: Self-paced learning (SPL)
Revised: Self-paced learning (SPL)

Original: [1] is a classical learning paradigm and has attracted increasingattention in the communities of machine learning [2, 3, 4, 5, 6], data mining [7, 8] and computer vision[9, 10, 11].
Revised: [1] is a classical learning paradigm and has attracted increasing attention in the communities of machine learning [2, 3, 4, 5, 6], data mining [7, 8] and computer vision [9, 10, 11].

Original: The philosophy under this paradigm is simulating the strategy that how human-beingslearn new knowledge.
Revised: The philosophy under this paradigm is simulating the strategy that how human-beings learn new knowledge.

Original: In other words, SPL starts learning from easy tasks and gradually levelsup the difficulty while training samples are fed to the model sequentially.
Revised: In other words, SPL starts learning from easy tasks and gradually levels up the difficulty while training samples are fed to the model sequentially.

Original: At its core, SPL can beviewed as an automatic variant of curriculum learning (CL)
Revised: At its core, SPL can be viewed as an automatic variant of curriculum learning (CL)

Original: [12, 13] which uses prior knowledge todiscriminate between simple instances and hard ones along the training process.
Revised: [12, 13], which uses prior knowledge to discriminate between simple instances and hard ones along the training process.

Original: Different from CL,the SPL assigns a real-valued “easiness weight” to each sample implicitly by adding a self-pacedregularizer (SP-regularizer briefly) to the primal learning problem and optimizes the original modelparameters as well as these weights.
Revised: Different from CL, the SPL assigns a real-valued “easiness weight” to each sample implicitly by adding a self-paced regularizer (SP-regularizer briefly) to the primal learning problem and optimizes the original model parameters as well as these weights.

Original: Considering this setting, the SPL is reported to alleviate theproblem of getting stuck in bad local minima and provides better generalization as well as robustnessfor the models, especially in hard condition of heavy noises or a high outlier rate [1, 14].
Revised: Considering this setting, the SPL is reported to alleviate the problem of getting stuck in bad local minima and provides better generalization as well as robustness for the models, especially in hard condition of heavy noises or a high outlier rate [1, 14].

Original: There are two critical aspects in SPL, namely the SP-regularizer and a gradually increasing ageparameter.
Revised: There are two critical aspects in SPL, namely the SP-regularizer and a gradually increasing age parameter.

Original: Different SP-regularizers can be designed for different kinds of training tasks.
Revised: Different SP-regularizers can be designed for different kinds of training tasks.

Original: Atthe primary stage of SPL, only the hard SP-regularizer is utilized and leads to a binary variablefor weighting samples [1].
Revised: At the primary stage of SPL, only the hard SP-regularizer is utilized and leads to a binary variable for weighting samples [1].

Original: Going with the advancing of the diverse SP-regularizers [15], SPLequipped with different types of SP-regularizers has been successfully applied to various applications[16, 3, 17].
Revised: Going with the advancing of the diverse SP-regularizers [15], SPL equipped with different types of SP-regularizers has been successfully applied to various applications

Original: As for the age parameter ( a.k.a . pace parameter), the users are expected to increaseits value continually under the SPL paradigm, given that the age parameter represents the maturityof current model.
Revised: As for the age parameter ( a.k.a . pace parameter), the users are expected to increase its value continually under the SPL paradigm, given that the age parameter represents the maturity of current model.

Original: A lot of empirical practices have turned out that seeking out an appropriate ageparameter is crucial to the SPL procedure [18].
Revised: A lot of empirical practices have turned out that seeking out an appropriate age parameter is crucial to the SPL procedure [18].

Original: The SPL tends to obtain a worse performance in thepresence of noisy samples/outliers when the age parameter gets larger, or conversely, an insufficientage parameter makes the gained model immature ( i.e. underfitting.
Revised: The SPL tends to obtain a worse performance in the presence of noisy samples/outliers when the age parameter gets larger, or conversely, an insufficient age parameter makes the gained model immature ( i.e. underfitting.

Original: See Figure 1).
Revised: See Figure 1).

Original: On the age-path of SPL.
Revised: On the age-path of SPL.

Original: Although the SPL is a classical and widespread learning paradigm,when to stop the increasing process of age parameter in implementation is subject to surprisinglyfew theoretical studies.
Revised: Although the SPL is a classical and widespread learning paradigm, when to stop the increasing process of age parameter in implementation is subject to surprisingly few theoretical studies.

Original: In the majority of practices [19], the choice of the optimal model age has,for the time being, remained restricted to be made by experience or by using the trial-and-errorapproach, which is to adopt the alternate convex search (ACS)
Revised: In the majority of practices [19], the choice of the optimal model age has, for the time being, remained restricted to be made by experience or by using the trial-and-error approach, which is to adopt the alternate convex search (ACS)

Original: [20] multiple times at a predefinedsequence of age parameters.
Revised: [20] multiple times at a predefined sequence of age parameters.

Original: This operation is time-consuming and could miss some significantevents along the way of age parameter.
Revised: This operation is time-consuming and could miss some significant events along the way of age parameter.

Original: In addition, the SPL regime is a successive training process,which makes existing hyperparameter tuning algorithms like parallelizing sequential search [21] andbilevel optimization [22] difficult to apply.
Revised: In addition, the SPL regime is a successive training process, which makes existing hyperparameter tuning algorithms like parallelizing sequential search [21] and bilevel optimization [22] difficult to apply.

Original: Instead of training multiple subproblems at different ageparameters, a natural idea is to calculate the solution path about age parameter, namely age-path(e.g. , see Figure 2).
Revised: Instead of training multiple subproblems at different age parameters, a natural idea is to calculate the solution path about age parameter, namely age-path (e.g., see Figure 2).

Original: A solution path is a set of curves that demonstrate how the optimal solutionof a given optimization problem changes w.r.t.
Revised: A solution path is a set of curves that demonstrate how the optimal solution of a given optimization problem changes w.r.t.

Original: a hyperparameter.
Revised: a hyperparameter.

Original: Several papers like [23, 24] laidthe foundation of solution path algorithm in machine learning by demonstrating the rationale ofpath tracking, which is mainly built on the Karush-Khun-Tucker (KKT) theorem [25].
Revised: Several papers like [23, 24] laid the foundation of solution path algorithm in machine learning by demonstrating the rationale of path tracking, which is mainly built on the Karush-Khun-Tucker (KKT) theorem [25].

Original: Existingsolution path algorithms involve generalized Lasso [26], semi-supervised support vector classification[27], general parametric quadratic programming [28], etc.
Revised: Existing solution path algorithms involve generalized Lasso [26], semi-supervised support vector classification [27], general parametric quadratic programming [28], etc.

Original: However, none of the existing methods isavailable to SPL regime because they are limited to uni-convex optimization while the SPL objectiveis a biconvex formulation.
Revised: However, none of the existing methods is available to SPL regime because they are limited to uni-convex optimization while the SPL objective is a biconvex formulation.

Original: Assume we’ve got such an age-path, we can observe the whole self-pacedevolution process clearly and recover useful intrinsic patterns from it.
Revised: Assume we’ve got such an age-path, we can observe the whole self-paced evolution process clearly and recover useful intrinsic patterns from it.

Original: State of the art.
Revised: State of the art.

Original: Yet, a rapidly growing literature [29, 19, 30, 31] is devoted to developing betteralgorithms for solving the SPL optimization with ideas similar to age-path.
Revised: Yet, a rapidly growing literature [29, 19, 30, 31] is devoted to developing better algorithms for solving the SPL optimization with ideas similar to age-path.

Original: However, despite countlesstheoretical and empirical efforts, the understanding of age-path remains rather deficient.
Revised: However, despite countless theoretical and empirical efforts, the understanding of age-path remains rather deficient.

Original: Based ontechniques from incremental learning [32], [31] derived an exact age-path algorithm for mere hard
Revised: Based on techniques from incremental learning [32], [31] derived an exact age-path algorithm for mere hard

Original: SP-regularizer , where the path remains piecewise constant.
Revised: SP-regularizer , where the path remains piecewise constant.

Original: [29, 30] proposed a multi-objectiveself-paced learning (MOSPL) method to approximate the age-path by evolutionary algorithm, whichis not theoretically stable.
Revised: [29, 30] proposed a multi-objective self-paced learning (MOSPL) method to approximate the age-path by evolutionary algorithm, which is not theoretically stable.

Original: Unlike previous studies, the difficulty of revealing the exact generalizedage-path lies in the continuance of imposed weight and the alternate optimization procedure used tosolve the minimization function.
Revised: Unlike previous studies, the difficulty of revealing the exact generalized age-path lies in the continuance of imposed weight and the alternate optimization procedure used to solve the minimization function.

Original: From this point of view, the technical difficulty inherent in the studyof age-path with general SP-regularizer is intrinsically more challenging.
Revised: From this point of view, the technical difficulty inherent in the study of age-path with general SP-regularizer is intrinsically more challenging.

Original: Proposed Method.
Revised: Proposed Method.

Original: In order to tackle this issue, we establish a novel Generalized Age-path
Revised: In order to tackle this issue, we establish a novel Generalized Age-path

Original: Algorithm ( GAGA ) for various self-paced regularizers, which prevents a straightforward calculation of every age parameter.
Revised: Algorithm ( GAGA ) for various self-paced regularizers, which prevents a straightforward calculation of every age parameter.

Original: Our analysis is based on the theorem of partial optimum while previoustheoretical results are focused on the implicit SPL objective.
Revised: Our analysis is based on the theorem of partial optimum while previous theoretical results are focused on the implicit SPL objective.

Original: In particular, we enhance the originalobjective to a single-variable analysis problem, and use different sets to partition samples and functions by their confidence level and differentiability.
Revised: In particular, we enhance the original objective to a single-variable analysis problem, and use different sets to partition samples and functions by their confidence level and differentiability.

Original: Afterward, we conduct our main theorem resultsbased on the technique of ordinary differential equations (ODEs).
Revised: Afterward, we conduct our main theorem results based on the technique of ordinary differential equations (ODEs).

Original: In the process, the solution pathhits, exits, and slides along the various constraint boundaries.
Revised: In the process, the solution path hits, exits, and slides along the various constraint boundaries.

Original: The path itself is piecewise smoothwith kinks at the times of boundary hitting and escaping.
Revised: The path itself is piecewise smooth with kinks at the times of boundary hitting and escaping.

Original: Moreover, from this perspective we are ableto explain some shortcomings of conventional SPL practices and point out how we can improve them.
Revised: Moreover, from this perspective we are able to explain some shortcomings of conventional SPL practices and point out how we can improve them.

Original: We believe that the proposed method may be of independent interest beyond the particular problemstudied here and might be adapted to similar biconvex schemes.
Revised: We believe that the proposed method may be of independent interest beyond the particular problem studied here and might be adapted to similar biconvex schemes.

Original: Contributions.
Revised: Contributions.

Original: Therefore, the main contributions brought by this work are listed as follows.
Revised: Therefore, the main contributions brought by this work are listed as follows.

Original: • We firstly connect SPL paradigm to the concept of partial optimum and emphasize its importancehere that has been ignored before which gives a novel viewpoint to the robustness of SPL.
Revised: • We firstly connect SPL paradigm to the concept of partial optimum and emphasize its importance here that has been ignored before, which gives a novel viewpoint to the robustness of SPL.

Original: Theoretical studies are conducted to reveal that our result does exist some equivalence with previous literature, which makes our study more stable. 
Revised: Theoretical studies are conducted to reveal that our result does exist some equivalence with previous literature, which makes our study more stable. [Figure]

Original:  • A framework of computing the exact age-path for generalized SP-regularizer is derived using the technique of ODEs, which allows for the time-consuming ACS to be avoided.
Revised: [Figure] [Figure] • A framework of computing the exact age-path for generalized SP-regularizer is derived using the technique of ODEs, which allows for the time-consuming ACS to be avoided.

Original: Concretealgorithmic steps of classic SVM [33] and Lasso [34] are given for implementation.
Revised: Concrete algorithmic steps of classic SVM [33] and Lasso [34] are given for implementation.

Original: • Simulations on real and synthetic data are provided to validate our theoretical findings and justifytheir impact on the designing future SPL algorithms of practical interest.
Revised: • Simulations on real and synthetic data are provided to validate our theoretical findings and justify their impact on the designing future SPL algorithms of practical interest.

Original: Notations.
Revised: Notations.

Original: We write matrices in uppercase ( e.g. , X ) and vectors in lowercase with bold font ( e.g. ,x ).
Revised: We write matrices in uppercase ( e.g. , X ) and vectors in lowercase with bold font ( e.g. , x ).

Original: Given the index set E (or D ), X E (or X ED ) denotes the submatrix that taking rows with indexesin E (or rows/columns with indexes in E / D , respectively).
Revised: Given the index set E (or D ), X E (or X ED ) denotes the submatrix that taking rows with indexes in E (or rows/columns with indexes in E / D , respectively).

Original: Similarly notations lie on v E for vectorv , ℓ E ( x ) for vector functions ℓ ( x ) .
Revised: Similarly notations lie on v E for vector v , ℓ E ( x ) for vector functions ℓ ( x ) .

Original: For a set of scalar functions { ℓ i ( x ) } ni =1 , we denote the vectorfunction ℓ ( x ) where ℓ ( x ) =
Revised: For a set of scalar functions { ℓ i ( x ) } ni =1 , we denote the vector function ℓ ( x ) where ℓ ( x ) =

Original: ( ℓ i ( x ))
Revised: ( ℓ i ( x ))

Original: ni
Revised: ni

Original: =1 without statement and vice versa.
Revised: =1 without statement and vice versa.

Original: Moreover, we defer thefull proofs as well as the algorithmic steps on applications to the Appendix.[Figure] [Figure]
Revised: Moreover, we defer the full proofs as well as the algorithmic steps on applications to the Appendix.

Original: S o l u t i o n s

Original: [Figure]

Original:  Preliminaries 
Revised:  Preliminaries 2.1 Self-paced Learning

Original: [Figure] 2.1 Self-paced Learning

Original: [Figure] Suppose we have a dataset containing the label vector y ∈ R n and X ∈ R n × d , where n samples withd features are included.
Revised: Suppose we have a dataset containing the label vector y ∈ R n and X ∈ R n × d , where n samples with d features are included.

Original: The i -th row X i represents the i -th data sample x i ( i.e. , the i -th observation).
Revised: The i -th row X i represents the i -th data sample x i ( i.e. , the i -th observation).

Original: In this paper, the following unconstrained learning problem is considered
Revised: In this paper, the following unconstrained learning problem is considered

Original: [Equation]
Revised: [Equation]

Original: where R j ( · ) is the regularization item with a positive trade-off parameter α j , and ℓ i ( w ) 1 denotesloss function w.r.t.
Revised: where R j ( · ) is the regularization item with a positive trade-off parameter α j , and ℓ i ( w ) 2 denotes loss function w.r.t.

Original: w .
Revised: w .

Original: Definition 1 ( PC r Function) .
Revised: Definition 1 ( PC r Function) .

Original: Let f : U → R be a continuous function on the open set U ∈ R n .
Revised: Let f : U → R be a continuous function on the open set U ∈ R n .

Original: If { f i } i ∈ I f is a set of C r (i.e., r-times continuously differentiable) functions such that f ( x ) ∈{ f i ( x ) } i ∈
Revised: If { f i } i ∈ I f is a set of C r (i.e., r-times continuously differentiable) functions such that f ( x ) ∈ { f i ( x ) } i ∈

Original: I f holds for every x ∈ U , then f is an r-times piecewise continuously differentiablefunction, namely PC r function.
Revised: I f holds for every x ∈ U , then f is an r-times piecewise continuously differentiable function, namely PC r function.

Original: The { f i } i ∈
Revised: The { f i } i ∈

Original: I is a set of selection functions of f .
Revised: I is a set of selection functions of f .

Original: Assumption 1.
Revised: Assumption 1.

Original: We assume that ℓ i ( w ) and R j ( w ) are convex PC r functions each with a set ofselection functions (cid:83) k ∈ I ℓi (cid:8) D kℓ i (cid:9) and (cid:83)
Revised: We assume that ℓ i ( w ) and R j ( w ) are convex PC r functions each with a set of selection functions (cid:83) k ∈ I ℓi (cid:8) D k ℓ i (cid:9) and (cid:83)

Original: k ∈I
Revised: k ∈

Original:  R j (cid:110)
Revised: I R j (cid:110)

Original: D k R j (cid:111) , respectively.
Revised: D k R j (cid:111) , respectively.

Original: In self-paced learning, the goal is to jointly train the model parameter w and the latent weight variablev by minimizing 
Revised: In self-paced learning, the goal is to jointly train the model parameter w and the latent weight variable v by minimizing argmin w ∈ R d , v ∈ [0 , 1] n L ( w , v ) :

Original: [Equation]
Revised: = m (cid:88)

Revised: j = α j R j ( w ) + n (cid:88) i = (cid:104) v i

Original:  where f ( v, λ ) represents the SP-regularizer.
Revised: l i ( w ) + f ( v i , λ ) (cid:105) , (2) where f ( v, λ ) represents the SP-regularizer.

Original: 2.2 SP-regularizer
Revised: 2.2 SP-regularizer

Original: Definition 2 (SP-regularizer [35]) .
Revised: Definition 2 (SP-regularizer [35]) .

Original: Suppose that v is a weight variable, ℓ is the loss, and λ is the ageparameter.
Revised: Suppose that v is a weight variable, ℓ is the loss, and λ is the age parameter.

Original: f ( v, λ ) is called a self-paced regularizer, if
Revised: f ( v, λ ) is called a self-paced regularizer, if

Original: (i) f
Revised: (i) f

Original: ( v, λ ) is convex with respect to v ∈ [0 , 1] ; (ii) v ∗ ( ℓ, λ ) is monotonically decreasing w.r.t.
Revised: ( v, λ ) is convex with respect to v ∈ [0 , 1] ; (ii) v ∗ ( ℓ, λ ) is monotonically decreasing w.r

Original:  ℓ , and holds lim ℓ → 0 v ∗ ( ℓ,λ ) = 1 , lim ℓ →∞ v ∗ ( ℓ, λ ) = 0 ;
Revised: t. ℓ , and holds lim ℓ → 0 v ∗ ( ℓ, = 1 , lim ℓ →∞ v ∗ ℓ

Original:  (iii) v ∗ ( ℓ, λ ) is monotonically increasing w.r
Revised: ; (iii) v ∗ ( ℓ, λ ) is monotonically increasing w.r

Original: t. λ , and holds limλ →∞ v ∗ ( ℓ, λ ) ≤ 1 , lim λ
Revised: t. λ , and holds lim

Original:  →0 v ∗ ( ℓ, λ ) =0 ,
Revised: λ λ → λ ) =

Original:  where v ∗ ( ℓ, λ ) = arg min v ∈ [0 , 1] vℓ + f ( v, λ ) .
Revised: , where v ∗ ( ℓ, λ ) = arg min v ∈ [0 , 1] vℓ + f ( v, λ ) .

Original: The Definition 2 gives axiomatic definition of SP-regularizer.
Revised: The Definition 2 gives axiomatic definition of SP-regularizer.

Original: Some frequently utilized SP-regularizersinclude f H ( v
Revised: Some frequently utilized SP-regularizers include f H ( v, λ )

Original: [Equation]
Revised: = − λv , f L ( v, λ ) = λ (cid:0) 12 v 2 − v (cid:1) , f M ( v, λ, γ ) = γ 2 v + γ/λ and f LOG ( v, λ, α ) = 1 α KL (1 + αλ, v ) , which represents hard, linear, mixture, LOG SP-regularizer, respectively.

Original: 2.3 Biconvex 
Revised: 2.3 Biconvex Optimization

Original: Optimization

Original: Definition 3 (Biconvex Function) .
Revised: Definition 3 (Biconvex Function) .

Original: A function f : B → R on a biconvex set B ⊆
Revised: A function f : B → R on a biconvex set B ⊆

Original: X × Y is calleda biconvex function on B , if f x ( · ) :
Revised: X × Y is called a biconvex function on B , if f x ( · ) :

Original: = f ( x, · ) :
Revised: = f ( x, · ) :

Original: B x → R is a convex function on B x for every fixedx ∈ X and f y ( · ) :
Revised: B x → R is a convex function on B x for every fixed x ∈ X and f y ( · ) :

Original: = f ( · , y ) :
Revised: = f ( · , y ) :

Original: B y
Revised: B y

Original: → R is a convex function on B y for every fixed y ∈ Y .
Revised: → R is a convex function on B y for every fixed y ∈ Y .

Original: Definition 4 (Partial Optimum) .
Revised: Definition 4 (Partial Optimum) .

Original: Let f : B
Revised: Let f : B

Original: → R be a given biconvex function and let ( x ∗ , y ∗ ) ∈
Revised: → R be a given biconvex function and let ( x ∗ , y ∗ ) ∈

Original: B .
Revised: B .

Original: Then, z ∗ =
Revised: Then, z ∗ =

Original: ( x ∗ , y ∗ ) is called a partial optimum of f on B , if f ( x ∗ , y ∗ ) ≤ f ( x, y ∗ ) ∀ x ∈
Revised: ( x ∗ , y ∗ ) is called a partial optimum of f on B , if f ( x ∗ , y ∗ ) ≤ f ( x, y ∗ ) ∀ x ∈

Original: B y ∗ and f ( x ∗ , y ∗ ) ≤ f ( x ∗ , y )
Revised: B y ∗ and f ( x ∗ , y ∗ ) ≤ f ( x ∗ , y )

Original: ∀
Revised: ∀

Original: y ∈
Revised: y ∈

Original: B x ∗ .
Revised: B x ∗ .

Original: Require: Dataset X and y , age parameter λ .
Revised: Require: Dataset X and y , age parameter λ .

Original: 1: Initialize w .
Revised: 1: Initialize w .

Original: 2: while not converged do 3:
Revised: 2: while not converged do 3:

Original: Update v ∗ = argmin v L ( w ∗ , v ) .
Revised: Update v ∗ = argmin v L ( w ∗ , v ) .

Original: Update w ∗ =
Revised: Update w ∗ =

Original: argmin w L ( w , v ∗ ) .
Revised: argmin w L ( w , v ∗ ) .

Original: 5: end while
Revised: 5: end while

Original: Ensure: ˆ w
Revised: Ensure: ˆ w

Original: Optimizing (2) leads to a biconvex optimizationproblem and is generally non-convex with fixed λ , inwhich a number of local minima exist and previousconvex optimization tools can’t achieve a promisingeffect [36].
Revised: Optimizing (2) leads to a biconvex optimization problem and is generally non-convex with fixed λ , in which a number of local minima exist and previous convex optimization tools can’t achieve a promising effect [36].

Original: It’s reasonably believed that algorithmstaking advantage of the biconvex structure are moreefficient in the corresponding setting.
Revised: It’s reasonably believed that algorithms taking advantage of the biconvex structure are more efficient in the corresponding setting.

Original: For frequentlyused one, ACS ( c.f. Algorithm 1) is presented tooptimize x and y in f ( x, y ) alternately until terminating condition is met.
Revised: For frequently used one, ACS ( c.f. Algorithm 1) is presented to optimize x and y in f ( x, y ) alternately until terminating condition is met.

Original: Remark 1.
Revised: Remark 1.

Original: The order of the optimization subproblems in line 3 & 4 in Algorithm 1 can be permuted.
Revised: The order of the optimization subproblems in line 3 & 4 in Algorithm 1 can be permuted.

Original: Theorem 1.
Revised: Theorem 1.

Original: [37] Let X ⊆ R n and Y ⊆
Revised: [37] Let X ⊆ R n and Y ⊆

Original: R m be closed sets and let f : X × Y
Revised: R m be closed sets and let f : X × Y

Original: → R be continuous.
Revised: → R be continuous.

Original: Let the sequence { z i } i ∈ N + generated by ACS converges to z ∗ ∈ X ×Y .
Revised: Let the sequence { z i } i ∈ N + generated by ACS converges to z ∗ ∈ X ×Y .

Original: Then z ∗ is a partial optimum.
Revised: Then z ∗ is a partial optimum.

Original: 2.4 Theoretical Consistency
Revised: 2.4 Theoretical Consistency

Original: Researchers in earlier study [35] theoretically conducted the latent SPL loss ( a.k.a. , implicit objective)and further proved that the SPL paradigm converges to the stationary point of the latent objectiveunder some mild assumptions, which gives explanation to the robustness of the SPL [35, 38].
Revised: Researchers in earlier study [35] theoretically conducted the latent SPL loss ( a.k.a. , implicit objective) and further proved that the SPL paradigm converges to the stationary point of the latent objective under some mild assumptions, which gives explanation to the robustness of the SPL [35, 38].

Original: In thispaper, we focus on the partial optimum of original SPL objective and result is given in Theorem 2.
Revised: In this paper, we focus on the partial optimum of original SPL objective and result is given in Theorem 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the same assumptions in Theorem 2 of [38], the partial optimum of SPL objectiveconsists with the stationary point of implicit SPL objective G λ .
Revised: Under the same assumptions in Theorem 2 of [38], the partial optimum of SPL objective consists with the stationary point of implicit SPL objective G λ .

Original: Factoring in both Theorem 1 & 2, the ACS procedure (or its variations) used in SPL paradigm indeedfinds the partial optimum of SPL objective, which unifies the two proposed analysis frameworks andprovides more in-depth understanding to the intrinsic mechanism behind the SPL regime.
Revised: Factoring in both Theorem 1 & 2, the ACS procedure (or its variations) used in SPL paradigm indeed finds the partial optimum of SPL objective, which unifies the two proposed analysis frameworks and provides more in-depth understanding to the intrinsic mechanism behind the SPL regime.

Original:  Age-Path Tracking 
Revised:  Age-Path Tracking 3.1 Objective Reformulation

Original: 3.1 Objective Reformulation

Original: For the convenience of derivation, we denote the set
Revised: For the convenience of derivation, we denote the set

Original: I R or ¯ I R to be the set of indexes
Revised: I R or ¯ I R to be the set of indexes

Original: j where R j isdifferentiable or non-differentiable at w , respectively.
Revised: j where R j is differentiable or non-differentiable at w , respectively.

Original: Similarly, we have I ℓ and ¯ I ℓ w.r.t.
Revised: Similarly, we have I ℓ and ¯ I ℓ w.r.t.

Original: ℓ i .x i ∈
Revised: ℓ i .

Original:  

Original: [Equation] D [Equation] only used in mixture
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Moreover, the training of the
Revised: Moreover, the training of the

Original: SPL is essentially a process ofadaptive sample selection, sowe classify all the sample pointsin the training set into different sets P :=
Revised: SPL is essentially a process of adaptive sample selection, so we classify all the sample points in the training set into different sets P :=

Original: {E , D , M , ... } according to their confidence (or loss) 2 .
Revised: {E , D , M , ... } according to their confidence (or loss) 3 .

Original: Figure 3 illustrates a partition example when hard, linear or
Revised: Figure 3 illustrates a partition example when hard, linear or

Original: LOG SP-regularizer is used.
Revised: LOG SP-regularizer is used.

Original: Since subproblem in line 3 of Algorithm 1 always gives closed-formsolutions in iterations 3 , we can rewrite SPL optimization objective as (3), which is indeed equivalentto searching a partial optimum of (2).
Revised: Since subproblem in line 3 of Algorithm 1 always gives closed-form solutions in iterations 4 , we can rewrite SPL optimization objective as (3), which is indeed equivalent to searching a partial optimum of (2).

Original: [Equation]
Revised: [Equation]

Original: 3.2 Piecewise Smooth Age-path
Revised: 3.2 Piecewise Smooth Age-path

Original: The KKT theorem [25] states that (3) holds iff
Revised: The KKT theorem [25] states that (3) holds iff

Original: [Equation]
Revised: [Equation]

Original: [Equation] 169 where ∂ ( · ) denotes the subdifferential (set of all subgradients).
Revised: where ∂ ( · ) denotes the subdifferential (set of all subgradients).

Original: In the PC r setting, subgradient canbe expressed explicitly by essentially active functions ( c.f. Lemma 1).
Revised: In the PC r setting, subgradient can be expressed explicitly by essentially active functions ( c.f. Lemma 1).

Original: Definition 5 (Essentially Active Set) .
Revised: Definition 5 (Essentially Active Set) .

Original: Let f : U → R be a PC r function on the open set U ∈ R nwith a set of selection functions { f i } i ∈ I f .
Revised: Let f : U → R be a PC r function on the open set U ∈ R n with a set of selection functions { f i } i ∈ I f .

Original: For x ∈ U , we call I af ( x ) :
Revised: For x ∈ U , we call I af ( x ) :

Original: = { i ∈ I f : f ( x ) =
Revised: = { i ∈ I f : f ( x ) =

Original: f i ( x ) }is the active set at x , and I ef ( x ) :
Revised: f i ( x ) } is the active set at x , and I ef ( x ) :

Original: = (cid:110) i ∈ I f : x ∈ cl (cid:16) int (cid:0) {
Revised: = (cid:110) i ∈ I f : x ∈ cl (cid:16) int (cid:0) {

Original: y ∈
Revised: y ∈

Original: U : f ( y ) =
Revised: U : f ( y ) =

Original: f i ( y ) } (cid:1)(cid:17)(cid:111) is theessentially active set at x , where cl ( · ) and int ( · ) denote the closure and interior of a set.
Revised: f i ( y ) } (cid:1)(cid:17)(cid:111) is the essentially active set at x , where cl ( · ) and int ( · ) denote the closure and interior of a set.

Original: Lemma 1.
Revised: Lemma 1.

Original: [39] Let f : U → R be a PC r function on an open set U and (cid:83) i ∈ I f { f i } is a set of selection functions of f , then ∂f ( x ) =
Revised: [39] Let f : U → R be a PC r function on an open set U and (cid:83) i ∈ I f { f i } is a set of selection functions of f , then ∂f ( x ) =

Original: conv( (cid:83) i ∈ I ef ( x ) { f i ( x ) } ) =
Revised: conv( (cid:83) i ∈ I ef ( x ) { f i ( x ) } ) =

Original: { (cid:80) i ∈ I ef ( x ) t i ∇ f i ( x ) : (cid:80)
Revised: { (cid:80) i ∈ I ef ( x ) t i ∇ f i ( x ) : (cid:80)

Original: i ∈ I
Revised: i ∈ I

Original: ef ( x ) t i
Revised: ef ( x ) t i

Original: =1 , t i ≥ 0 } .
Revised: = 1 , t i ≥ 0 } .

Original: Especially, if f is differentiable at x , ∂f ( x ) = {∇ f ( x ) } .
Revised: Especially, if f is differentiable at x , ∂f ( x ) = {∇ f ( x ) } .

Original: Assumption 2.
Revised: Assumption 2.

Original: We assume that I a R j ( x ) =
Revised: We assume that I a R j ( x ) =

Original: I e R j ( x ) , I aℓ i ( x ) =
Revised: I e R j ( x ) , I aℓ i ( x ) =

Original: I eℓ i ( x ) holds for all x considered andall R j , ℓ i in the following.
Revised: I eℓ i ( x ) holds for all x considered and all R j , ℓ i in the following.

Original: We adopt a mild relaxation as shown in Assumption 2.
Revised: We adopt a mild relaxation as shown in Assumption 2.

Original: Investigation [40] confirmed that it can beeasily established in most practical scenarios.
Revised: Investigation [40] confirmed that it can be easily established in most practical scenarios.

Original: Without loss of generality, we suppose the following
Revised: Without loss of generality, we suppose the following

Original: Assumption 3 also holds to further ease the notation burden.
Revised: Assumption 3 also holds to further ease the notation burden.

Original: Assumption 3.
Revised: Assumption 3.

Original: We assume that R j , ℓ i are non-differentiable at x with multiple active selectionfunctions, where j ∈ { 1 , . . .
Revised: We assume that R j , ℓ i are non-differentiable at x with multiple active selection functions, where j ∈ { 1 , . . .

Original: , m } , i ∈ { 1 , . . . , n } .
Revised: , m } , i ∈ { 1 , . . . , n } .

Original: Therefore, the condition (4) can be rewritten in detail.
Revised: Therefore, the condition (4) can be rewritten in detail.

Original: Formally, there exists ˆ t R and ˆ t ℓ such thatm
Revised: Formally, there exists ˆ t R and ˆ t ℓ such that m

Original: [Equation]
Revised: [Equation]

Original: where r j , l i is randomly selected from I a R j , I aℓ i and being fixed.
Revised: where r j , l i is randomly selected from I a R j , I aℓ i and being fixed.

Original: The second and third equations in(5) describe the active sets while the last two equations describe the subgradients.
Revised: The second and third equations in (5) describe the active sets while the last two equations describe the subgradients.

Original: When the partialoptimum is on the smooth part, we denote the left side of equations (5) to be a C 1 function F , thusrevealing that the solution path lies on the smooth manifold F ( w , λ, t R , t ℓ ) = 0 .
Revised: When the partial optimum is on the smooth part, we denote the left side of equations (5) to be a C 1 function F , thus revealing that the solution path lies on the smooth manifold F ( w , λ, t R , t ℓ ) = 0 .

Original: By the time itcomes across the kink 4 , we need to refresh the index partitions and update (5) to run next segment ofpath.
Revised: By the time it comes across the kink 5 , we need to refresh the index partitions and update (5) to run next segment of path.

Original: WLOG, we postulate that the initial point is non-degenerate ( i.e. , the J w , t R , t ℓ is invertible).
Revised: WLOG, we postulate that the initial point is non-degenerate ( i.e. , the J w , t R , t ℓ is invertible).

Original: Bydirectly applying the implicit function theorem, the existence and uniqueness of a local C 1 solutionpath (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) can be established over here.
Revised: By directly applying the implicit function theorem, the existence and uniqueness of a local C 1 solution path (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) can be established over here.

Original: Drawing from the theory of differential geometrygives another intuitive understanding of age-path, which tells that the first equation in (5) indeed usesan analogue moving frame [41] to represent a smooth curve that consists of the smooth structure.
Revised: Drawing from the theory of differential geometry gives another intuitive understanding of age-path, which tells that the first equation in (5) indeed uses an analogue moving frame [41] to represent a smooth curve that consists of the smooth structure.

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a partial optimum ( ˆ w , v ∗ ( ˆ w , λ )) at λ 0 , ˆ t R , ˆ t ℓ in (5) can be solved from
Revised: Given a partial optimum ( ˆ w , v ∗ ( ˆ w , λ )) at λ 0 , ˆ t R , ˆ t ℓ in (5) can be solved from

Original: F (cid:0) ˆ w , λ 0 , ˆ t R , ˆ t ℓ (cid:1)
Revised: F (cid:0) ˆ w , λ 0 , ˆ t R , ˆ t ℓ (cid:1)

Original: = 0 .
Revised: = 0 .

Original: If the Jacobian J w , t R , t ℓ is invertible at (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) , then in an openneighborhood of λ 0 , (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) is a C 1 function w.r.t.
Revised: If the Jacobian J w , t R , t ℓ is invertible at (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) , then in an open neighborhood of λ 0 , (cid:0) ˆ w , ˆ t R , ˆ t ℓ (cid:1) is a C 1 function w.r.t.

Original: λ and fits the ODEs
Revised: λ and fits the ODEs

Original: [Equation]
Revised: [Equation]

Original: in which the explicit expressions of J − 1 w , t R , t ℓ , J λ are listed in Appendix A.
Revised: in which the explicit expressions of J − 1 w , t R , t ℓ , J λ are listed in Appendix A.

Original: Corollary 1.
Revised: Corollary 1.

Original: If all the functions are smooth in a neighborhood of the initial point, then (6) can besimplified as d ˆ w /dλ =
Revised: If all the functions are smooth in a neighborhood of the initial point, then (6) can be simplified as d ˆ w /dλ =

Original: − J − 1 w · J λ .
Revised: − J − 1 w · J λ .

Original: Remark 2.
Revised: Remark 2.

Original: Our supplement parts in Appendix A present additional discussions.
Revised: Our supplement parts in Appendix A present additional discussions.

Original: 3.3 Critical Points
Revised: 3.3 Critical Points

Original: By solving the initial value problem (6) numerically with ODE solvers, the solution path regardingto λ can be computed swiftly before any of P , I R or I ℓ changes.
Revised: By solving the initial value problem (6) numerically with ODE solvers, the solution path regarding to λ can be computed swiftly before any of P , I R or I ℓ changes.

Original: We denote such point where theset changes a critical point , which can be divided into turning point or jump point on the basis ofpath’s property at that point.
Revised: We denote such point where the set changes a critical point , which can be divided into turning point or jump point on the basis of path’s property at that point.

Original: To be more specific, the age-path is discontinuous at a jump point, whilebeing continuous but non-differentiable at the turning point.
Revised: To be more specific, the age-path is discontinuous at a jump point, while being continuous but non-differentiable at the turning point.

Original: This is also verified by Figure 2 andlarge quantity of numerical experiments.
Revised: This is also verified by Figure 2 and large quantity of numerical experiments.

Original: At turning points, the operation of the algorithm is to update P , I R , I ℓ according to index violator(s)and move on to the next segment.
Revised: At turning points, the operation of the algorithm is to update P , I R , I ℓ according to index violator(s) and move on to the next segment.

Original: At jump points, path is no longer continuous and warm-start 5 canbe utilized to speed up the training procedure.
Revised: At jump points, path is no longer continuous and warm-start 6 can be utilized to speed up the training procedure.

Original: The total number of critical points on the solution pathis estimated at approximately O ( |D ∪ M| ) 6 .
Revised: The total number of critical points on the solution path is estimated at approximately O ( |D ∪ M| ) 7 .

Original: Consequently, we present a heuristic trick to figure outthe type of a critical point with complexity O ( d ) , so as to avoid excessive restarts.
Revised: Consequently, we present a heuristic trick to figure out the type of a critical point with complexity O ( d ) , so as to avoid excessive restarts.

Original: As a matter of fact,the solutions returned by the numerical ODE solver is continuous with the fixed set, despite it mayactually passes a jump point.
Revised: As a matter of fact, the solutions returned by the numerical ODE solver is continuous with the fixed set, despite it may actually passes a jump point.

Original: In this circumstance, the solutions returned by ODEs have deviatedfrom the ground truth partial optimum.
Revised: In this circumstance, the solutions returned by ODEs have deviated from the ground truth partial optimum.

Original: Hence it’s convenient that we can detect KKT conditions tomonitor this behavior.
Revised: Hence it’s convenient that we can detect KKT conditions to monitor this behavior.

Original: This approach enjoys higher efficiency than detecting the partition conditionsthemselves, especially when the set partition is extraordinarily complex.
Revised: This approach enjoys higher efficiency than detecting the partition conditions themselves, especially when the set partition is extraordinarily complex.

Original: 3.4 GAGA Algorithm
Revised: 3.4 GAGA Algorithm

Original: Input : Initial solution ˆ w | λ t = λ min , X , y , λ min and λ max .

Original: Output : Age-Path ˆ w ( λ ) on [ λ

Original: min , λ max ] .

Original: 1: λ t ← λ min , set P , I R , I ℓ according to ˆ w | λ t .

Original: 2: while λ t ≤ λ max do 3:

Original: Solve (6) and examine partition P , I R , I ℓ simultaneously.

Original: 4: if Partition P , I R , I ℓ was not met then 5:

Original: Update P , I R , I ℓ according to index violator(s).

Original: Solve (6) with updated P , I R , I ℓ .

Original: 7: if KKT conditions are not met then 8:

Original: Warm start at λ t + δ (for a small δ > 0 ).

Original: 9: end if 10: end if 11: end while

Original: There has been extensive research in applied mathematics onnumerical methods for solving ODEs, where the solver couldautomatically determine the stepsize of λ when solving (6).
Revised: There has been extensive research in applied mathematics on numerical methods for solving ODEs, where the solver could automatically determine the step size of λ when solving (6).

Original: Inthe tracking process, we compute the solutions with regard toλ .
Revised: In the tracking process, we compute the solutions with regard to λ .

Original: After detecting a new criticalpoint, we need to reset P , I R , I ℓat turning point while warm-startis required for jump point.
Revised: After detecting a new critical point, we need to reset P , I R , I ℓ at turning point while warm-start is required for jump point.

Original: Theabove procedure is repeated until we traverse the entire interval[ λ min , λ max ] .
Revised: The above procedure is repeated until we traverse the entire interval [ λ min , λ max ] .

Original: We show the detailed procedure in Algorithm 2.
Revised: We show the detailed procedure in Algorithm 2.

Original: The main computational burden occurs in solving J − 1 in (6) with an approximate complexity
Revised: The main computational burden occurs in solving J − 1 in (6) with an approximate complexity

Original: O
Revised: O

Original: ( p 3 )in general, where p denotes the dimension of J .
Revised: ( p 3 ) in general, where p denotes the dimension of J .

Original: Further promotion can be made via decompositionor utilizing the sparse representation of J on specific learning problems.
Revised: Further promotion can be made via decomposition or utilizing the sparse representation of J on specific learning problems.

Original:  Practical Guides
Revised:  Practical Guides

Original: In this section, we provide practical guides of using the GAGA to solve two important learningproblems, i.e. , classic SVM and Lasso.
Revised: In this section, we provide practical guides of using the GAGA to solve two important learning problems, i.e. , classic SVM and Lasso.

Original: The detailed steps of algorithms are displayed in Appendix C.
Revised: The detailed steps of algorithms are displayed in Appendix C.

Original: 4.1 Support Vector Machines
Revised: 4.1 Support Vector Machines

Original: Support vector machine (SVM)
Revised: Support vector machine (SVM)

Original: [33] has attracted much attention from researchers in the areas ofbioinformatics, computer vision and pattern recognition.
Revised: [33] has attracted much attention from researchers in the areas of bioinformatics, computer vision and pattern recognition.

Original: Given the dataset X and label y , we focusto interval length of λ , the nature of objective distribution data. on the classic support vector classification as
Revised: Given the dataset X and label y , we focus on the classic support vector classification as

Original: [Equation]
Revised: [Equation]

Original: 248 where H is the reproducing kernel Hilbert space (RKHS) with the inner product ⟨·⟩ and correspondingkernel function ϕ .
Revised: where H is the reproducing kernel Hilbert space (RKHS) with the inner product ⟨·⟩ and corresponding kernel function ϕ .

Original: Seeing that (5) still holds in infinite dimensional H , the above analyses can bedirectly applied here.
Revised: Seeing that (5) still holds in infinite dimensional H , the above analyses can be directly applied here.

Original: We also utilize the kernel trick [42] to avoid involving the explicit expression ofϕ .
Revised: We also utilize the kernel trick [42] to avoid involving the explicit expression of ϕ .

Original: In consistent with the framework, we have ℓ i =
Revised: In consistent with the framework, we have ℓ i =

Original: C max { 0 , g i } and g i = 1 − y i ( ⟨ ϕ ( x i ) , w ⟩ + b ) .
Revised: C max { 0 , g i } and g i = 1 − y i ( ⟨ ϕ ( x i ) , w ⟩ + b ) .

Original: The I ℓ and P are determined by g , thus we merely need to refine the division of E as E N = { i ∈ E :g i < 0 } , E Z = { i ∈ E : g i = 0 } and E P = { i ∈ E : g i > 0 } , which gives I ℓ = E N ∪ E P ∪ D ( ∪M ) .
Revised: The I ℓ and P are determined by g , thus we merely need to refine the division of E as E N = { i ∈ E : g i < 0 } , E Z = { i ∈ E : g i = 0 } and E P = { i ∈ E : g i > 0 } , which gives I ℓ = E N ∪ E P ∪ D ( ∪M ) .

Original: Afterwards, with some simplifications and denoting ˆ α = C v ∗ ⊙ ˆ t , we can obtain a simplified versionof (5), from where the age-path can be equivalently calculated w.r.t.
Revised: Afterwards, with some simplifications and denoting ˆ α = C v ∗ ⊙ ˆ t , we can obtain a simplified version of (5), from where the age-path can be equivalently calculated w.r.t.

Original: optimal (ˆ α , ˆ b ) .
Revised: optimal (ˆ α , ˆ b ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: When α , b indicate a partial optimum, the dynamics of optimal α , b in (7) w.r.t.
Revised: When α , b indicate a partial optimum, the dynamics of optimal α , b in (7) w.r.t.

Original: λ forthe linear and mixture SP-regularizer are shown as[-
 [Equation]-]
Revised: λ for the linear and mixture SP-regularizer are shown as[-
 [Equation]-]

Original: [Equation]
Revised: [Equation]

Original: respectively, where ˜ Q ME Z
Revised: respectively, where ˜ Q ME Z

Original: =
Revised: =

Original: Diag { ℓ − 32 M } Q ME Z , ˜ Q MM = 
Revised: Diag { ℓ − 32 M } Q ME Z , ˜ Q MM = Diag { ℓ − 32 M [Equation]

Original: Diag { ℓ − 32 M } Q MM , ˜ y M = ℓ − 32 M ⊙ y M .
Revised: Q MM [Equation] y M = ℓ − 32 M ⊙ y M [Equation]

Original: Other components are constant as α E N = 

Revised: Other components are constant as α E N =

Original: E N , α D = 0 D .
Revised: E N , α D = 0 D .

Original: Only for mixture regularizer, α E P = 1 E P .
Revised: Only for mixture regularizer, α E P = 1 E P .

Original: Critical Point.
Revised: Critical Point.

Original: We track g along the path.
Revised: We track g along the path.

Original: The critical point is sparked off by any set in
Revised: The critical point is sparked off by any set in

Original: E N , E Z , E P , D ( , M ) changes.
Revised: E N , E Z , E P , D ( , M ) changes.

Original: 4.2 Lasso
Revised: 4.2 Lasso

Original: Lasso [34] uses a sparsity based regularization term that can produce sparse solutions.
Revised: Lasso [34] uses a sparsity based regularization term that can produce sparse solutions.

Original: Given thedataset X and label y , the Lasso regression is stated as
Revised: Given the dataset X and label y , the Lasso regression is stated as

Original: [Equation]
Revised: [Equation]

Original: 267 We expand ∥ w ∥ 1 = (cid:80) d j =1 | w j | and treat | w j | as R j in (5), hence the I R = { 1 ≤ j ≤ d : w j ̸ = 0 } .
Revised: We expand ∥ w ∥ 1 = (cid:80) dj =1 | w j | and treat | w j | as R j in (5), hence the I R = { 1 ≤ j ≤ d : w j ̸ = 0 } .

Original: We denote the set of active or inactive functions (components) by A
Revised: We denote the set of active or inactive functions (components) by A

Original: = I R , ¯ A
Revised: = I R , ¯ A

Original: = ¯ I R , respectively.
Revised: = ¯ I R , respectively.

Original: Inview of the fact that ∂ | w j | removes t R j from the equations in (5) for j ∈ ¯ A , we only pay attention tothe A part w.r.t.
Revised: In view of the fact that ∂ | w j | removes t R j from the equations in (5) for j ∈ ¯ A , we only pay attention to the A part w.r.t.

Original: the ( w A , λ ) .
Revised: the ( w A , λ ) .

Original: The ℓ is defined as 12 n ( X w − y ) 2 in the following.
Revised: The ℓ is defined as 12 n ( X w − y ) 2 in the following.

Original: Proposition 2.
Revised: Proposition 2.

Original: When ( w , v ∗ ( w , λ )) is a partial optimum, the dynamics of optimal w in (10) w.r.t.
Revised: When ( w , v ∗ ( w , λ )) is a partial optimum, the dynamics of optimal w in (10) w.r.t.

Original: λfor the linear and mixture SP-regularizer are described as
Revised: λ for the linear and mixture SP-regularizer are described as

Original: [Equation]
Revised: [Equation]

Revised: respectively, where ˜ X E∪MA =

Original: [Equation]
Revised: [Equation]

Original: respectively, where ˜ X E∪MA =
Revised: X [Equation] X

Original: [Equation]
Revised: [Equation]

Original:  and w ¯ A = 0{+
+} ¯A .
Revised: (cid:19) and w ¯ A = {+
+} ¯

Revised: [Equation]

Original: Critical Point.
Revised: Critical Point.

Original: The critical point is encountered when A or P changes.[Table] [Table]
Revised: The critical point is encountered when A or P changes.

Original:  Experimental Evaluation
Revised:  Experimental Evaluation

Original: We present the empirical results of the proposed GAGA on two tasks: SVM for binary classificationand Lasso for robust regression in the noisy environment.
Revised: We present the empirical results of the proposed GAGA on two tasks: SVM for binary classification and Lasso for robust regression in the noisy environment.

Original: The results demonstrate that our approachoutperforms existing SPL implementations on both tasks.
Revised: The results demonstrate that our approach outperforms existing SPL implementations on both tasks.

Original: Baselines.
Revised: Baselines.

Original: GAGA is compared against three baseline methods: 1) Original learning model without
Revised: GAGA is compared against three baseline methods: 1) Original learning model without

Original: SPL regime.
Revised: SPL regime.

Original: 2) ACS [20] performs sequential search of λ , which is the most commonly usedalgorithm in SPL implementations.
Revised: 2) ACS [20] performs sequential search of λ , which is the most commonly used algorithm in SPL implementations.

Original: 3) MOSPL [29] is a state-of-the-art age-path approach that usingthe multi objective optimization, in which the solution is derived with the age parameter λ implicitly.
Revised: 3) MOSPL [29] is a state-of-the-art age-path approach that using the multi objective optimization, in which the solution is derived with the age parameter λ implicitly [Figure] [Figure] [Figure]

Revised: [Table]

Original: Datasets.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  The Table 1 summarizes the datasets information.
Revised: [Table] [Table] Datasets. The Table 1 summarizes the datasets information.

Original: As universally known that SPL enjoysrobustness in noisy environments, we impose 30% of noises into the real-world datasets.
Revised: As universally known that SPL enjoys robustness in noisy environments, we impose 30% of noises into the real-world datasets.

Original: In particular,we generate noises by turning normal samples into poisoning ones by flipping their labels [45, 46]for classification tasks.
Revised: In particular, we generate noises by turning normal samples into poisoning ones by flipping their labels [45, 46] for classification tasks.

Original: For regression problem, noises are generated by the similar distribution of thetraining data as performed in [47].[Figure]
Revised: For regression problem, noises are generated by the similar distribution of the training data as performed in [47].

Original: Experiment Setting.
Revised: Experiment Setting.

Original: In experiments, we first verify the performance of GAGA and traditional ACSalgorithm under different noise intensity to reflect the robustness of GAGA .
Revised: In experiments, we first verify the performance of GAGA and traditional ACS algorithm under different noise intensity to reflect the robustness of GAGA .

Original: We further study thegeneralization performance of GAGA with competing methods, so as to show its ability to selectoptimal model during the learning process.
Revised: We further study the generalization performance of GAGA with competing methods, so as to show its ability to select optimal model during the learning process.

Original: Meanwhile, we also evaluate the running efficiencybetween GAGA and existing SPL implementations in different settings, which examines the speedupof GAGA as well as its practicability.
Revised: Meanwhile, we also evaluate the running efficiency between GAGA and existing SPL implementations in different settings, which examines the speedup of GAGA as well as its practicability.

Original: Finally, we count the number of restarts and different types ofcritical points when using GAGA , to investigate its ability to address critical points.
Revised: Finally, we count the number of restarts and different types of [Figure] critical points when using GAGA , to investigate its ability to address critical points.

Original: For SVM, we usethe Gaussian kernel K ( x 1 , x 2 ) =
Revised: For SVM, we use the Gaussian kernel K ( x 1 , x 2 ) =

Original: exp( − γ κ ∥ x 1 − x 2 ∥ 2 ) .
Revised: exp( − γ κ ∥ x 1 − x 2 ∥ 2 ) .

Original: More details can be found in Appendix D.
Revised: More details can be found in Appendix D.

Original: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: Figure 4 illustrates that conventional ACS fails to reducethe generalization error due to heavy noises fed to the model at alarge age ( i.e. , overfits the dataset), while GAGA makes up for thisshortcoming by selecting the optimal model that merely learns thetrust-worthy samples during the continuous learning process.
Revised: Figure 4 illustrates that conventional ACS fails to reduce the generalization error due to heavy noises fed to the model at a large age ( i.e. , overfits the dataset), while GAGA makes up for this shortcoming by selecting the optimal model that merely learns the trust-worthy samples during the continuous learning process.

Original: Table 2 and 3 demonstrate an overall performance enhancing in GAGAthan competing algorithms.
Revised: Table 2 and 3 demonstrate an overall performance enhancing in GAGA than competing algorithms.

Original: The ‘†’ in tables denotes 30% of artificial noise, while ‘‡’ represents 20%.
Revised: The ‘†’ in tables denotes 30% of artificial noise, while ‘‡’ represents 20%.

Original: Note that performancesare measured by accuracy and generalization error for classification and regression, respectively.
Revised: Note that performances are measured by accuracy and generalization error for classification and regression, respectively.

Original: The results guarantee that GAGAoutperforms the state-of-the-art approaches in SPL under differentcircumstances.
Revised: The results guarantee that GAGA outperforms the state-of-the-art approaches in SPL under different circumstances.

Original: Figure 5 shows that GAGA also enjoys a high computational efficiency by changing the sample size as well as thepredefined age sequence, emphasizing the potentials of utilizing GAGA in practice.
Revised: Figure 5 shows that GAGA also enjoys a high computational efficiency by changing the sample size as well as the predefined age sequence, emphasizing the potentials of utilizing GAGA in practice.

Original: The number ofdifferent types of critical points on some datasets is given in Figure 6.
Revised: The number of different types of critical points on some datasets is given in Figure 6.

Original: Corresponding restarting timescan be found in Table 2 and 3, hence indicate that GAGA is capable of identifying different types ofcritical points and uses the heuristic trick to avoids restarts at massive turning points.
Revised: Corresponding restarting times can be found in Table 2 and 3, hence indicate that GAGA is capable of identifying different types of critical points and uses the heuristic trick to avoids restarts at massive turning points.

Original: Additional Experiments in Appendix D.
Revised: Additional Experiments in Appendix D.

Original: We further demonstrate the ability of GAGA to addressthe relatively large sample size and present more histograms.
Revised: We further demonstrate the ability of GAGA to address the relatively large sample size and present more histograms.

Original: In addition, we apply GAGA to thelogistic regression [33] for classification.
Revised: In addition, we apply GAGA to the logistic regression [33] for classification.

Original: We also verify that conventional ACS indeed tracks anapproximation path of partial optimum in experiments, which provides a more in-depth understandingtowards SPL and the performance promotion brought by GAGA .
Revised: We also verify that conventional ACS indeed tracks an approximation path of partial optimum in experiments, which provides a more in-depth understanding towards SPL and the performance promotion brought by GAGA .

Original: We also conduct a comparative studyto the state-of-the-art robust model for SVMs [48, 49, 50] and Lasso [51] besides the SPL domain.
Revised: We also conduct a comparative study to the state-of-the-art robust model for SVMs [48, 49, 50] and Lasso [51] besides the SPL domain.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we connect the SPL paradigm to the partial optimum for the first time.
Revised: In this paper, we connect the SPL paradigm to the partial optimum for the first time.

Original: Using this idea,we propose the first exact age-path algorithm able to tackle general SP-regularizer, namely GAGA .
Revised: Using this idea, we propose the first exact age-path algorithm able to tackle general SP-regularizer, namely GAGA .

Original: Experimental results demonstrate GAGA outperforms traditional SPL paradigm and the state-of-the-artage-path approach in many aspects, especially in the highly noisy environment.
Revised: Experimental results demonstrate GAGA outperforms traditional SPL paradigm and the state-of-the-art age-path approach in many aspects, especially in the highly noisy environment.

Original: We further build therelationship between our framework and existing theoretical analyses on SPL regime, which providesmore in-depth understanding towards the principle behind SPL.Checklist
Revised: We further build the relationship between our framework and existing theoretical analyses on SPL regime, which provides more in-depth understanding towards the principle behind SPL.

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes]

Original: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Appendix E.(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix D.2.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Table 1.

Original: (b) Did you mention the license of the assets?

Original: [Yes] Licenses are available referring to the

Original: provided links.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: This paper does not use crowdsourcing.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: This paper does not use crowdsourcing.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: This paper does not use crowdsourcing.

Original: Appendix 
Revised: Appendix A Supplementary Notes for Section

Original: A Supplementary Notes for Section 3

Original: In this section, we present some additional discussions and theoretical results for Section 3.
Revised: In this section, we present some additional discussions and theoretical results for Section 3.

Original: A.1 Explicit Expression
Revised: A.1 Explicit Expression

Original: The goal of this section is to obtain precise and explicit expressions of the Jacobian matrices in (6).
Revised: The goal of this section is to obtain precise and explicit expressions of the Jacobian matrices in (6).

Original: Firstly, the J − 1 w , t R , t ℓ has the form of
Revised: Firstly, the J − 1 w , t R , t ℓ has the form of

Original: [Equation]
Revised: [Equation]

Original: 13) in which 
Revised: in which ˜

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: The symbol ++ denotes column matrix concatenation, following the convention in Haskell Language.
Revised: The symbol ++ denotes column matrix concatenation, following the convention in Haskell Language.

Original: Similarly, J λ is given by
Revised: Similarly, J λ is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: As an outcome, (6) can be explicitly expressed via
Revised: As an outcome, (6) can be explicitly expressed via

Original: [Equation]
Revised: [Equation]

Original: A.2 On the Numerical ODEs Solving
Revised: A.2 On the Numerical ODEs Solving

Original: Matrix Inversion.
Revised: Matrix Inversion.

Original: Here we give some relative discussions to support the assumption in Section3.2 that J w , t R , t ℓ ( J for short) is non-singular.
Revised: Here we give some relative discussions to support the assumption in Section 3.2 that J w , t R , t ℓ ( J for short) is non-singular.

Original: First note that J is singular if and only if the valueof its determinant | J | is zero, where | J | is indeed a polynomial w.r.t.
Revised: First note that J is singular if and only if the value of its determinant | J | is zero, where | J | is indeed a polynomial w.r.t.

Original: the uncertain elements in J .
Revised: the uncertain elements in J .

Original: Denote the number of these unknown components as q , then the probability that J is singular canbe somewhat equivalently seen as the measure of the hypersurface S = { x ∈ R q : | J ( x ) | = 0 } in522 R q .
Revised: Denote the number of these unknown components as q , then the probability that J is singular can be somewhat equivalently seen as the measure of the hypersurface S = { x ∈ R q : | J ( x ) | = 0 } in R q .

Original: For the polynomial function | J | , it’s easy to prove that S is a zero-measured set in R q , which indicates the probability of J being non-invertible is zero.
Revised: For the polynomial function | J | , it’s easy to prove that S is a zero-measured set in R q , which indicates the probability of J being non-invertible is zero.

Original: This result shows the non-singularityassumption fits the common situation in practice.
Revised: This result shows the non-singularity assumption fits the common situation in practice.

Original: Secondly, there are some general cases that J isguaranteed to be invertible.
Revised: Secondly, there are some general cases that J is guaranteed to be invertible.

Original: We refer some of these claims to [40].
Revised: We refer some of these claims to [40].

Original: Moreover, during the extensiveempirical studies, none of the singular J is observed, which again validates the rationality of theassumption.
Revised: Moreover, during the extensive empirical studies, none of the singular J is observed, which again validates the rationality of the assumption.

Original: Robustness.
Revised: Robustness.

Original: In practice, we avoid directly computing the inverse of J by the consideration ofrobustness.
Revised: In practice, we avoid directly computing the inverse of J by the consideration of robustness.

Original: Instead, we adopt the Moore-Penrose inverse [52] in implementation.
Revised: Instead, we adopt the Moore-Penrose inverse [52] in implementation.

Original: The Moore-Penroseinverse exists for any matrix X even if the matrix owns singularity, which guarantees our algorithmto be robust.
Revised: The Moore-Penrose inverse exists for any matrix X even if the matrix owns singularity, which guarantees our algorithm to be robust.

Original: Complexity.
Revised: Complexity.

Original: Our approach utilizes the singular value decomposition (SVD) [52] when solvingthe Moore-Penrose inverse in (15), which is demonstrated to be a state-of-the-art technique via acomputationally simple and precise way [53].
Revised: Our approach utilizes the singular value decomposition (SVD) [52] when solving the Moore-Penrose inverse in (15), which is demonstrated to be a state-of-the-art technique via a computationally simple and precise way [53].

Original: Consequently, the computational efficiency as well asthe accuracy of our algorithm is guaranteed.
Revised: Consequently, the computational efficiency as well as the accuracy of our algorithm is guaranteed.

Original: Stability.
Revised: Stability.

Original: Our algorithm uses ODE solvers from the LSODE package [54] when solving the initialvalue problem (6).
Revised: Our algorithm uses ODE solvers from the LSODE package [54] when solving the initial value problem (6).

Original: The solver will automatically select a proper method to solve different initial valueproblems which guarantees the general performance of our algorithm.
Revised: The solver will automatically select a proper method to solve different initial value problems which guarantees the general performance of our algorithm.

Original: Especially, when the problemtends to be unstable, the solver adopts the backward differentiation formula (BDF) method [55] toavoid extremely small step sizes while preserving the stability and accuracy of output solutions.
Revised: Especially, when the problem tends to be unstable, the solver adopts the backward differentiation formula (BDF) method [55] to avoid extremely small step sizes while preserving the stability and accuracy of output solutions.

Original: B Proofs
Revised: B Proofs

Original: In this section, we give complete proofs to all the theorems and properties stated in the main article.
Revised: In this section, we give complete proofs to all the theorems and properties stated in the main article.

Original: B.1 Proof of Theorem2
Revised: B.1 Proof of Theorem

Original: Prior to the proof, we first review the relevant background about latent SPL loss [38].
Revised: Prior to the proof, we first review the relevant background about latent SPL loss [38].

Original: Regarding theunconstrained learning problem (1), its latent SPL objective is defined as[-
-]
Revised: Regarding the unconstrained learning problem (1), its latent SPL objective is defined as[-
-]

Original: G λ ( w ) : = m (cid:88) j = α j R j ( w ) + n (cid:88) i =
Revised: [Equation]

Original: F λ ( ℓ i ( w )) , (16) where F λ ( ℓ ) =
Revised: (16) where F λ ( ℓ ) =

Original: (cid:82) ℓ 0 v ∗
Revised: (cid:82) ℓ 0 v ∗

Original: λ ( τ ) dτ .
Revised: λ ( τ ) dτ .

Original: Theorem 4.
Revised: Theorem 4.

Original: In the SPL objective (2), suppose ℓ is bounded below, w (cid:55)→ ℓ ( · ) is continuouslydifferentiable, v ∗ λ ( · ) is continuous, and (cid:80) mj
Revised: In the SPL objective (2), suppose ℓ is bounded below, w (cid:55)→ ℓ ( · ) is continuously differentiable, v ∗ λ ( · ) is continuous, and (cid:80) mj

Original: =

Revised: =

Original: α j R j is coercive and lower semi-continuous.
Revised: α j R j is coercive and lower semi-continuous.

Original: Then forany initial parameter w 0 , every cluster point of the produced sequence (cid:8) w k (cid:9) , obtained by the ACSalgorithm on solving (2), is a critical point of the implicit objective G λ (16).
Revised: Then for any initial parameter w 0 , every cluster point of the produced sequence (cid:8) w k (cid:9) , obtained by the ACS algorithm on solving (2), is a critical point of the implicit objective G λ (16).

Original: In Theorem 2, the relationship between the partial optimum of original SPL objective L and thecritical point of implicit SPL objective G λ is constructed.
Revised: In Theorem 2, the relationship between the partial optimum of original SPL objective L and the critical point of implicit SPL objective G λ is constructed.

Original: Its proof is given as follows.
Revised: Its proof is given as follows.

Original: Proof.
Revised: Proof.

Original: On one hand,w
Revised: On one hand,

Original: [Equation]
Revised: [Equation]

Original: is a critical point [Equation] On the other hand, assuming ( w 0 , v ) is a partial optimum of the SPL objective L , it’s obvious that v i
Revised: On the other hand, assuming ( w 0 , v ) is a partial optimum of the SPL objective L , it’s obvious that v i

Original: = v ∗ λ ( l i ( w 0 ))
Revised: = v ∗ λ ( l i ( w 0 ))

Original: (for i = 1 . . . N ), where we write v = v ∗ λ ( w 0 ) in short.
Revised: (for i = 1 . . . N ), where we write v = v ∗ λ ( w 0 ) in short.

Original: Then ( w 0 , v ∗ λ ( w 0 )) is a partial optimum of L
Revised: Then ( w 0 , v ∗ λ ( w 0 )) is a partial optimum of L

Original: [Equation]
Revised: [Equation]

Original: [Equation] Combine the above two results we can conclude Theorem 2.
Revised: Combine the above two results we can conclude Theorem [Equation]

Original: B.2 Proof of Theorem
Revised: B.2 Proof of Theorem

Original: Proof.
Revised: Proof.

Original: In Section 3.2, we have shown that any ( w , v ∗ ( ℓ ( w ) , λ )) is a partial optimum iff thereexists t R , t ℓ such that (5) holds.
Revised: In Section 3. [Equation] we have shown that any ( w , v ∗ ( ℓ ( w ) , λ )) is a partial optimum iff there exists t R , t ℓ such that (5) holds.

Original: Given a certain partial optimum ( ˆ w , v ∗ ( ℓ ( ˆ w ) , λ )) , solving thecorresponding ˆ t R , ˆ t ℓ is indeed calculating the linear equationsm (cid:88)
Revised: Given a certain partial optimum ( ˆ w , v ∗ ( ℓ ( ˆ w ) , λ )) , solving the corresponding ˆ t R [Equation] ˆ t ℓ is indeed calculating the linear equations

Original: [Equation]
Revised: [Equation]

Original: On a non-critical point, suppose we’ve obtained a partial optimum ( ˆ w , v ∗ ( ℓ ( ˆ w ) , λ )) at λ .
Revised: On a non-critical point, suppose we’ve obtained a partial optimum ( ˆ w , v ∗ ( ℓ ( ˆ w ) , λ )) at λ .

Original: Now acritical point is triggered by either of two conditions: 1) Partition P changes.
Revised: Now a critical point is triggered by either of two conditions: 1) Partition P changes.

Original: This means the value ofsome ℓ i lie on the boundary between two distinct sets in P . 2)
Revised: This means the value of some ℓ i lie on the boundary between two distinct sets in P . 2)

Original: One of I R , I ℓ changes.
Revised: One of I R , I ℓ changes.

Original: This indicatesthe existence of some R i ( i ∈ ¯ I R ( I R ) ) becomes (non-)differentiable at ˆ w , or some ℓ j ( j ∈ ¯ I ℓ ( I ℓ ) )becomes (non-)differentiable at ˆ w .
Revised: This indicates the existence of some R i ( i ∈ ¯ I R ( I R ) ) becomes (non-)differentiable at ˆ w , or some ℓ j ( j ∈ ¯ I ℓ ( I ℓ ) ) becomes (non-)differentiable at ˆ w .

Original: The latter can be detected by the value of t i .
Revised: The latter can be detected by the value of t i .

Original: For example, assumethat i ∈ I ℓ holds along a segment of the path, i.e., there exists k such that t kℓ i = 1 , while t ˜ kℓ i = 1holds for all ˜ k ̸
Revised: For example, assume that i ∈ I ℓ holds along a segment of the path, i.e., there exists k such that t kℓ i = 1 , while t ˜ kℓ i =holds for all ˜ k ̸

Original: = k.
Revised: = k.

Original: At the kink, ℓ i changes into non-differentiable.
Revised: At the kink, ℓ i changes into non-differentiable.

Original: As a result, the value of t kℓ i willdecrease from 1, since some other selection functions turn to essentially active status.
Revised: As a result, the value of t kℓ i will decrease from 1, since some other selection functions turn to essentially active status.

Original: Altogether,at the optimal ˆ w , all the inequalities in (5) are strict .
Revised: Altogether, at the optimal ˆ w , all the inequalities in (5) are strict .

Original: In case that F ( w , λ, t R , t ℓ ) =0 deduces acontinuous solution path passing ( ˆ w , ˆ t R , ˆ t ℓ ) , the (5) will be maintained along the path until the nextcritical point occurs.
Revised: In case that F ( w , λ, t R , t ℓ ) = deduces a continuous solution path passing ( ˆ w , ˆ t R , ˆ t ℓ ) , the (5) will be maintained along the path until the next critical point occurs.

Original: Denote the Jacobin of F w.r.t.
Revised: Denote the Jacobin of F w.r.t.

Original: ( w , t R , t ℓ ) , λ as J w , t R , t ℓ , J λ , respectively.
Revised: ( w , t R , t ℓ ) , λ as J w , t R , t ℓ , J λ , respectively.

Original: Following our settingin Section 3.2, F is C 1 and J w , t R , t ℓ is invertible at the initial point.
Revised: Following our setting in Section 3.2, F is C 1 and J w , t R , t ℓ is invertible at the initial point.

Original: In this condition, the implicitfunction theorem directly indicates the existence and uniqueness of a local C 1 solution path of optimal( ˆ w , ˆ t R , ˆ t ℓ ) w.r.t.
Revised: In this condition, the implicit function theorem directly indicates the existence and uniqueness of a local C 1 solution path of optimal ( ˆ w , ˆ t R , ˆ t ℓ ) w.r.t.

Original: λ , started from the initial point.
Revised: λ , started from the initial point.

Original: Furthermore, the theorem also guarantees that(6) is valid along the path.
Revised: Furthermore, the theorem also guarantees that (6) is valid along the path.

Original: Owing to the C 1 function F , the right side of (6) is continuous w.r.t.λ .
Revised: Owing to the C 1 function F , the right side of (6) is continuous w.r.t.

Original:  Therefore, the Picard–Lindelöf theorem [56] straightforwardly proves that the solution of (6) isunique and can be extended to the nearest boundary of λ ( i.e. , a new critical point appears).
Revised: λ . Therefore, the Picard–Lindelöf theorem [56] straightforwardly proves that the solution of (6) is unique and can be extended to the nearest boundary of λ ( i.e. , a new critical point appears).

Original: Geometric Intuition.
Revised: Geometric Intuition.

Original: There exists a geometric understanding towards (5) either.
Revised: There exists a geometric understanding towards (5) either.

Original: Rewriting the firstequation in (5) by the differentiability of each function gives− (cid:88)
Revised: Rewriting the first equation in (5) by the differentiability of each function gives −

Original: [Equation]
Revised: [Equation]

Original: where ˆ t k R j , ˆ t kℓ i meet restrictions in (5) and all symbols are in terms of ˆ w .
Revised: where ˆ t k R j , ˆ t kℓ i meet restrictions in (5) and all symbols are in terms of ˆ w .

Original: As λ varies, theleft side in (18) describes a smooth curve using the standard frame in R d , while the rightside is actually sum of vectors chosen from a convex hull conv(cid:18) (cid:83)
Revised: As λ varies, the left side in (18) describes a smooth curve using the standard frame in R d , while the right side is actually sum of vectors chosen from a convex hull conv

Original: k ∈
Revised: [Equation]

Original: I
Revised: conv (cid:16)(cid:83)

Original: [Equation]
Revised: [Equation]

Original: R j { α j
Revised: v

Original: [Equation]
Revised: [Equation]

Original: (cid:110) D k R j (cid:111) } (cid:19) orconv (cid:16)(cid:83)
Revised: ( ℓ i , λ )

Original: k ∈
Revised: [Equation]

Original: I aℓi (cid:8) v ∗ ( ℓ i , λ ) ∇ D kℓ i (cid:9)(cid:17) and can be viewed as the vector 1 under an analogue movingframe made up of these selected vectors, from the perspective of differential geometry.
Revised: D kℓ i (cid:9)(cid:17) and can be viewed as the vector 1 under an analogue moving frame made up of these selected vectors, from the perspective of differential geometry.

Original: In otherwords, (18) indeed uses an analogue moving frame to re-depict a smooth curve.
Revised: In other words, (18) indeed uses an analogue moving frame to re-depict a smooth curve.

Original: It is worth adding that a recent work [40] shows a similar intuition.
Revised: It is worth adding that a recent work [40] shows a similar intuition.

Original: Specifically, under someassumptions the solution surface near a given point is a projection of certain smooth manifold evenwithout the convexity assumption.
Revised: Specifically, under some assumptions the solution surface near a given point is a projection of certain smooth manifold even without the convexity assumption.

Original: B.3 Proof of Corollary1
Revised: B.3 Proof of Corollary

Original: Proof.
Revised: Proof.

Original: Suppose that R j , ℓ i in (1) are all differentiable at the given ˆ w , then it yields I R
Revised: Suppose that R j , ℓ i in (1) are all differentiable at the given ˆ w , then it yields I R

Original: = I ℓ = ∅ .
Revised: = I ℓ = ∅ .

Original: Consequently, (5) is degenerate into its first equation[Equation]
Revised: Consequently, (5) is degenerate into its first equation

Original: [Equation]
Revised: [Equation]

Original: Hence applying Theorem 3 straightly gives our Corollary 1.
Revised: Hence applying Theorem 3 straightly gives our Corollary 1.

Original: B.4 Proof of Prop.1
Revised: B.4 Proof of Prop.

Original: Here we only present the detailed proof for SVM with linear SP-regularizer, due to the fact that theproof for mixture SP-regularizer is almost the same as the former except some partition difference.
Revised: Here we only present the detailed proof for SVM with linear SP-regularizer, due to the fact that the proof for mixture SP-regularizer is almost the same as the former except some partition difference.

Original: Proof.
Revised: Proof.

Original: Given a partial optimum (( ˆ w , b ) v ∗ ( ˆ w , b )) at λ , (5) can be directly applied here with obvioussimplifications.
Revised: Given a partial optimum (( ˆ w , b ) v ∗ ( ˆ w , b )) at λ , (5) can be directly applied here with obvious simplifications.

Original: Mathematically, there exits ˆ t = (cid:0)
Revised: Mathematically, there exits ˆ t = (cid:0)

Original: ˆ t i (cid:1) n i =1 such that
Revised: ˆ t i (cid:1) n i =1 such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ⪯ denotes the element-wise comparison between vectors.
Revised: where ⪯ denotes the element-wise comparison between vectors.

Original: Denote ˆ α = C v ∗ ⊙ ˆ t , then (20) canbe equivalently converted to equations w.r.t.
Revised: Denote ˆ α = C v ∗ ⊙ ˆ t , then (20) can be equivalently converted to equations w.r.t.

Original: (ˆ α , b ) as
Revised: (ˆ α , b ) as

Original: [Equation]
Revised: [Equation]

Original: where Q = y T K y , K =
Revised: where Q = y T K y , K =

Original: ( k ( x i , x j )) 1 ≤ i,j ≤ n is the kernel matrix and k is the kernel function.
Revised: ( k ( x i , x j )) 1 ≤ i,j ≤ n is the kernel matrix and k is the kernel function.

Original: Thenthe optimal ˆ w = (cid:80) ni
Revised: Then the optimal ˆ w = (cid:80) ni

Original: =

Revised: =

Original: ˆ α
Revised: ˆ α

Original: i
Revised: i

Original: y
Revised: y

Original: i ϕ
Revised: i ϕ

Original: ( x i ) , hence problem (7) is transformed into solving equations merelyrelated to (ˆ α , b ) .
Revised: ( x i ) , hence problem (7) is transformed into solving equations merely related to (ˆ α , b ) .

Original: Specifically, the decision function can be rewritten as d ( x ) = (cid:80) ni
Revised: Specifically, the decision function can be rewritten as d ( x ) = (cid:80) ni

Original: =

Revised: =

Original: y i ˆ
Revised: y i ˆ

Original: α i k
Revised: α i k

Original: ( x i , x )+ b.
Revised: ( x i , x )+ b.

Original: Now supposed that (ˆ α , ˆ b ) is not a critical point, then the fourth equation in (21) is actually aninequality constraint by changing the value of ˆ t E Z in [0 , 1] |E Z | .
Revised: Now supposed that (ˆ α , ˆ b ) is not a critical point, then the fourth equation in (21) is actually an inequality constraint by changing the value of ˆ t E Z in [0 , 1] |E Z | .

Original: As a result, (21) is only relatedto (ˆ α , ˆ b ) and the left side of the first three equations accords with the function F in Theorem 3.
Revised: As a result, (21) is only related to (ˆ α , ˆ b ) and the left side of the first three equations accords with the function F in Theorem 3.

Original: Consequently, (8) can be derived using Theorem 3.
Revised: Consequently, (8) can be derived using Theorem 3.

Original: In detail, denote
Revised: In detail, denote

Original: [Equation]
Revised: [Equation]

Original: then the jacobian can be calculated as
Revised: then the jacobian can be calculated as

Original: [Equation]
Revised: [Equation]

Original: The implicit function theorem immediately indicates the following ODEs hold
Revised: The implicit function theorem immediately indicates the following ODEs hold

Original: [Equation]
Revised: [Equation]

Original: B.5 Proof of Prop.
Revised: B.5 Proof of Prop.

Original: Proof.
Revised: Proof.

Original: Following discussions in Section 4.2, the objective (10) of Lasso is reformulated under theself-paced paradigm as
Revised: Following discussions in Section 4. [Equation] the objective (10) of Lasso is reformulated under the self-paced paradigm as

Original: [Equation]
Revised: [Equation]

Original: (22) where V ∗ denotes Diag { v ∗ } and √ V ∗ denotes Diag (cid:8) √ v ∗ (cid:9) .
Revised: (22) where V ∗ denotes Diag { v ∗ } and √ V ∗ denotes Diag (cid:8) √ v ∗ (cid:9) .

Original: Given a partial optimum ˆ w at λ ,applying (5) to the objective (10) deduces
Revised: Given a partial optimum ˆ w at λ , applying (5) to the objective (10) deduces

Original: [Equation]
Revised: [Equation]

Original: (23) where − 1 ¯ A ⪯ ˆ t ¯ A ⪯ 1 ¯ A and w ¯ A = 0 ¯ A .
Revised: where − 1 ¯ A ⪯ ˆ t ¯ A ⪯ 1 ¯ A and w ¯ A = 0 ¯ A .

Original: Suppose that ˆ w is not a critical point, the second equation isindeed converted to an inequality constraint via varying ˆ t ¯ A in [ − 1 , 1] | ¯ A| .
Revised: Suppose that ˆ w is not a critical point, the second equation is indeed converted to an inequality constraint via varying ˆ t ¯ A in [ − 1 , 1] | ¯ A| .

Original: As a result, (23) is merelyin connection with w A and the left side in the first equation consists with the function F in Theorem3.
Revised: As a result, (23) is merely in connection with w A and the left side in the first equation consists with the function F in Theorem 3.

Original: Take the mixture SP-regularizer as an example, the optimality of estimation when using mixture
Revised: Take the mixture SP-regularizer as an example, the optimality of estimation when using mixture

Original: SP-regularizer is described as
Revised: SP-regularizer is described as

Original: [Equation]
Revised: [Equation]

Original: For the sake of simplicity, we derive the result of d Z ( ˆ w A , λ ) dλ first.
Revised: For the sake of simplicity, we derive the result of d Z ( ˆ w A , λ ) dλ first.

Original: Note that l k =
Revised: Note that l k =

Original: ( x k ˆ w A − y k ) 2 .d [Equation]
Revised: ( x k ˆ w A − y k ) 2 . d Z ( ˆ w A

Original: [Equation]
Revised: [Equation]

Original: A
Revised: λ )

Original: [Equation]
Revised: [Equation]

Original: Similar to the proof of Theorem 3, we have
Revised: Similar to the proof of Theorem 3, we have

Original: [Equation]
Revised: [Equation]

Original: And final result comes from combining and vectoring the terms w.r.t.
Revised: And final result comes from combining and vectoring the terms w.r.t.

Original: d ˆ w A dλ , which can be utilized toderive the Prop. 2.
Revised: d ˆ w A dλ , which can be utilized to derive the Prop. 2.

Original: C Detailed Algorithms
Revised: C Detailed Algorithms

Original: In this section, we present more details of the concrete algorithms derived for the SVM and Lasso.
Revised: In this section, we present more details of the concrete algorithms derived for the SVM and Lasso.

Original: C.1 Support Vector Machines
Revised: C.1 Support Vector Machines

Original: The goal of GAGA is to calculate the age-path on the interval [ λ min , λ max ] .
Revised: The goal of GAGA is to calculate the age-path on the interval [ λ min , λ max ] .

Original: As mentioned in Section3, started from an initial point, the algorithm solves the derived ODEs (8) while examining all thepartitions along the way of λ .
Revised: As mentioned in Section 3, started from an initial point, the algorithm solves the derived ODEs (8) while examining all the partitions along the way of λ .

Original: In SVM, partition violating is merely caused by the change in g ( c.f.
Revised: In SVM, partition violating is merely caused by the change in g ( c.f.

Original: Section 4.1).
Revised: Section 4.1).

Original: For example, some g i varying from a negative value to zero will lead the violation of
Revised: For example, some g i varying from a negative value to zero will lead the violation of

Original: E N , resulting in a critical point.
Revised: E N , resulting in a critical point.

Original: In case that the point belongs to a turning point, the only need is toresign i from E N to E Z .
Revised: In case that the point belongs to a turning point, the only need is to resign i from E N to E Z .

Original: Otherwise the point is a jump point and we have to perform the warm startto re-calculate the next solution, which could be time consuming.
Revised: Otherwise the point is a jump point and we have to perform the warm start to re-calculate the next solution, which could be time consuming.

Original: Since it’s non-trivial to identifythe type of the critical point as a prior, we adopt a heuristic operation to avoid excessive warm starts.
Revised: Since it’s non-trivial to identify the type of the critical point as a prior, we adopt a heuristic operation to avoid excessive warm starts.

Original: When the partition violation occurs, we directly resign all the violated indexes into the right statusby the partition rule.
Revised: When the partition violation occurs, we directly resign all the violated indexes into the right status by the partition rule.

Original: Suppose that a turning point is encountered, the solutions return by numerical
Revised: Suppose that a turning point is encountered, the solutions return by numerical

Original: ODEs solver with the updated index sets will keep the KKT condition, allowing our algorithm toproceed.
Revised: ODEs solver with the updated index sets will keep the KKT condition, allowing our algorithm to proceed.

Original: The algorithmic steps are given in Algorithm 3.
Revised: The algorithmic steps are given in Algorithm 3.

Revised: Algorithm 3 GAGA for SVM

Original: Input : Initial solution ( α , b ) | λ t = λ min , X , y , λ min and λ max
Revised: Input : Initial solution ( α , b ) | λ t = λ min , X , y , λ min and λ max

Original: Parameter :
Revised: Parameter :

Original: Cost parameter C
Revised: Cost parameter C

Original: Output : Age-path ( α , b ) on [ λ min , λ max ] 1: λ t ← λ min , set E N , E Z , E P , D ( , M ) in Proposition 1 according to w | λ t = λ min 2: while λ t ≤ λ max do 3:
Revised: Output : Age-path ( α , b ) on [ λ min , λ max ] 1: λ t ← λ min , set E N , E Z , E P , D ( , M ) in Proposition 1 according to w | λ t = λ min 2: while λ t ≤ λ max do 3:

Original: Solve (8) or (9) and partition samples in X and components of α simultaneously.
Revised: Solve (8) or (9) and partition samples in X and components of α simultaneously.

Original: 4: if Partition E N , E Z , E P , D ( , M ) was not met then 5:
Revised: 4: if Partition E N , E Z , E P , D ( , M ) was not met then 5:

Original: Resign violated indexes in P by g i .
Revised: Resign violated indexes in P by g i .

Original: 6: end if 7: α E N = 0 E N , α D = 0 D .
Revised: 6: end if 7: α E N = 0 E N , α D = 0 D .

Original: For the mixture regularizer, α E P = 1 E P .
Revised: For the mixture regularizer, α E P = 1 E P .

Original: Solve (8) or (9) with updated E N , E Z , E P , D ( , M ) 9: if KKT conditions are not met then 10:
Revised: Solve (8) or (9) with updated E N , E Z , E P , D ( , M ) 9: if KKT conditions are not met then 10:

Original: Warm start at λ t + δ (for a small δ > 0 ).
Revised: Warm start at λ t + δ (for a small δ > 0 ).

Original: 11: end if 12: end while
Revised: 11: end if 12: end while

Original: C.2 Lasso
Revised: C.2 Lasso

Original: In Lasso, the main routine of GAGA is similar with that in SVM.
Revised: In Lasso, the main routine of GAGA is similar with that in SVM.

Original: The only difference is that here weneed to examine the partition A additionally.
Revised: The only difference is that here we need to examine the partition A additionally.

Original: Due to the property of the ℓ 1 norm, monitoring A isoperated by observing if the value of w i equals to zero or conversely, whether the subgradient ofinactive component is reached to 1 .
Revised: Due to the property of the ℓ 1 norm, monitoring A is operated by observing if the value of w i equals to zero or conversely, whether the subgradient of inactive component is reached to 1 .

Original: The details of the algorithms are shown in Algorithm 4.
Revised: The details of the algorithms are shown in Algorithm 4.

Original: D Additional Results
Revised: D Additional Results

Original: In this section, we present additional experimental results on the logistic regression and path consistency to obtain a more comprehensive evaluation of proposed algorithm.
Revised: In this section, we present additional experimental results on the logistic regression and path consistency to obtain a more comprehensive evaluation of proposed algorithm.

Original: D.1 Logistic Regression
Revised: D.1 Logistic Regression

Original: Given the dataset X and label y , the logistic regression gives the optimization problem as (24)
Revised: Given the dataset X and label y , the logistic regression gives the optimization problem as (24)

Original: [Equation]
Revised: [Equation]

Original: (24) where C > 0 is the trade-off parameter.
Revised: (24) where C > 0 is the trade-off parameter.

Original: Note the objective (24) is smooth on the entire domain,hence we apply Corollary 1 to derive the ODEs and the only P is needde to be tracked and reset Input
Revised: Note the objective (24) is smooth on the entire domain, hence we apply Corollary 1 to derive the ODEs and the only P is needde to be tracked and reset α

Original: Parameter : strength α Output :Age- w λ , λ max ] 1: λt ← λ min , set A , P in Proposition 2 according to w | λ t= λ min 2: while λt ≤ λ max do 3:
Revised: Output : max ] 1: λ min , set A , P in Proposition 2 according to w | λ t min 2: while λ ≤ λ max do 3:

Original: Solve (11) or (12)and partition w A , X A simultaneously.
Revised: Solve (11) or (12) partition w A , X A simultaneously.

Original: 4: if Partition A
Revised: 4: if Partition A

Original: Pnot met then 5: if A was not met then 6: if k -th element turns to inactive then 7: wk = 0 .
Revised: P met then 5: if A was not met then 6: if k -th element turns to inactive then 7: w = 0 .

Original: Remove k from A .
Revised: Remove k from A .

Original: 9: else if k -th element becomes active then 10:
Revised: 9: else if k -th element becomes active then 10:

Original: Put k into A .
Revised: Put k into A .

Original: 11: end if 12: end if 13: if P was not met then 14:
Revised: 11: end if 12: end if 13: if P was not met then 14:

Original: Resign violated indexes in P by ℓ i .
Revised: Resign violated indexes in P by ℓ i .

Original: 15: end if 16:
Revised: 15: end if 16:

Original: Solve (11) or (12) with updated A , P .
Revised: Solve (11) or (12) with updated A , P .

Original: 17: if KKT conditions are not met then 18:
Revised: 17: if KKT conditions are not met then 18:

Original: Warm start at λ t + δ (for a small δ > 0 ).
Revised: Warm start at λ t + δ (for a small δ > 0 ).

Original: 19: end if 20: end if 21: end while along the path.
Revised: 19: end if 20: end if 21: end while along the path.

Original: To start with, let ℓ i
Revised: To start with, let ℓ i

Original: =
Revised: =

Original: C ln(1 + e −
Revised: C ln(1 + e −

Original: y
Revised: y

Original: i
Revised: i

Original: ( X i w + b ) ) , then for the linear SP-regularizer, thepartition P = {E , D} , where E = { 1 ≤ i ≤ n : ℓ i < λ } , D = { 1 ≤ i ≤ n : ℓ i ≥ λ } .
Revised: ( X i w + b ) ) , then for the linear SP-regularizer, the partition P = {E , D} , where E = { 1 ≤ i ≤ n : ℓ i < λ } , D = { 1 ≤ i ≤ n : ℓ i ≥ λ } .

Original: For themixture SP-regularizer, P = {E , M , D} , where E = { 1 ≤ i ≤ n : ℓ i < (cid:16) λγλ + γ (cid:17) 2 } , M = { 1 ≤ i ≤n : (cid:16) λγλ + γ (cid:17) 2 ≤ ℓ i ≤ λ 2 } , and D = { 1 ≤ i ≤ n : ℓ i > λ 2 } .
Revised: For the mixture SP-regularizer, P = {E , M , D} , where E = { 1 ≤ i ≤ n : ℓ i < (cid:16) λγλ + γ (cid:17) 2 } , M = { 1 ≤ i ≤ n : (cid:16) λγλ + γ (cid:17) 2 ≤ ℓ i ≤ λ 2 } , and D = { 1 ≤ i ≤ n : ℓ i > λ 2 } .

Original: Applying Corollary 1 obtains Theorem5.
Revised: Applying Corollary 1 obtains Theorem 5.

Original: Theorem 5.
Revised: Theorem 5.

Original: When w , b indicate a partial optimum, the dynamics of optimal w , b in (24) w.r.t.
Revised: When w , b indicate a partial optimum, the dynamics of optimal w , b in (24) w.r.t.

Original: λ forthe linear and mixture SP-regularizer are shown as
Revised: λ for the linear and mixture SP-regularizer are shown as

Original: [Equation]
Revised: [Equation]

Original: where UE = Diag (cid:8) y
Revised: where U

Original: [Equation]
Revised: [Equation]

Original: [Equation] = [Equation]
Revised: where A = E ∪ M

Original: [Equation]
Revised: [Equation]

Original: where A = E ∪ M , U A =

Original: [-Diag (cid:8) y 
-]
Revised: [-Diag (cid:8) y 
-]

Original: A ⊙ u A
Revised: U A

Original: [Equation]
Revised: [Equation]

Original: , u E =

Original: [Equation]
Revised: [Table]

Original: Proof.
Revised: Proof.

Original: The proof is nearly the same as that of SVM and Lasso, hence we merely present the mainstructure in the following.
Revised: The proof is nearly the same as that of SVM and Lasso, hence we merely present the main structure in the following.

Original: The linear SP-regularizer is utilized during the derivation, while the proofof mixture SP-regularizer is quite similar.
Revised: The linear SP-regularizer is utilized during the derivation, while the proof of mixture SP-regularizer is quite similar.

Original: Given a partial optimum ( w , b ) at λ, (5) is rewritten in detail with the form of
Revised: Given a partial optimum ( w , b ) at λ, (5) is rewritten in detail with the form of

Original: [Equation]
Revised: [Equation]

Original: Similarly, we set the F as
Revised: Similarly, we set the F as

Original: [Equation]
Revised: [Equation]

Original: Afterwards, the corresponding jaconbian is derived as
Revised: Afterwards, the corresponding jaconbian is derived as

Original: [Equation]
Revised: [Equation]

Original:   [Equation] where U E =Diag (cid:8) y 2 E ⊙ u E (cid:9) , u E = (cid:18) ℓ E
Revised:   , where U E =

Original: [Equation] C λ −(cid:19) ⊙ e − [Equation] ℓ E C
Revised: Diag (cid:8) y

Original: [Equation] (cid:18) [Equation] E λ
Revised: 2 E ⊙ u E (cid:9) , u E =

Original: [Equation]
Revised: [Equation]

Original: (cid:19) ⊙ e − ℓ E C − C λ .

Original: Therefore, the implicit function theorem implies that
Revised: Therefore, the implicit function theorem implies that

Original: [Equation]
Revised: [Equation]

Original: D.2 Detailed Experimental Setting
Revised: D.2 Detailed Experimental Setting

Original: We use the Scikit-learn package [57] to optimize the subproblems of SVM, logistic regressionand Lasso.
Revised: We use the Scikit-learn package [57] to optimize the subproblems of SVM, logistic regression and Lasso.

Original: The MOSPL method is implemented using the toolbox geatpy [58].
Revised: The MOSPL method is implemented using the toolbox geatpy [58].

Original: All codes wereimplemented in Python and all experiments were conducted on a machine with 48 2.2GHz cores,80GB of RAM and 4 Nvidia 1080ti GPUs.
Revised: All codes were implemented in Python and all experiments were conducted on a machine with 48 2.2GHz cores, 80GB of RAM and 4 Nvidia 1080ti GPUs.

Original: In all experiments of performance comparison, we evaluate the average performance in 20 runs.
Revised: In all experiments of performance comparison, we evaluate the average performance in 20 runs.

Original: To maintain the reproducibility, the random seed is fixed with 40 .
Revised: To maintain the reproducibility, the random seed is fixed with 40 .

Original: In each trail, the each dataset israndomly divided into a training set and a testing set by the ratio of 3 : 1 .
Revised: In each trail, the each dataset is randomly divided into a training set and a testing set by the ratio of 3 : 1 .

Original: When carrying out GAGAand ACS, the predefined interval of λ is set to [0 . 1 , 20] , and the step size in ACS equals to 0 .
Revised: When carrying out GAGA and ACS, the predefined interval of λ is set to [0 . 1 , 20] , and the step size in ACS equals to 0 .

Original: We utilize the NSGA-III as the framework of MOSPL, in which N p is set to 150 and Gen = 800 9 .
Revised: We utilize the NSGA-III as the framework of MOSPL, in which N p is set to 150 and Gen = 800 10 .

Original: Whenapplying the mixture regularizer, we utilize the polynomials loss in [19] as ℓ i to transform the originalproblem into a multi-objective problem.
Revised: When applying the mixture regularizer, we utilize the polynomials loss in [19] as ℓ i to transform the original problem into a multi-objective problem.

Original: Afterwards, the polynomial order t is fixed at 1.2 and 1.35,respectively.
Revised: Afterwards, the polynomial order t is fixed at 1. and 1.35, respectively.

Original: D.3 Simulation Study on Logistic Regression
Revised: D.3 Simulation Study on Logistic Regression

Original: We present additional experimental results on the logistic regression (24) to validate our ODEs.
Revised: We present additional experimental results on the logistic regression (24) to validate our ODEs.

Original: The utilized datasets are listed in Table 4.
Revised: The utilized datasets are listed in Table 4.

Original: The averaged results using the linear and mixture SPDatasetSource Samples Dimensions
Revised: The averaged results using the linear and mixture SPDataset

Original: Task

Original: [Figure]

Original: UCI

Original: [Figure]
Revised: [Table]

Original: UCI  hiva agnostic

Original: OpenML  nomao

Original: OpenML MagicTelescope OpenML 685 regularizer are illustrated in Table 5 and 6, respectively, in which the performance is measured bythe classification accuracy.
Revised: regularizer are illustrated in Table 5 and 6, respectively, in which the performance is measured by the classification accuracy.

Original: Meanwhile, Figure 7 confirms the computational efficiency of GAGA onlarge-scale dataset.
Revised: Meanwhile, Figure 7 confirms the computational efficiency of GAGA on large-scale dataset.

Original: Taking all results into consideration, GAGA outperforms than the baseline methodson all datasets and parameter settings, hence demonstrates the performance of GAGA in classificationtasks with large data size.
Revised: Taking all results into consideration, GAGA outperforms than the baseline methods on all datasets and parameter settings, hence demonstrates the performance of GAGA in classification tasks with large data size.

Original: [Figure]
Revised: [Figure]

Original: predefined search space.

Original: D.4 Path consistency
Revised: D.4 Path consistency

Original: In this section we illustrate that the age-path tracked by GAGA exactly consists with the path of realpartial optimum, which is produced by the ACS algorithm.
Revised: In this section we illustrate that the age-path tracked by GAGA exactly consists with the path of real partial optimum, which is produced by the ACS algorithm.

Original: Due to the expensive computational costof finding the partial optimum using ACS (over 30 loops on average), we choose the toy datasetsfrom the Scikit-learn package to trace and plot the age-path.
Revised: Due to the expensive computational cost of finding the partial optimum using ACS (over 30 loops on average), we choose the toy datasets from the Scikit-learn package to trace and plot the age-path.

Original: In detail, we use the Boston house andbreast cancer datasets for regression tasks, and the classification is performed on the handwrittendigits dataset.
Revised: In detail, we use the Boston house and breast cancer datasets for regression tasks, and the classification is performed on the handwritten digits dataset.

Original: In order to track the exact path of partial optimum, we set the step size to be 1e-4 and3e-1 in ACS and GAGA , respectively.
Revised: In order to track the exact path of partial optimum, we set the step size to be 1e-4 and 3e-1 in ACS and GAGA , respectively.

Original: The graphs of the tracked path are illustrated in Figure 8, Figure9 and Figure 10.
Revised: The graphs of the tracked path are illustrated in Figure 8, Figure 9 and Figure 10.

Original: The path of partial optimum is plotted in blue solid lines while the age-path tracedby GAGA is marked with red dashed lines.
Revised: The path of partial optimum is plotted in blue solid lines while the age-path traced by GAGA is marked with red dashed lines.

Original: This result empirically validates the path consistency between the computed age-path by GAGA andthe ground truth age-path ( i.e. path of the partial optimum), which is stated in Theorem (3).
Revised: This result empirically validates the path consistency between the computed age-path by GAGA and the ground truth age-path ( i.e. path of the partial optimum), which is stated in Theorem (3).

Original: D.5 More Histograms
Revised: D.5 More Histograms

Original: We further show the intrinsic property of the age-path in more experiments.
Revised: We further show the intrinsic property of the age-path in more experiments.

Original: Specifically, we changethe dataset, SP-regularizer as well as the value of other hyper-parameter in SVM, Lasso and logisticregression, while recording the number of different types of critical points.
Revised: Specifically, we change the dataset, SP-regularizer as well as the value of other hyper-parameter in SVM, Lasso and logistic regression, while recording the number of different types of critical points.

Original: The results are illustratedin Figure 11, 12 and 13.
Revised: The results are illustrated in Figure 11, 12 and 13.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: These histograms demonstrate that there exists more turning points on the age-path compared withthe jump points.
Revised: These histograms demonstrate that there exists more turning points on the age-path compared with the jump points.

Original: As a result, the heuristic technique applied in GAGA can avoid extensive unnecessarywarm starts by identifying the exact type of each critical point.
Revised: As a result, the heuristic technique applied in GAGA can avoid extensive unnecessary warm starts by identifying the exact type of each critical point.

Original: D.6 Experimental Comparison of robust SVMs and Lasso
Revised: D.6 Experimental Comparison of robust SVMs and Lasso

Original: We also conduct a comparative study to the state-of-the-art robust model for SVMs [48, 49, 50] and
Revised: We also conduct a comparative study to the state-of-the-art robust model for SVMs [48, 49, 50] and

Original: Lasso [51] besides the SPL domain.
Revised: Lasso [51] besides the SPL domain.

Original: Please take notice that RLSSVM and Re-LSSVM proposed by[48, 49] are variations of LS-SVM.
Revised: Please take notice that RLSSVM and Re-LSSVM proposed by [48, 49] are variations of LS-SVM.

Original: Especially, we implement the Huber-loss Lasso as a special caseof the generalized model proposed in [51].
Revised: Especially, we implement the Huber-loss Lasso as a special case of the generalized model proposed in [51].

Original: The hyperparameters of these baselines are chosen fromthe best performance by grid search.
Revised: The hyperparameters of these baselines are chosen from the best performance by grid search.

Original: All the experiments conducted on regression tasks are measuredby the generalization error.
Revised: All the experiments conducted on regression tasks are measured by the generalization error.

Original: We’d like to emphasize again that our GAGA framework pursues the bestpractice of conventional SPL while the SPL, as a special case of curriculum learning naturally ownscertain shortcomings on the sample diversity [13].
Revised: We’d like to emphasize again that our GAGA framework pursues the best practice of conventional SPL while the SPL, as a special case of curriculum learning naturally owns certain shortcomings on the sample diversity [13].

Original: Even subject to the inherent defects of vanilla SPL,the result in Table.7 reveals that our method still surpassed SOTA baselines in half of the experiments.
Revised: Even subject to the inherent defects of vanilla SPL, the result in Table.7 reveals that our method still surpassed SOTA baselines in half of the experiments.

Original: The Table.8 further implies that GAGA outperforms the robust SVM in all conducted trails.
Revised: The Table.8 further implies that GAGA outperforms the robust SVM in all conducted trails.

Original: Thesenumerical results strongly validate the performance of our method under various noise levels.
Revised: These numerical results strongly validate the performance of our method under various noise levels.

Original: Concretely speaking, the linear SP-regularizer is utilized in GAGA in the implementation.
Revised: Concretely speaking, the linear SP-regularizer is utilized in GAGA in the implementation.

Original: The γ 1 andγ 2 are hyperparameters of RLSSVM and RE-LSSVM .
Revised: The γ 1 and γ 2 are hyperparameters of RLSSVM and RE-LSSVM .

Original: We use the same hyperparameters of GAGA in
Revised: We use the same hyperparameters of GAGA in

Original: Appendix D.7.
Revised: Appendix D.7.

Original: D.7 Sensitivity Analysis
Revised: D.7 Sensitivity Analysis

Original: In this subsection, we use the same settings except the backbone parameters α , C and the noise level.
Revised: In this subsection, we use the same settings except the backbone parameters α , C and the noise level.

Original: The linear SP-regularizer is utilized in GAGA.
Revised: The linear SP-regularizer is utilized in GAGA.

Original: The classical SVM with linear kernel is chosen asthe base model.
Revised: The classical SVM with linear kernel is chosen as the base model.

Original: Results in Table 9, 10, 11, 12 confirm the performances of competing methods withdifferent backbone parameters while retaining the same noise level.
Revised: Results in Table 9, 10, 11, 12 confirm the performances of competing methods with different backbone parameters while retaining the same noise level.

Original: Table 13, 14, 15, 16 display therunning results under different noise levels while the other backbone parameters are kept.
Revised: Table 13, 14, 15, 16 display the running results under different noise levels while the other backbone parameters are kept.

Original: The resultsof the massive simulation studies again strongly demonstrate that our GAGA owns the best practiceof the conventional SPL compared with the baseline methods, regardless of the specific parametricselections.
Revised: The results of the massive simulation studies again strongly demonstrate that our GAGA owns the best practice of the conventional SPL compared with the baseline methods, regardless of the specific parametric selections.

Original: s o l u t i o n    5 0.
Revised: s o l u t i o n    5 0.

Original: s o l u t
Revised: s o l u t

Original: i o n    5 0.
Revised: i o n    5 0.

Original: s o l u t
Revised: s o l u t

Original: i o n
Revised: i o n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: l u t i o n
Revised: l u t i o n

Original: [Table]
Revised: [Table]

Original: l u t i o n
Revised: l u t i o n

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: l u t
Revised: l u t

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: s o l u t
Revised: s o l u t

Original: i o n
Revised: i o n

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: s
Revised: s

Original: [Table]
Revised: [Table]

Original: l u
Revised: l u

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: n    5 0.
Revised: n    5 0.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: u t i o n
Revised: u t i o n

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: s o l u t
Revised: s o l u t

Original: i o n
Revised: i o n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: Dataset
Revised: Dataset

Original: Parameter
Revised: Parameter

Original: Robust-SVM
Revised: Robust-SVM

Original: GAGA
Revised: GAGA

Original: Noise Level C γ
Revised: Noise Level C γ

Original: Mean
Revised: Mean

Original: Std
Revised: Std

Original: Mean
Revised: Mean

Original: Std mfeat-pixel 0.
Revised: Std mfeat-pixel 0.

Original: mfeat-pixel 0. mfeat-pixel 0.
Revised: mfeat-pixel 0. mfeat-pixel 0.

Original: pendigts 0. pendigts 0.
Revised: pendigts 0. pendigts 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: music 0. 
Revised: music 0. music 0. music 0. music 0.

Original: music 0. 
Revised: music 0. music 0. music 0. music 0.

Original: music 0. 
Revised: music 0. music 0. music 0. music 0.

Original: music 0. music 0.music 0.
Revised: music 0. music 0.

Original: music 0. music 0.music 0.
Revised: music 0. music 0.

Original: music 0. music 0.music 0.
Revised: music 0. music 0.

Original: music 0.music 0. music 0. music 0.
Revised: music 0.

Original: music 0.music 0. music 0. music 0.
Revised: music 0.

Original: music 0.
Revised: music 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: pendigts 0. pendigts 0.
Revised: pendigts 0. pendigts 0.

Original: pendigts 0. pendigts 0.
Revised: pendigts 0. pendigts 0.

Original: pendigts 0. pendigts 0.
Revised: pendigts 0. pendigts 0.

Original: pendigts 1. pendigts 1.
Revised: pendigts 1. pendigts 1.

Original: pendigts 1. pendigts 1.
Revised: pendigts 1. pendigts 1.

Original: pendigts 1. pendigts 1.
Revised: pendigts 1. pendigts 1.

Original: [Table]
Revised: [Table]

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.

Original: s o l u t
Revised: s o l u t

Original: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.
Revised: i o n 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.

Original: s o l u t
Revised: s o l u t

Original: i o n
Revised: i o n

Original: Dataset
Revised: Dataset

Original: Parameter
Revised: Parameter

Original: Competing Methods
Revised: Competing Methods

Original: Noise Level α
Revised: Noise Level α

Original: ACS
Revised: ACS

Original: MOSPL
Revised: MOSPL

Original: GAGA ailerons 0.
Revised: GAGA ailerons 0.

Original: ailerons 0. ailerons 0.
Revised: ailerons 0. ailerons 0.

Original: ailerons 0. ailerons 0.
Revised: ailerons 0. ailerons 0.

Original: ailerons
Revised: ailerons

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E Limitations and Broader Impact
Revised: E Limitations and Broader Impact

Original: Limitations.
Revised: Limitations.

Original: The current framework GAGA is only applicable to the vanilla SPL paradigm, whilesome variants of SPL ( e.g. , self-paced learning with diversity [2]) could have a group of hyperparameters more than the mere λ .
Revised: The current framework GAGA is only applicable to the vanilla SPL paradigm, while some variants of SPL ( e.g. , self-paced learning with diversity [2]) could have a group of hyperparameters more than the mere λ .

Original: In this instance, the original solution paths escalates into the solutionsurfaces, and made it harder to track the solutions.
Revised: In this instance, the original solution paths escalates into the solution surfaces, and made it harder to track the solutions.

Original: Another point is that our argument assumed thatthe target function is a biconvex problem, while many complex losses ( e.g. , deep neural networks)could be the function of strongly non-convex.
Revised: Another point is that our argument assumed that the target function is a biconvex problem, while many complex losses ( e.g. , deep neural networks) could be the function of strongly non-convex.

Original: Broader Impact.
Revised: Broader Impact.

Original: The parties with limited computational resource may benefit from the efficiencyof our proposed work.
Revised: The parties with limited computational resource may benefit from the efficiency of our proposed work.

Original: Meanwhile, the research groups that sensitive to age parameters in SPL maybenefit from the robustness of our work.
Revised: Meanwhile, the research groups that sensitive to age parameters in SPL may benefit from the robustness of our work.

Original: For social impact, our work improve the hyperparametersearch and helps reducing the computation cost, which saves carbon emissions during the trainingprocess.
Revised: For social impact, our work improve the hyperparameter search and helps reducing the computation cost, which saves carbon emissions during the training process.

Original: F Code Readme
Revised: F Code Readme

Original: This section explains how to use the code implementing the proposed GAGA .
Revised: This section explains how to use the code implementing the proposed GAGA .

Revised: The codebase is available via github.com/diyang-lee/GAGA.

Original: F.1 Archive content
Revised: F.1 Archive content

Original: The fundamental implementation of our methodology GAGA is called GAGA py which includes 
Revised: The fundamental implementation of our methodology GAGA is called GAGA.py , which includes all the functions used in our experiment.

Original: all

Original: the functions used

Original: in our experiment In more detail, they are named in the form of [GAGA]_[Model_Name]_[SPL_Regularizer], e.g. , GAGA_svm_linear.
Revised: In more detail, they are named in the form of [GAGA]_[Model_Name]_[SPL_Regularizer], e.g. , GAGA_svm_linear.

Original: The Evaluation.py provides functions for evaluating solution path from different models including ACS.
Revised: The Evaluation.py provides functions for evaluating solution path from different models including ACS.

Original: The Input_Data.py offers file-IO for all datasets used in our experiment.
Revised: The Input_Data.py offers file-IO for all datasets used in our experiment.

Original: The ACS.py provides ACS algorithm for solving SPL.
Revised: The ACS.py provides ACS algorithm for solving SPL.

Original: F.2 Reproducing the results of the article
Revised: F.2 Reproducing the results of the article

Original: For the sake of quick and convenient reproducing experiments and checking our findings, we provide ademo named main.py .
Revised: For the sake of quick and convenient reproducing experiments and checking our findings, we provide a demo named main.py .

Original: We use open-source datasets and provide file-IO functions along with detailedpre-processing pipeline for users.
Revised: We use open-source datasets and provide file-IO functions along with detailed pre-processing pipeline for users.

Original: Any used datasets in our experiment can be easily downloaded themfrom the UCI and OpenML website and put them in the 
Revised: Any used datasets in our experiment can be easily downloaded them from the UCI and OpenML website and put them in the datasets folder.

Revised: The required dependencies and running environment is recorded in Environment.txt .