Original: An Accuracy Guaranteed Online Solver for Learning in Dynamic Feature Space
Revised: An Accuracy Guaranteed Online Solver for Learning in Dynamic Feature Space

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: We study the problem of adding or deleting features of data from machine learning models trained using empirical risk minimization.
Revised: We study the problem of adding or deleting features of data from machine learning models trained using empirical risk minimization.

Original: Our focus is on algorithms in an online manner which is capable for a more general regularization term, and present practical guides to two classical regularizers, i e. , the group Lasso and (cid:96) p -norm regularizer.
Revised: Our focus is on algorithms in an online manner which is capable for a more general regularization term, and present practical guides to two classical regularizers, i e. , the group Lasso and (cid:96) p -norm regularizer.

Original: Across a variety of benchmark datasets, our algorithm improves upon the runtime of prior methods while maintaining the same generalization accuracy
Revised: Across a variety of benchmark datasets, our algorithm improves upon the runtime of prior methods while maintaining the same generalization accuracy

Original: 1. Introduction
Revised: 1. Introduction

Original: In traditional machine learning, we normally assume that the set of features is known and ﬁxed.
Revised: In traditional machine learning, we normally assume that the set of features is known and ﬁxed.

Original: However, it’s unrealistic to many real-world scenarios [5, 6, 17].
Revised: However, it’s unrealistic to many real-world scenarios [5, 6, 17].

Original: To address this, incremental (or decremental) algorithm [8] was proposed to incorporate or eliminate features, which is a promising family of learning algorithms due to the updated model can reﬁne its knowledge without re-training from scratch [16].
Revised: To address this, incremental (or decremental) algorithm [8] was proposed to incorporate or eliminate features, which is a promising family of learning algorithms due to the updated model can reﬁne its knowledge without re-training from scratch [16].

Original: In supervised learning, an extensive number of algorithms essentially solve 
Revised: In supervised learning, an extensive number of algorithms essentially solve the regularized minimization problem, and for a linear model, it reads

Original: [Equation]
Revised: [Equation]

Original: where R ( · ) is the regularization term and L ( · ) denotes the loss function.
Revised: where R ( · ) is the regularization term and L ( · ) denotes the loss function.

Revised: The w represents learnable parameters inside the model.

Original: Although there have been many efforts made to incorporate or eliminate features in an exact way, they are still limited to primitive regularization terms such as (cid:96) 1 and (cid:96) 2 norms in (1).
Revised: Although there have been many efforts made to incorporate or eliminate features in an exact way, they are still limited to primitive regularization terms such as (cid:96) 1 and (cid:96) 2 norms in (1).

Original: The Lasso regularizer has been solved in [7] using forward stagewise regression.
Revised: Speciﬁcally, the Lasso regularizer has been solved in [7] using forward stagewise regression.

Original: Other simple regularization like (cid:96) 2 -norm leads to analytical solutions [9], hence result can be computed directly in the new feature space.
Revised: Other simple regularization like (cid:96) 2 -norm leads to analytical solutions [9], hence result can be computed directly in the new feature space.

Original: To ﬁll this blank, we propose a learning system that handles a class of regularized linear statistical models with dynamic features, termed Dynamic Feature Learning System ( DFLS ), where the updated model is exactly the same as a model trained from scratch using the entire new feature space.
Revised: To ﬁll this blank, we propose a learning system that handles a class of regularized linear statistical models with dynamic features, termed Dynamic Feature Learning System ( DFLS ), where the updated model is exactly the same as a model trained from scratch using the entire new feature space.

Original: Extensive experimental results validate its efﬁciency compared to the existing training algorithm.
Revised: Extensive experimental results validate its efﬁciency compared to the existing training algorithm.

Original: 2. PreliminariesObjective Rewriting
Revised: 2. Preliminaries

Original: Given the dataset D = { X ∈ R n × m , y ∈ R n } with n observations and each observation has m features.
Revised: Given the dataset D = { X ∈ R n × m , y ∈ R n } with n observations and each observation has m features.

Original: The y i ∈ R for the regression problem {+
+} and y i ∈ {− 1 , 1 } for the binary classiﬁcation task.
Revised: The y i ∈ R for the regression problem {+
+} and y i ∈ {− 1 , 1 } for the binary classiﬁcation.

Original: To simplify notations we assume that the new (or obsolete) features appears last, we organize (or partition)
Revised: Assumption 1 We assume that the loss function L ( · ) and regularization item R ( · ) in (1) are piecewise (or element-wise) second-order differentiable. Assumption 2 We assume that the optimal solution of (1) can be achieved. Problem Setting First, we consider the case of adding new features in (1).

Revised: Assume we currently own n labeled instances D cur = { X, y } and an optimal ML model trained using D cur , in which the parameter is denoted by

Revised: w (1) ∈

Revised: R m (already known).

Revised: With the time passed, newly acquired data can be formalized as (cid:101) X ∈ R n × m (cid:48) that includes m (cid:48) features and we want to add (cid:101) X into the existing training set 2 .

Revised: In other words, we need an optimal ML model that trained on D

Original: the traning data X as (cid:104) X, (cid:101) X (cid:105)and coefﬁcient vector w =
Revised: new = (cid:110)(cid:104) X, (cid:101) X (cid:105)

Original: (cid:20) ww(cid:21) by two types of features, where (cid:101) X ∈ Rn × m (cid:48) includes m (cid:48) features we intend to add into (or delete from) the original feature set.
Revised: , y (cid:111) and the parameter of optimal model on the new dataset is w (2) ∈ R m + m (cid:48) (currently unknown). A conventional practice is to re-train a new model on D new , while our DFLS aims to compute the w (2) directly ( i.e. , without re-training from scratch) using the information of w (1) .

Original: © .
Revised: Conversely, when we think about deleting m (cid:48) old features (cid:101) X ∈ R n × m (cid:48) from existing dataset D cur = (cid:110)(cid:104) X, (cid:101) X (cid:105) , y (cid:111) , our goal is to restore an optimal ML model trained on D new = { X, y } in like manner. Objective Rewriting

Original: Assumption 1 We assume that the loss function L ( · ) and regularization item R ( · ) in (1) are piecewise (or element-wise) second-order differentiable. Assumption 2 We assume that the optimal solution of (1) can be achieved.
Revised: To simplify notations we organize (or partition) the traning data as (cid:104) X, (cid:101) X (cid:105) and coefﬁcient vector w = (cid:20) ww(cid:21) by two types of features, where (cid:101) X ∈ R n × m (cid:48) includes m (cid:48) features we intend to add into (or delete from) the original feature set.

Original: The A and E are active sets that containing indices of w 1 and w 2 that w ij (cid:54) = 

Revised: The A and E are active sets that containing indices of w 1 and w 2 that w ij (cid:54) =

Original: ( i = 1 , 2) , so as to store active components, i.e. , we have w 1 = (cid:16) w T A , w T A (cid:17) T , w 2 = (cid:16) w T E , w T E (cid:17) T , where elements in w A , w E remains 0.
Revised: ( i = 1 , 2) , so as to store active components, i.e. , we have w 1 = (cid:16) w T A , w T A (cid:17) T , w 2 = (cid:16) w T E , w T E (cid:17) T , where elements in w A , w E remains 0.

Revised: We denote a point where the active set changes a “ transition point ”.

Original: Now we introduce the following concrete formulation
Revised: Now we introduce the following concrete formulation

Original: [Equation]
Revised: [Equation]

Original: where α > 0 is regularization parameter to balance the prediction loss and penalty.
Revised: where α > 0 is regularization parameter to balance the prediction loss and penalty.

Original: A rational φ θ in (2) can be used to reparameterize the addition or subtraction procedure of a chunk feature (cid:101) X , which satisﬁes (i).
Revised: A rational φ θ in (2) can be used to reparameterize the addition or subtraction procedure of a chunk feature (cid:101) X , which satisﬁes (i).

Original: φ θ are monotonically increasing function w.r.t.
Revised: φ θ are monotonically increasing function w.r.t.

Original: θ in closed interval [0 , 1] .
Revised: θ in closed interval [0 , 1] .

Original: (ii).
Revised: (ii).

Original: φ θ have the properties of continuity and smoothness, meanwhile φ θ | θ =

Revised: φ θ have the properties of continuity and smoothness, meanwhile φ θ | θ =

Original: = 0 , φ θ | θ =1 = 1 should be kept.
Revised: = 0 , φ θ | θ =1 = 1 should be kept.

Original: Here we can follow the solution path of w when varying θ from 0 to 1 to add (cid:101) X .
Revised: Here we can follow the solution path of w when varying θ from 0 to 1 to add (cid:101) X .

Original: Conversely, as we vary θ from 1 to 0 , the elimination of features in (cid:101) X can then be performed in a similar way.
Revised: Conversely, as we vary θ from 1 to 0 , the elimination of features in (cid:101) X can then be performed in a similar way.

Original: Investigate Optimality Conditions
Revised: Investigate Optimality Conditions

Original: The KKT point [3] of the problem (2) can be obtained by
Revised: The KKT point [3] of the problem (2) can be obtained by

Original: [Equation]
Revised: [Equation]

Original: where u ∈ ∂ R ( w ) is subgradient representation.
Revised: where u ∈ ∂ R ( w ) is subgradient representation.

Original: 3. Accuracy Guaranteed Online Solver 3.1. Algorithm Outline
Revised: 3. Accuracy Guaranteed Online Solver 3.1. Algorithm Outline

Original: [Table]
Revised: [Table]

Original: We ﬁrst pointed out that there potentially exists two stages for tracking the optimal w .
Revised: We ﬁrst pointed out that there potentially exists two stages for tracking the optimal w .

Original: The proposed updating stages are summarized as (i).
Revised: The proposed updating stages are summarized as (i).

Original: The solution path of θ would remain constant untill the ﬁrst transition point (cid:101) θ is met.
Revised: The solution path of θ would remain constant untill the ﬁrst transition point (cid:101) θ is met.

Original: We compute its value in closed form and update w directly.
Revised: We compute its value in closed form and update w directly.

Original: (ii).
Revised: (ii).

Original: It’s non-trivial for analytical solutions of w ( θ ) in (3), especially some complex penalizations.
Revised: It’s non-trivial for analytical solutions of w ( θ ) in (3), especially some complex penalizations.

Original: We use the differential equation to gain the desired solution path 2 untill θ = 1 .
Revised: We use the differential equation to gain the desired solution path 4 untill θ = 1 .

Original: The usage of each stage depends on the properties of regularization term.
Revised: The usage of each stage depends on the properties of regularization term.

Original: The criterion of whether to use each phase are summarized in Table 1.
Revised: The criterion of whether to use each phase are summarized in Table 1.

Original: The decremental learning can be viewed as an inverse process for tracking the path from θ = 1 to θ = 0 .
Revised: The decremental learning can be viewed as an inverse process for tracking the path from θ = 1 to θ = 0 .

Original: 3.2. Phase I Update
Revised: 3.2. Phase I Update

Original: [Equation]
Revised: [Equation]

Original:  Due to the existence of R ( w ) , at ﬁrst ( i.e. θ = 0 ) all coefﬁcients in w 2 will converges to 0 .
Revised: [Equation] [Equation] Due to the existence of R ( w ) , at ﬁrst ( i.e. θ = 0 ) all coefﬁcients in w 2 will converges to 0 .

Original: As w 2 | θ =0 = 0 , by plugging it into (3) we have (4) and (5).
Revised: As w 2 | θ =0 = 0 , by plugging it into (3) we have (4) and (5).

Original: Considering inactive components in (5), by the properties of subgradient we get (6).
Revised: Considering inactive components in (5), by the properties of subgradient we get (6).

Original: Since φ θ is continuous w.r.t.
Revised: Since φ θ is continuous w.r.t.

Original: θ , we can conclude that there exists a (cid:101) θ > 0 such that the inequality (6) holds in [0 , (cid:101) θ ) .
Revised: θ , we can conclude that there exists a (cid:101) θ > 0 such that the inequality (6) holds in [0 , (cid:101) θ ) .

Original: We denote such a point where the active set changes a “ transition point ”.

Original: The solution path would be smooth before approaches a transition point, and non-smooth graphically at transition points.
Revised: The solution path would be smooth before approaches a transition point, and non-smooth graphically at transition points.

Original: From (4) we can see the optimal w 1 has nothing to do with θ when θ ∈ [0 , (cid:101) θ ) , it will be a constant vector till the θ reaches (cid:101) θ .
Revised: From (4) we can see the optimal w 1 has nothing to do with θ when θ ∈ [0 , (cid:101) θ ) , it will be a constant vector till the θ reaches (cid:101) θ .

Original: Assume there are one component becomes active, we calculate the ﬁrst encountered transition point (cid:101) θ in exact solution
Revised: Assume there are one component becomes active, we calculate the ﬁrst encountered transition point (cid:101) θ in exact solution

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then we update
Revised: Then we update

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) θ
Revised: where (cid:98) θ

Original: = min (cid:110)(cid:101) θ, 1 (cid:111) .
Revised: = min (cid:110)(cid:101) θ, 1 (cid:111) .

Original: If (cid:98) θ = 1 , then we can skip the second stage and ﬁnish the algorithm.
Revised: If (cid:98) θ = 1 , then we can skip the second stage and ﬁnish the algorithm.

Original: From expression in (7), we know that the location of the ﬁrst transition point is proportional to parameter α , which makes our approach very efﬁcient when α in relatively large quantities.
Revised: From expression in (7), we know that the location of the ﬁrst transition point is proportional to parameter α , which makes our approach very efﬁcient when α in relatively large quantities.

Original: Remark 1 When we use decremental learning, there are no analytical solution for (cid:98) θ .
Revised: Remark 1 When we use decremental learning, there are no analytical solution for (cid:98) θ .

Original: The boundary condition for Phase I (Phase II) is that whether E = ∅ (or w 2 = 0 ).
Revised: The boundary condition for Phase I (Phase II) is that whether E = ∅ (or w 2 = 0 ).

Original: 3.3. Phase II Update
Revised: 3.3. Phase II Update

Original: Theorem 2 When the active sets A and E are ﬁxed, the solution path of w satisﬁes the following  ODE system:
Revised: Theorem 2 When the active sets A and E are ﬁxed (in a subinterval of [0 , 1] ), the solution path of w satisﬁes the following ﬁrst-order ODE system

Original: [Equation]
Revised: [Equation]

Original: where ( · ) (cid:48) is the derivative note.
Revised: where ( · ) (cid:48) is the derivative note.

Original: The key matrix (cid:98) K has the form of (cid:20) U
Revised: The key matrix (cid:98) K has the form of (cid:20) U

Original: Z T
Revised: [Equation]

Original: Z
Revised: Z

Original: V (cid:21) , where U =
Revised: V [Equation] , where U =

Original: [Equation]
Revised: [Equation]

Original: Theorem 3 We have (cid:98) K is a real symmetric matrix.
Revised: Theorem 3 We have (cid:98) K is a real symmetric matrix.

Original: The proof of Theorems are listed in Appendix A. With Theorem 3, we can use the low-rank update or Woodbury formulae [14], which essentially solves (8) with at most O ( |A ∪ E| 2 ) work.
Revised: The proof of Theorems are listed in Appendix A. With Theorem 3, we can use the low-rank update or Woodbury formulae [14], which essentially solves (8) with at most O ( |A ∪ E| 2 ) work.

Original: As we obtaining the w | θ =0 and d w A∪E dθ , by solving the initial-value problem numerically with ODE solvers, the solution path regarding to θ can be computed swiftly before active sets A or E changes. 
Revised: As we obtaining the w | θ =0 and d w A∪E dθ , by solving the initial-value problem numerically with ODE solvers, the solution path regarding to θ can be computed swiftly before active sets A or E changes. Some visualization examples of solution path are listed in Appendix C.

Original:  Transition Point
Revised: Handling Transition Point

Original: The emergence of a transition point is owing to indices change in A or E , which is characterized by the variation of optimality conditions (3).
Revised: The emergence of a transition point is owing to indices change in A or E , which is characterized by the variation of optimality conditions (3).

Original: Meanwhile the ODE wouldn’t work due to they are non-differentiable points.
Revised: Meanwhile the ODE wouldn’t work due to they are non-differentiable points.

Original: We can keep observation on the optimality conditions while solving the path to estimate if there exists transition point(s), and update the active sets A and 
Revised: We can keep observation on the optimality conditions while solving the path to estimate if there exists transition point(s), and update the active sets A and E accordingly .

Original: E accordingly.
Revised: The violator(s) could be easily identiﬁed by the thresholding conditions.

Original: For non-sparsity penalizations, there will be no changes for A or E , hence d w dθ is continuous of θ on (0 , 1) .
Revised: For non-sparsity penalizations, there will be no changes for A or E , hence d w dθ is continuous of θ on (0 , 1) .

Original: Some visualization examples of solution path are listed in Appendix C.

Original: 4. DFLS Implementation
Revised: 4. DFLS Implementation

Original: In this part, we select the transformation as φ θ def = θ .
Revised: In this part, we select the transformation as φ θ def = θ .

Original: Due to the space limit, we defer the algorithmic steps and feature decremental learning procedure in our Appendix B, Appendix E, respectively.[Equation]
Revised: Due to the space limit, we defer the algorithmic steps and feature decremental learning procedure in our Appendix B, Appendix E, respectively.

Original: Group Lasso
Revised: Group Lasso

Original: Group Lasso [18] has been successful in many practical applications [4, 10].
Revised: Group Lasso [18] has been successful in many practical applications [4, 10].

Original: There has a total of G groups, d g is the number of features in g -th group.
Revised: There has a total of G groups, d g is the number of features in g -th group.

Original: We focus on group regularized optimization problem as (9). 
Revised: We focus on group regularized optimization problem as (9). [Equation]

Original: (cid:96) p Penalization
Revised: (cid:96) p Penalization

Original: Optimization method towards (cid:96) p -norm penalty has been studied extensively by researchers [11].
Revised: Optimization method towards (cid:96) p -norm penalty has been studied extensively by researchers [11].

Original: Here we present the case of p > 1 , which gives a convex setting of R ( · ) .
Revised: Here we present the case of p > 1 , which gives a convex setting of R ( · ) .

Original: Our learning objective is shown as (10). 
Revised: Our learning objective is shown as (10). [Equation]

Revised: [Equation]

Original: 5. Simulation Experiments 5.1. Experimental Setup
Revised: 5. Simulation Experiments 5.1. Experimental Setup

Original: [Figure]
Revised: [Table]

Original: (MR#1 ∼ 3) in experiments.
Revised: In this part, we provide experiments on real-world datasets to validate our theoretical results.

Original: Our experiments are delivered from the following two perspectives.
Revised: Our experiments are delivered from the following two perspectives.

Original: Table 2 summarizes these datasets we used.
Revised: Table 2 summarizes these datasets we used.

Original: Accuracy. To prove the practicability of proposed approach, we compare the generalization performance of DFLS with batch algorithms.
Revised: Accuracy To prove the practicability of proposed approach, we compare the generalization performance of DFLS with batch algorithms.

Original: We conduct training tasks on real-world data and randomly select about 20% features in whole feature space to perform dynamic updating.
Revised: We conduct training tasks on real-world data and randomly select about 20% features in whole feature space to perform dynamic updating.

Original: In detail, we utilize the welltrained model to add or remove around 20% features and compare the average Root Mean Squared Error (aRMSE) between several methods in ten runs.
Revised: In detail, we utilize the well-trained model to add or remove around 20% features and compare the average Root Mean Squared Error (aRMSE) between several methods in ten runs.

Original: The batch algorithms re-train the model with both cold-start and warm-start (i.e., the subsequent calls will not re-initialise parameters).
Revised: The batch algorithms re-train the model with both cold-start and warm-start (i.e., the subsequent calls will not re-initialise parameters).

Original: Numerical results are shown in Table 3 and Table 4, while we put the results of decremental learning in Appendix E.
Revised: Numerical results are shown in Tableand Table 4, while we put the results of decremental learning in Appendix E.

Original: Efﬁciency. We evaluate average processing times in ﬁve runs when executing one dynamic updating (feature numbers m (cid:48)
Revised: Efﬁciency We evaluate average processing times in ﬁve runs when executing one dynamic updating (feature numbers m (cid:48)

Original: = 5 ) using our DFLS and other methods under diverse data scales.
Revised: = 5 ) using our DFLS and other methods under diverse data scales.

Original: Additionally, we test the training time on m (cid:29) n cases based on MR#1 ∼ 2.
Revised: Additionally, we test the training time on m (cid:29) n cases based on MR#1 ∼ 2.

Original: In order to verify the inﬂuence of high-dimensional situation on efﬁciency, in our generated MR#1 ∼ 3, we keep the data size while increasing number of variable dimensions.
Revised: In order to verify the inﬂuence of high-dimensional situation on efﬁciency, in our generated MR#1 ∼ 3, we keep the data size while increasing number of variable dimensions.

Original: Baselines. We adopt a widely-used batch training algorithm of group Lasso using FISTA optimiser [1] with a gradient-based adaptive restarting scheme 3 [13].
Revised: Baselines Setup We adopt a widely-used batch training algorithm of group Lasso using FISTA optimiser [1] with a gradient-based adaptive restarting scheme 5 [13].

Original: The batch training algorithm for (cid:96) p pe-

Original: [Figure]

Original: [ MR#2 ] of different algorithms.

Original: The α = 0 . 2 for (cid:96) p penalty. [ · ] indicates the name of dataset. nalization uses Stochastic Gradient Descent (SGD) [2] to optimize the model.
Revised: The batch training algorithm for (cid:96) p penalization uses Stochastic Gradient Descent (SGD) [2] to optimize the model.

Original: Other implementation details are in Appendix D .
Revised: Other implementation details are in Appendix D .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5.2. Results & Discussions
Revised: [Figure] Lasso [ Yolanda ] [Figure] [Table]

Original: From above Table 3 and Table
Revised: [Figure] [Table] [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: our algorithm enjoys high precision
Revised: p penalty [ BNG(libras move) ] [Figure] [Figure] image ]

Original: [Table]
Revised: [Figure]

Original:  which is characterized by very closing to the aRMSE of batch training method.
Revised: [ MR#2 ] 5.2. Results & Discussions From above Table 3 and Table 4, our algorithm enjoys high precision, which is characterized by very closing to the aRMSE of batch training method.

Original: Moreover, Figure 1 clearly demonstrate that there exists a large time gap compared to other learning strategies while our DFLS keeping the very similar precision.
Revised: Moreover, Figure 1 clearly demonstrate that there exists a large time gap compared to other learning strategies while our DFLS keeping the very similar precision.

Original: This is due to the fact that our algorithm can track the path of θ without training the whole model in new feature space from scratch.
Revised: This is due to the fact that our algorithm can track the path of θ without training the whole model in new feature space from scratch.

Original: Furthermore, the training time in Figure(d).
Revised: Furthermore, the training time in Figure(d).

Original: and (e).
Revised: and (e).

Original: suggest our algorithm is inefﬁcient compared to batch training in exceptional highdimensional cases due to the increased time on solving the linear system in (8).
Revised: suggest our algorithm is inefﬁcient compared to batch training in exceptional highdimensional cases due to the increased time on solving the linear system in (8).

Original: Particularly, in m (cid:29) n , the advantages of DFLS would gradually erased as m grows. 
Revised: Particularly, in m (cid:29) n , the advantages of DFLS would gradually erased as m grows. 6. Conclusion

Revised: At this work we propose an efﬁcient online solver via ODEs to perform features expansion and shrinkage for linear models and present practical guides to solving group Lasso and (cid:96) p -norm regularized regression.

Revised: Simulation studies show that the proposed algorithm can substantially ease the computational cost while keeping the accuracy performance.

Original: Appendix A. Detailed Proofs A.1. Proof of Theorem
Revised: Appendix A. Detailed Proofs A.1. Proof of Theorem

Original: Proof By discarding the zero-component 4 , the optimality conditions for objective (2) are formulated as follows.
Revised: Proof By discarding the zero-component 6 , the optimality conditions for objective (2) are formulated as follows.

Original: [Equation]
Revised: [Equation]

Original: We derive a system of ODEs to compute the solution path w.r
Revised: We derive a system of ODEs to compute the solution path w.r.t.

Original: [Equation]

Original: t. θ when the active sets A and E are ﬁxed.
Revised: θ when the active sets A and E are ﬁxed.

Original: With the aid of derivation rules for compound function [Equation] we take the derivative of θ in (11) and (12).
Revised: With the aid of derivation rules for compound function, we take the derivative of θ in (11) and (12).

Original: We can build
Revised: We can build

Original: [Equation]
Revised: [Equation]

Original: The above two linear equations (13) can be converted to the following form:  [Equation] F (
Revised: The above two linear equations (13) can be converted to the following form:

Original: [Equation]
Revised: [Equation]

Original: (cid:35) , (14) where ( · ) (cid:48) is the derivative note.
Revised: (14) where ( · ) (cid:48) is the derivative note.

Original: By computing the remaining partial derivative terms in (14), we know that (cid:98) K has the form of (cid:20) U
Revised: By computing the remaining partial derivative terms in (14), we know that (cid:98) K has the form of (cid:20) U

Original: Z T
Revised: Z T

Original: Z
Revised: Z

Original: V (cid:21) , where U = 1 nX T A X A + α ∇ 2 R ( w A ) , V = φ 2 θ n · (cid:101) X T E (cid:101)
Revised: V (cid:21) , where U = 1 nX T A X A + α ∇ 2 R ( w A ) , V = φ 2 θ n · (cid:101) X T E (cid:101)

Original: X E + α ∇ 2 R ( w E ) and
Revised: X E + α ∇ 2 R ( w E ) and

Original: Z = φ θ n · (cid:101)
Revised: Z = φ θ n · (cid:101)

Original: X T E X A .
Revised: X T E X A .

Original: A.2. Proof of Theorem
Revised: A.2. Proof of Theorem

Original: Proof Based on the symmetry property of any Hessian matrix ∇ 2 R ( · ) = ∇ ( ∇R ( · )) , we can conclude U T = 1 n ( X T
Revised: Proof Based on the symmetry property of any Hessian matrix ∇ 2 R ( · ) = ∇ ( ∇R ( · )) , we can conclude U T = 1 n ( X T

Original: A X A ) T + α ( ∇ 2 R ) T
Revised: A X A ) T + α ( ∇ 2 R ) T

Original: =
Revised: =

Original: U .
Revised: U .

Original: Meanwhile, the V T =
Revised: Meanwhile, the V T =

Original: V can be proved similarly.
Revised: V can be proved similarly.

Original: From the foregoing, we have (cid:98) K T =
Revised: From the foregoing, we have (cid:98) K T =

Original: (cid:20) U T
Revised: (cid:20) U T

Original: Z T
Revised: Z T

Original: Z
Revised: Z

Original: V T (cid:21) =
Revised: V T (cid:21) =

Original: (cid:20) U
Revised: (cid:20) U

Original: Z T
Revised: Z T

Original: Z
Revised: Z

Original: V (cid:21) =
Revised: V (cid:21) =

Original: (cid:98) K , which completes the proof.
Revised: (cid:98) K , which completes the proof.

Original: Appendix B. Practical Guides
Revised: Appendix B. Practical Guides

Original: This part we present the online incremental solver in detail for two signiﬁcant learning models mentioned in the main body.
Revised: This part we present the online incremental solver in detail for two signiﬁcant learning models mentioned in the main body.

Original: B.1. Group Lasso
Revised: B.1. Group Lasso

Original: At ﬁrst, we compute the (sub)gradient of regularization term as
Revised: At ﬁrst, we compute the (sub)gradient of regularization term as

Original: [Equation]
Revised: [Equation]

Original: There follows ∇ 2 R ( w A∪E ) =
Revised: There follows ∇ 2 R ( w A∪E ) =

Original: diag ( B i ) , where B g = 
Revised: diag ( B i ) , where B g = [Equation] repre

Original: [Equation]
Revised: [Equation]

Original: sents identity matrix.
Revised: sents identity matrix.

Original: Transition point for group Lasso
Revised: Transition point for group Lasso

Original: First of all, we deﬁne the C g as X Tg (cid:16)
Revised: First of all, we deﬁne the C g as X Tg (cid:16)

Original: y − X A 
Revised: y − X A w A − θ

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: X

Original: [Equation]
Revised: [Equation]

Original: where g = 1 , · · · , G denotes g -th predeﬁned group.
Revised: where g = 1 , · · · , G denotes g -th predeﬁned group.

Original: The KKT conditions for group Lasso problem are formulated as follows.
Revised: The KKT conditions for group Lasso problem are formulated as follows.

Original: [Equation]
Revised: [Equation]

Original: A non-zero w g will decay to 0 as (cid:107)C g (cid:107) (cid:54) α (cid:112)
Revised: A non-zero w g will decay to 0 as (cid:107)C g (cid:107) (cid:54) α (cid:112)

Original: d g , g ∈ A (or θ (cid:107)C g (cid:107) (cid:54)
Revised: d g , g ∈ A (or θ (cid:107)C g (cid:107) (cid:54)

Original: α (cid:112) d g , g ∈ E ) is found.
Revised: α (cid:112) d g , g ∈ E ) is found.

Original: Based on the continuity of solutions given by ODEs solver, we can set w g = 

Revised: Based on the continuity of solutions given by ODEs solver, we can set w g =

Original: ( g ∈ A ∪ E ) as soon as it reaches the opposite sign (i.e. path pass through 0), meanwhile, index g is removed from set
Revised: ( g ∈ A ∪ E ) as soon as it reaches the opposite sign (i.e. path pass through 0), meanwhile, index g is removed from set

Original: A .
Revised: A .

Original: Alternatively, when the norm value of C g , g ∈ A (or φ θ (cid:107)C g (cid:107) , g ∈ E ) comes to α (cid:112) d g , the g -th group becomes active and will be added into active set.
Revised: Alternatively, when the norm value of C g , g ∈ A (or φ θ (cid:107)C g (cid:107) , g ∈ E ) comes to α (cid:112) d g , the g -th group becomes active and will be added into active set.

Original: (16) indicates that C g ( g ∈ A ∪ E ) shares the collinearity with w g .
Revised: (16) indicates that C g ( g ∈ A ∪ E ) shares the collinearity with w g .

Original: Therefore an extreme short syntropic coefﬁcient vector (cid:15) C g may be used as a approximation near transition point, which will give a start value for the following ODE solver, where (cid:15) is a user-deﬁned tuning parameter and only lead into a controllable approximation.
Revised: Therefore an extreme short syntropic coefﬁcient vector (cid:15) C g may be used as a approximation near transition point, which will give a start value for the following ODE solver, where (cid:15) is a user-deﬁned tuning parameter and only lead into a controllable approximation.

Original: Note that C g has the property of continuity, but is non-monotonic about θ .
Revised: Note that C g has the property of continuity, but is non-monotonic about θ .

Original: With that, we can keep observation on the sign and value of (cid:107)C g (cid:107) − α (cid:112) d g for each g in A (or θ (cid:107)C g (cid:107) − α (cid:112)
Revised: With that, we can keep observation on the sign and value of (cid:107)C g (cid:107) − α (cid:112) d g for each g in A (or θ (cid:107)C g (cid:107) − α (cid:112)

Original: d g for each g in
Revised: d g for each g in

Original: E ) while solving the solution path to estimate transition point.
Revised: E ) while solving the solution path to estimate transition point.

Original: B.2. (cid:96) p Penalization
Revised: B.2. (cid:96) p Penalization

Original: We give a start value of w 2 at θ = 0 by optimal conditions.
Revised: We give a start value of w 2 at θ = 0 by optimal conditions.

Original: Detailed steps are shown in our
Revised: Detailed steps are shown in our

Original: Algorithm 2.
Revised: Algorithm 2.

Original: Require:
Revised: Require:

Original: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =0 , regularization strength α .
Revised: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =0 , regularization strength α .

Original: 1: Set A , A and E , E according to w | θ =0 .
Revised: 1: Set A , A and E , E according to w | θ =0 .

Original: 2: Compute the ﬁrst transition point (cid:101) θ = min   αn (cid:112)
Revised: 2: Compute the ﬁrst transition point (cid:101) θ = min   αn (cid:112)

Original: d g (cid:13)(cid:13)(cid:13) (cid:101)
Revised: d g (cid:13)(cid:13)(cid:13) (cid:101)

Original: X Tg ( y − X
Revised: X Tg ( y − X

Original: A w A ) (cid:13)(cid:13)(cid:13)   , for g ∈ E .
Revised: A w A ) (cid:13)(cid:13)(cid:13)   , for g ∈ E .

Original: 3: Compute clipped (cid:98) θ = min (cid:110)(cid:101) θ, 1 (cid:111) .
Revised: 3: Compute clipped (cid:98) θ = min (cid:110)(cid:101) θ, 1 (cid:111) .

Original: 4: Update w | θ = (cid:98) θ = w | θ =0 .
Revised: 4: Update w | θ = (cid:98) θ = w | θ =0 .

Original: 5: if (cid:98) θ < 1 then 6: while θ ≤ 1 do 7:
Revised: 5: if (cid:98) θ < 1 then 6: while θ ≤ 1 do 7:

Original: Solve (8) and detect transition point simultaneously.
Revised: Solve (8) and detect transition point simultaneously.

Original: 8: if g -th group turns to inactive then 9: w g = 0 .
Revised: 8: if g -th group turns to inactive then 9: w g = 0 .

Original: Move g from A (or E ) to A (or E ).
Revised: Move g from A (or E ) to A (or E ).

Original: 11: else if g -th group becomes active then 12: w g = (cid:15) C g .
Revised: 11: else if g -th group becomes active then 12: w g = (cid:15) C g .

Original: Put g into A (or E ).
Revised: Put g into A (or E ).

Original: 14: end if 15:
Revised: 14: end if 15:

Original: Update w A , w E , X A and (cid:101) X E according to the updated A and E .
Revised: Update w A , w E , X A and (cid:101) X E according to the updated A and E .

Original: 16: end while 17: end
Revised: 16: end while 17: end

Original: if
Revised: if

Original: Ensure: w | θ =
Revised: Ensure: w | θ =

Original: Algorithm 
Revised: Algorithm 2 DFLS on (cid:96) p Penalization ( incremental )

Original: Require:
Revised: Require:

Original: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels  solution  | θ =0 ,  strength α , parameter p > 
Revised: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =0 , regularization strength α , parameter p > 1 .

Original: 1: Set w 2 | θ =0 by optimal conditions.
Revised: 1: Set w 2 | θ =0 by optimal conditions.

Original: 2: while θ ≤ 1 do 3:
Revised: 2: while θ ≤ 1 do 3:

Original: Solve (8).
Revised: Solve (8).

Original: 4: end while
Revised: 4: end while

Original: Ensure: w | θ =
Revised: Ensure

Original: Appendix C. Path Visualization
Revised: Appendix C. Path Visualization

Original: We plot the solution path of two types of regularizers in this section, which helps readers better understand their property.
Revised: We plot the solution path of two types of regularizers in this section, which helps readers better understand their property.

Original: Figure 2 shows an example of solution path of group Lasso with respect to successively varied θ , where θ = 1 corresponds to the adjustment of incorporating new features and θ = 0 corresponds to removing these features.
Revised: Figure 2 shows an example of solution path of group Lasso with respect to successively varied θ , where θ = 1 corresponds to the adjustment of incorporating new features and θ = 0 corresponds to removing these features.

Original: [Figure]
Revised: [Figure]

Original: The (cid:96) p penalization does not bring sparsity, so there is no transition point.
Revised: The (cid:96) p penalization does not bring sparsity, so there is no transition point.

Original: The ﬁgure 3 shows an example of its path visualization.
Revised: The ﬁgure 3 shows an example of its path visualization.

Original: [Figure]
Revised: [Figure]

Original: Appendix D. Implementation Details.
Revised: Appendix D. Implementation Details.

Original: We implement our novel algorithms in Python 3.7.
Revised: We implement our novel algorithms in Python 3.7.

Original: All the experiments were conducted on a Ubuntu machine with Intel 2.30GHz CPU × 72 and 47.0GB RAM.
Revised: All the experiments were conducted on a Ubuntu machine with Intel 2.30GHz CPU × 72 and 47.0GB RAM.

Original: Our real-world datasets are all available online at OpenML [15], we randomly selected 70% of the samples as training set for each data set.
Revised: Our real-world datasets are all available online at OpenML [15], we randomly selected 70% of the samples as training set for each data set.

Original: The MR#1 ∼ 3 are generated by applying a random linear regression model with no bias term.
Revised: The MR#1 ∼ 3 are generated by applying a random linear regression model with no bias term.

Original: The standard deviation of the Gaussian centered noise applied to the output is 1 .
Revised: The standard deviation of the Gaussian centered noise applied to the output is 1 .

Original: This setting is common in lots of related works.
Revised: This setting is common in lots of related works.

Original: The stop criterion of the batch training algorithm is set to 1e-7.
Revised: The stop criterion of the batch training algorithm is set to 1e-7.

Original: More importantly, we will provide our directly runnable code with a “Readme” tutorial in Supplementary Material, interested researchers can run Python 3.7 code to reproduce the main results.
Revised: More importantly, we will provide our directly runnable code with a “Readme” tutorial in Supplementary Material, interested researchers can run Python 3.7 code to reproduce the main results.

Original: Appendix E. Feature Decremental Learning E.1. Group Lasso
Revised: Appendix E. Feature Decremental Learning E.1. Group Lasso

Original: We present the feature decremental learning procedure of group Lasso in Algorithm 3.
Revised: We present the feature decremental learning procedure of group Lasso in Algorithm 3.

Original: To reduce dimensionality, we can track the solution path as θ varies from 1 to 0 , which can be considered as an inverse process of incremental learning.
Revised: To reduce dimensionality, we can track the solution path as θ varies from 1 to 0 , which can be considered as an inverse process of incremental learning.

Original: In particular, we can directly eliminate (cid:101) X as we found E = ∅ ( i.e. , they are irrelevant or redundant features).
Revised: In particular, we can directly eliminate (cid:101) X as we found E = ∅ ( i.e. , they are irrelevant or redundant features).

Original: Similar to incremental learning for features, this is supported by equations (4) & (5) of main body.
Revised: Similar to incremental learning for features, this is supported by equations (4) & (5) of main body.

Original: At last, we set w E∪E to exactto complete the whole process.
Revised: At last, we set w E∪E to exactto complete the whole process.

Original: Algorithm 3 DFLS on Group Lasso ( decremental )
Revised: Algorithm 3 DFLS on Group Lasso ( decremental )

Original: Require:
Revised: Require:

Original: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =1 , regularization strength α .
Revised: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =1 , regularization strength α .

Original: 1: Set A , A and E , E according to w | θ =1 .
Revised: 1: Set A , A and E , E according to w | θ =1 .

Original: 2: while E (cid:54) = ∅ and θ > 0 do 3:
Revised: 2: while E (cid:54) = ∅ and θ > 0 do 3:

Original: Solve d w A∪E dθ =
Revised: Solve d w A∪E dθ =

Original: (cid:20) β
Revised: (cid:20) β

Original: A β E (cid:21) and detect transition point simultaneously.
Revised: A β E (cid:21) and detect transition point simultaneously.

Original: 4: if g -th group turns to inactive then 5: w g = 0 .
Revised: 4: if g -th group turns to inactive then 5: w g = 0 .

Original: Move g from A (or E ) to A (or E ).
Revised: Move g from A (or E ) to A (or E ).

Original: 7: else if g -th group becomes active then 8: w g = (cid:15) C g .
Revised: 7: else if g -th group becomes active then 8: w g = (cid:15) C g .

Original: Put g into A (or E ).
Revised: Put g into A (or E ).

Original: 10: end if 11:
Revised: 10: end if 11:

Original: Update wA , w E , X A and (cid:101)X E according to the updated A and E .
Revised: Update w (cid:101) to the updated A and E .

Original: 12: end while 13: Set w E∪E = 0 .
Revised: 12: end while 13: Set w E∪ E = 0 .

Original: Ensure: w | θ =
Revised: Ensure: w | θ =

Original: E.2. (cid:96) p -norm Regularized Regression
Revised: E.2. (cid:96) p -norm Regularized Regression

Original: At this part, we present the pseudo-code of feature decremental algorithm for (cid:96) p -norm regularized regression in Algorithm 4.
Revised: At this part, we present the pseudo-code of feature decremental algorithm for (cid:96) p -norm regularized regression in Algorithm 4.

Original: E.3. Simulation Studies

Original: In this subsection, we empirically show the accuracy of derived ODEs in Table 5 and Table 6.

Original: In experimental settings, DFLS on different group partitions in group Lasso (denotes as G ) and

Original: Require:
Revised: Require:

Original: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =1 , regularization strength α , parameter p > 1 .
Revised: Samples X = (cid:104) X, (cid:101) X (cid:105) , labels y , initial solution w | θ =1 , regularization strength α , parameter p > 1 .

Original: 1: while θ > 0 do 2:
Revised: 1: while θ > 0 do 2:

Original: Solve ODEs to compute the solution path.
Revised: Solve ODEs to compute the solution path.

Original: 3: end while 4: Set w E∪E = 0 .
Revised: 3: end while 4: Set w E∪E = 0 .

Original: Ensure: w | θ =  various (cid:96) p -norm are used to perform features shrinkage while different regularization parameter α are chosen.
Revised: Ensure: w | θ = E.3. Simulation Studies In this subsection, we empirically show the accuracy of derived ODEs in Table 5 and Table 6. In experimental settings, DFLS on different group partitions in group Lasso (denotes as G ) and various (cid:96) p -norm are used to perform features shrinkage while different regularization parameter α are chosen.

Original: We conduct comparative experiments in the same environment as in the main paper. 
Revised: We conduct comparative experiments in the same environment as in the main paper. [Table] [Table] [Table]

Revised: II

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Any variance less than 10 − 5 are omitted.

Revised: DFLS and batch represent each iteration is trained with a chunk of new features, or using batch algorithm to retrain from scratch, respectively

Original: [Table]
Revised: [Table]

Original: Appendix F. Additional Discussion
Revised: Appendix F. Additional Discussion

Original: This section we will give some discussions on our DFLS .
Revised: This section we will give some discussions on our DFLS .

Original: Note if the whole path is smooth on (0 , 1) ( i.e., when (cid:101) θ ≥ 1 for sparsity-based penalization or non-sparsity regularization R ( · ) is used), algorithm needn’t cost extra time on detecting speciﬁc locations and recalculating the solution path, which apparently reduces the computation complexity.
Revised: Note if the whole path is smooth on (0 , 1) ( i.e., when (cid:101) θ ≥ 1 for sparsity-based penalization or non-sparsity regularization R ( · ) is used), algorithm needn’t cost extra time on detecting speciﬁc locations and recalculating the solution path, which apparently reduces the computation complexity.

Original: The concrete process of designing an algorithm for dynamic updating by DFLS mainly consists of computing ∇ 2 R ( · ) and investigating optimality conditions.
Revised: The concrete process of designing an algorithm for dynamic updating by DFLS mainly consists of computing ∇ 2 R ( · ) and investigating optimality conditions.

Original: Hence, from a computational perspective, regularizer with easily computed ∇ 2 R ( · ) and ∇R ( · ) can have a better adaptation to our
Revised: Hence, from a computational perspective, regularizer with easily computed ∇ 2 R ( · ) and ∇R ( · ) can have a better adaptation to our

Original: DFLS framework
Revised: DFLS framework.

Original: [Table]