Original: C ORESETS FOR A CCELERATING I NCREMENTAL G RADIENT M ETHODS
Revised: C ORESETS FOR A CCELERATING I NCREMENTAL G RADIENT M ETHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many machine learning problems reduce to the problem of minimizing an expected risk.
Revised: Many machine learning problems reduce to the problem of minimizing an expected risk.

Original: Incremental gradient (IG) methods, such as stochastic gradient descent and its variants, have been successfully used to train the largest of machine learning models.
Revised: Incremental gradient (IG) methods, such as stochastic gradient descent and its variants, have been successfully used to train the largest of machine learning models.

Original: IG methods, however, are in general slow to converge and sensitive to stepsize choices.
Revised: IG methods, however, are in general slow to converge and sensitive to stepsize choices.

Original: Therefore, much work has focused on speeding them up by reducing the variance of the estimated gradient or choosing better stepsizes.
Revised: Therefore, much work has focused on speeding them up by reducing the variance of the estimated gradient or choosing better stepsizes.

Original: An alternative strategy would be to select a carefully chosen subset of training data, train only on that subset, and hence speed up optimization.
Revised: An alternative strategy would be to select a carefully chosen subset of training data, train only on that subset, and hence speed up optimization.

Original: However, it remains an open question how to achieve this, both theoretically as well as practically, while not compromising on the quality of the ﬁnal model.
Revised: However, it remains an open question how to achieve this, both theoretically as well as practically, while not compromising on the quality of the ﬁnal model.

Original: Here we develop CRAIG, a method for selecting a weighted subset (or coreset) of training data in order to speed up IG methods.
Revised: Here we develop CRAIG, a method for selecting a weighted subset (or coreset) of training data in order to speed up IG methods.

Original: We prove that by greedily selecting a subset S of training data that minimizes the upper-bound on the estimation error of the full gradient, running IG on this subset will converge to the (near)optimal solution in the same number of epochs as running IG on the full data.
Revised: We prove that by greedily selecting a subset S of training data that minimizes the upper-bound on the estimation error of the full gradient, running IG on this subset will converge to the (near)optimal solution in the same number of epochs as running IG on the full data.

Original: But because at each epoch the gradients are computed only on the subset S , we obtain a speedup that is inversely proportional to the size of S .
Revised: But because at each epoch the gradients are computed only on the subset S , we obtain a speedup that is inversely proportional to the size of S .

Original: Our subset selection algorithm is fully general and can be applied to most IG methods.
Revised: Our subset selection algorithm is fully general and can be applied to most IG methods.

Original: We further demonstrate practical effectiveness of our algorithm, CRAIG, through an extensive set of experiments on several applications, including logistic regression and deep neural networks.
Revised: We further demonstrate practical effectiveness of our algorithm, CRAIG, through an extensive set of experiments on several applications, including logistic regression and deep neural networks.

Original: Experiments show that CRAIG, while achieving practically the same loss, speeds up IG methods by up to 10x for convex and 3x for non-convex (deep learning) problems.
Revised: Experiments show that CRAIG, while achieving practically the same loss, speeds up IG methods by up to 10x for convex and 3x for non-convex (deep learning) problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Mathematical optimization lies at the core of training large-scale machine learning systems, and is now widely used over massive data sets with great practical success, assuming sufﬁcient data resources are available.
Revised: Mathematical optimization lies at the core of training large-scale machine learning systems, and is now widely used over massive data sets with great practical success, assuming sufﬁcient data resources are available.

Original: Achieving this success, however, also requires large amounts of (often GPU) computing, as well as concomitant ﬁnancial expenditures and energy usage (Strubell et al., 2019).
Revised: Achieving this success, however, also requires large amounts of (often GPU) computing, as well as concomitant ﬁnancial expenditures and energy usage (Strubell et al., 2019).

Original: Signiﬁcantly decreasing these costs without decreasing the learnt system’s resulting accuracy is one of the grand challenges of machine learning and artiﬁcial intelligence today (Asi & Duchi, 2019).
Revised: Signiﬁcantly decreasing these costs without decreasing the learnt system’s resulting accuracy is one of the grand challenges of machine learning and artiﬁcial intelligence today (Asi & Duchi, 2019).

Original: Training machine learning models often reduces to the problem of optimizing a regularized empirical risk function.
Revised: Training machine learning models often reduces to the problem of optimizing a regularized empirical risk function.

Original: Given a convex loss l , and a µ -strongly convex regularizer r , one aims to ﬁnd model parameter vector x ∗ over the parameter space X that minimizes the loss f over the training data V :
Revised: Given a convex loss l , and a µ -strongly convex regularizer r , one aims to ﬁnd model parameter vector x ∗ over the parameter space X that minimizes the loss f over the training data V :

Original: [Equation]
Revised: [Equation]

Original: where V = { 1 , . . . , n } is an index set of the training data, and functions f i : R d → R are associated with training examples ( a i , y i ) , where a i ∈ R d is the feature vector, and
Revised: where V = { 1 , . . . , n } is an index set of the training data, and functions f i : R d → R are associated with training examples ( a i , y i ) , where a i ∈ R d is the feature vector, and

Original: y i is the point i ’s label.
Revised: y i is the point i ’s label.

Original: The standard Gradient Descent can ﬁnd the minimizer of this problem, but requires repeated computations of the full gradient ∇ f ( x ) —sum of the gradients over all training data points/functions i —and is therefore prohibitive for massive data sets.
Revised: The standard Gradient Descent can ﬁnd the minimizer of this problem, but requires repeated computations of the full gradient ∇ f ( x ) —sum of the gradients over all training data points/functions i —and is therefore prohibitive for massive data sets.

Original: This issue is further exacerbated in case of deep neural networks where gradient computations (backpropagation) are expensive.
Revised: This issue is further exacerbated in case of deep neural networks where gradient computations (backpropagation) are expensive.

Original: Incremental
Revised: Incremental

Original: Gradient (IG) methods, such as Stochastic Gradient Descent and its accelerated variants, including
Revised: Gradient (IG) methods, such as Stochastic Gradient Descent and its accelerated variants, including

Original: SGD with momentum (Qian, 1999), Adagrad (Duchi et al., 2011), Adam (Kingma & Ba, 2014),
Revised: SGD with momentum (Qian, 1999), Adagrad (Duchi et al., 2011), Adam (Kingma & Ba, 2014),

Original: SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) iteratively estimate the gradient on random subsets/batches of training data.
Revised: SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) iteratively estimate the gradient on random subsets/batches of training data.

Original: While this provides an unbiased estimate of the full gradient, the randomized batches introduce variance in the gradient estimate (Hofmann et al., 2015), and therefore stochastic gradient methods are in general slow to converge (Johnson & Zhang, 2013;
Revised: While this provides an unbiased estimate of the full gradient, the randomized batches introduce variance in the gradient estimate (Hofmann et al., 2015), and therefore stochastic gradient methods are in general slow to converge (Johnson & Zhang, 2013;

Original: Defazio et al., 2014).
Revised: Defazio et al., 2014).

Original: The majority of the work speeding up IG methods has thus primarily focused on reducing the variance of the gradient estimate (SAGA (Defazio et al., 2014), SVRG (Johnson &
Revised: The majority of the work speeding up IG methods has thus primarily focused on reducing the variance of the gradient estimate (SAGA (Defazio et al., 2014), SVRG (Johnson &

Original: Zhang, 2013), Katysha (Allen-Zhu, 2017)) or more carefully selecting the gradient stepsize (Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2014)).
Revised: Zhang, 2013), Katysha (Allen-Zhu, 2017)) or more carefully selecting the gradient stepsize (Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2014)).

Original: However, the direction that remains largely unexplored is how to carefully select a small subset
Revised: However, the direction that remains largely unexplored is how to carefully select a small subset

Original: S ⊆ V of the full training data V , so that the model can only be trained on the subset S while still (approximately) converging to the globally optimal solution (i.e., the model parameters that would be obtained if training/optimizing on the full V ).
Revised: S ⊆ V of the full training data V , so that the model can only be trained on the subset S while still (approximately) converging to the globally optimal solution (i.e., the model parameters that would be obtained if training/optimizing on the full V ).

Original: If such a subset S can be quickly found, then this would directly lead to a speedup of | V | / | S | (which can be very large if | S | (cid:28) | V | ) per epoch of IG.
Revised: If such a subset S can be quickly found, then this would directly lead to a speedup of | V | / | S | (which can be very large if | S | (cid:28) | V | ) per epoch of IG.

Original: There are four main challenges in ﬁnding such a subset S .
Revised: There are four main challenges in ﬁnding such a subset S .

Original: First is that a guiding principle to select
Revised: First is that a guiding principle to select

Original: S is unclear.
Revised: S is unclear.

Original: For example, selecting training points close to the decision boundary might allow the model to ﬁne tune the decision boundary, while picking the most diverse set of data points would allow the model to get a better sense of the training data distribution.
Revised: For example, selecting training points close to the decision boundary might allow the model to ﬁne tune the decision boundary, while picking the most diverse set of data points would allow the model to get a better sense of the training data distribution.

Original: Second is that ﬁnding S must be fast as otherwise identifying the set S may take longer than the actual optimization, and so no overall speed-up would be achieved.
Revised: Second is that ﬁnding S must be fast as otherwise identifying the set S may take longer than the actual optimization, and so no overall speed-up would be achieved.

Original: Third is that ﬁnding a subset S is not enough.
Revised: Third is that ﬁnding a subset S is not enough.

Original: One also has to decide on an ordering over S and a gradient stepsize for each data point in S , as they affect the convergence.
Revised: One also has to decide on an ordering over S and a gradient stepsize for each data point in S , as they affect the convergence.

Original: And last, while the method might work well empirically on some data sets, one also requires theoretical understanding and mathematical convergence guarantees.
Revised: And last, while the method might work well empirically on some data sets, one also requires theoretical understanding and mathematical convergence guarantees.

Original: Here we develop CoResets for Accelerating Incremental Gradient descent ( CRAIG ) , for selecting a subset of training data points to speed up training of large machine learning models.
Revised: Here we develop CoResets for Accelerating Incremental Gradient descent ( CRAIG ) , for selecting a subset of training data points to speed up training of large machine learning models.

Original: Our key idea is to directly approximate the gradient.
Revised: Our key idea is to directly approximate the gradient.

Original: That is, we aim to ﬁnd a weighted and ordered subset S of training data V that is representative of the full gradient of V .
Revised: That is, we aim to ﬁnd a weighted and ordered subset S of training data V that is representative of the full gradient of V .

Original: We prove that the subset S that minimizes an upper-bound on the error of estimating the full gradient of V maximizes a submodular facility location function.
Revised: We prove that the subset S that minimizes an upper-bound on the error of estimating the full gradient of V maximizes a submodular facility location function.

Original: As a result, the subset S can be efﬁciently found using a fast greedy algorithm.
Revised: As a result, the subset S can be efﬁciently found using a fast greedy algorithm.

Original: A further beneﬁt of our approach is that set S is created incrementally which induces a natural ordering over data in S .
Revised: A further beneﬁt of our approach is that set S is created incrementally which induces a natural ordering over data in S .

Original: Thus, rather than processing data points in a random or arbitrary order, CRAIG processes them using in the order provided by the procedure, which we show further speeds up the convergence of the method.
Revised: Thus, rather than processing data points in a random or arbitrary order, CRAIG processes them using in the order provided by the procedure, which we show further speeds up the convergence of the method.

Original: We also provide theoretical analysis of CRAIG and prove the convergence of our approach.
Revised: We also provide theoretical analysis of CRAIG and prove the convergence of our approach.

Original: In particular, for a µ -strongly convex risk function and a subset S selected by CRAIG that estimates the full gradient by an error of at most (cid:15) , we prove that IG method with diminishing stepsize α k
Revised: In particular, for a µ -strongly convex risk function and a subset S selected by CRAIG that estimates the full gradient by an error of at most (cid:15) , we prove that IG method with diminishing stepsize α k

Original: = α/k τ at epoch k
Revised: = α/k τ at epoch k

Original: (with 0 < τ < 1 and 0 < α ), converges to an 2 R(cid:15)/µ neighborhood of the optimal solution at rate O (1 / √ k ) .
Revised: (with 0 < τ < 1 and 0 < α ), converges to an 2 R(cid:15)/µ neighborhood of the optimal solution at rate O (1 / √ k ) .

Original: Here, R = min { d 0 , ( rγ max C + (cid:15) ) /µ } where d 0 is the initial distance to the optimum, C is an upper-bound on the norm of the gradients, r = | S | , and γ max is the largest weight for the elements in the subset obtained by CRAIG.
Revised: Here, R = min { d 0 , ( rγ max C + (cid:15) ) /µ } where d 0 is the initial distance to the optimum, C is an upper-bound on the norm of the gradients, r = | S | , and γ max is the largest weight for the elements in the subset obtained by CRAIG.

Original: Moreover, we prove that if in addition to the strong convexity, component functions have smooth gradients, IG with the same diminishing stepsize on subset S converges to a 2 (cid:15)/µ neighborhood of the optimum solution at rate O (1 /k τ ) .
Revised: Moreover, we prove that if in addition to the strong convexity, component functions have smooth gradients, IG with the same diminishing stepsize on subset S converges to a 2 (cid:15)/µ neighborhood of the optimum solution at rate O (1 /k τ ) .

Original: The above convergence rates are the same as convergence rate of IG on V for a strongly convex risk (and smooth component functions), and therefore IG on S converges in the same number epochs as IG on the full V .
Revised: The above convergence rates are the same as convergence rate of IG on V for a strongly convex risk (and smooth component functions), and therefore IG on S converges in the same number epochs as IG on the full V .

Original: But because every epoch only uses a subset S of the data, it requires fewer gradient computations and thus leads to a | V | / | S | speedup over traditional IG methods, while still (approximately) converging to the optimal solution.
Revised: But because every epoch only uses a subset S of the data, it requires fewer gradient computations and thus leads to a | V | / | S | speedup over traditional IG methods, while still (approximately) converging to the optimal solution.

Original: Furthermore, CRAIG only requires the knowledge of estimated gradient differences and does not involve any (exact) gradient calculations.
Revised: Furthermore, CRAIG only requires the knowledge of estimated gradient differences and does not involve any (exact) gradient calculations.

Original: Therefore, CRAIG can be used as a simple preprocessing step before IG starts and no additional storage or gradient calculations are required during IG, which makes CRAIG extremely practical.
Revised: Therefore, CRAIG can be used as a simple preprocessing step before IG starts and no additional storage or gradient calculations are required during IG, which makes CRAIG extremely practical.

Original: As such CRAIG can be used to speed up any existing IG methods, including IG, Adam, SAGA, SVRG as we show in the experiments section.
Revised: As such CRAIG can be used to speed up any existing IG methods, including IG, Adam, SAGA, SVRG as we show in the experiments section.

Original: We demonstrate the effectiveness of CRAIG via an extensive set of experiments using logistic regression (a convex optimization problem) as well as training neural networks (non-convex optimization problems).
Revised: We demonstrate the effectiveness of CRAIG via an extensive set of experiments using logistic regression (a convex optimization problem) as well as training neural networks (non-convex optimization problems).

Original: We show that CRAIG speeds up incremental gradient methods, including SGD, SAGA,
Revised: We show that CRAIG speeds up incremental gradient methods, including SGD, SAGA,

Original: SVRG, Adam, Adagrad, and NAG.
Revised: SVRG, Adam, Adagrad, and NAG.

Original: In particular, CRAIG while achieving practically the same loss and accuracy as the underlying incremental gradient descent methods, speeds up gradient methods by up to 10x for convex and 3x for non-convex loss functions.
Revised: In particular, CRAIG while achieving practically the same loss and accuracy as the underlying incremental gradient descent methods, speeds up gradient methods by up to 10x for convex and 3x for non-convex loss functions.

Original: We also demonstrate that the deliberate ordering scheme of the CRAIG algorithm signiﬁcantly improves convergence time.
Revised: We also demonstrate that the deliberate ordering scheme of the CRAIG algorithm signiﬁcantly improves convergence time.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Convergence of IG methods has been long studied under various conditions (Zhi-Quan & Paul, 1994;
Revised: Convergence of IG methods has been long studied under various conditions (Zhi-Quan & Paul, 1994;

Original: Mangasariany & Solodovy, 1994; Bertsekas, 1996; Solodov, 1998; Tseng, 1998), however IG’s convergence rate has been characterized only more recently (see (Bertsekas, 2015) for a survey).
Revised: Mangasariany & Solodovy, 1994; Bertsekas, 1996; Solodov, 1998; Tseng, 1998), however IG’s convergence rate has been characterized only more recently (see (Bertsekas, 2015) for a survey).

Original: In particular, Nedi´c & Bertsekas (2001) provides a O (1 / √ k ) convergence rate for diminishing stepsizes α k per epoch
Revised: In particular, Nedi´c & Bertsekas (2001) provides a O (1 / √ k ) convergence rate for diminishing stepsizes α k per epoch

Original: k under a strong convexity assumption, and Gürbüzbalaban et al.
Revised: k under a strong convexity assumption, and Gürbüzbalaban et al.

Original: (2015) proves a O (1 /k τ ) convergence rate with diminishing stepsizes α k = Θ(1 /k
Revised: (2015) proves a O (1 /k τ ) convergence rate with diminishing stepsizes α k = Θ(1 /k

Original: τ ) for τ ∈ (0 , 1] under an additional smoothness assumption for the components.
Revised: τ ) for τ ∈ (0 , 1] under an additional smoothness assumption for the components.

Original: While these works provide convergence on the full dataset, our analysis provides the same convergence rates on subsets obtained by CRAIG.
Revised: While these works provide convergence on the full dataset, our analysis provides the same convergence rates on subsets obtained by CRAIG.

Original: It has been empirically observed that ordering of data signiﬁcantly affects the convergence rate of IG.
Revised: It has been empirically observed that ordering of data signiﬁcantly affects the convergence rate of IG.

Original: However, ﬁnding a favorable ordering for IG has been a long standing open question.
Revised: However, ﬁnding a favorable ordering for IG has been a long standing open question.

Original: Among the few results are that of (Recht & Re, 2012) showing that without-replacement random sampling improves convergence of IG for least means squares problem, and the very recent result of (Gurbuzbalaban et al., 2017) showing that a Random Reshufﬂing (RR) method with iterate averaging and a diminishing stepsize Θ(1 /k τ ) for τ ∈ (1 / 2 , 1) converges at rate Θ(1 /k 2 τ ) with probability one in the suboptimality of the objective value, thus improving upon the Ω(1 /k ) rate of SGD.
Revised: Among the few results are that of (Recht & Re, 2012) showing that without-replacement random sampling improves convergence of IG for least means squares problem, and the very recent result of (Gurbuzbalaban et al., 2017) showing that a Random Reshufﬂing (RR) method with iterate averaging and a diminishing stepsize Θ(1 /k τ ) for τ ∈ (1 / 2 , 1) converges at rate Θ(1 /k 2 τ ) with probability one in the suboptimality of the objective value, thus improving upon the Ω(1 /k ) rate of SGD.

Original: Contrary to the above randomized analysis, we propose the ﬁrst deterministic ordering on the data points and empirically show that the ordering provided by CRAIG provides a signiﬁcant speedup for the convergence of IG.
Revised: Contrary to the above randomized analysis, we propose the ﬁrst deterministic ordering on the data points and empirically show that the ordering provided by CRAIG provides a signiﬁcant speedup for the convergence of IG.

Original: Techniques for speeding up SGD, are mostly focused on variance reduction techniques (Roux et al., 2012; Shalev-Shwartz & Zhang, 2013; Johnson & Zhang, 2013; Hofmann et al., 2015; Allen-Zhu et al., 2016), and accelerated gradient methods when the regularization parameter is small (Frostig et al., 2015; Lin et al., 2015; Xiao & Zhang, 2014).
Revised: Techniques for speeding up SGD, are mostly focused on variance reduction techniques (Roux et al., 2012; Shalev-Shwartz & Zhang, 2013; Johnson & Zhang, 2013; Hofmann et al., 2015; Allen-Zhu et al., 2016), and accelerated gradient methods when the regularization parameter is small (Frostig et al., 2015; Lin et al., 2015; Xiao & Zhang, 2014).

Original: Very recently, Hofmann et al. (2015); Allen-Zhu et al.
Revised: Very recently, Hofmann et al. (2015); Allen-Zhu et al.

Original: (2016) exploited neighborhood structure to further reduce the variance of stochastic gradient descent and improve its running time.
Revised: (2016) exploited neighborhood structure to further reduce the variance of stochastic gradient descent and improve its running time.

Original: Our CRAIG method and analysis are complementary to variance reduction and accelerated methods.
Revised: Our CRAIG method and analysis are complementary to variance reduction and accelerated methods.

Original: CRAIG can be applied to these methods as well to speed them up (as we show in experiments).
Revised: CRAIG can be applied to these methods as well to speed them up (as we show in experiments).

Original:  C ORESETS FOR I NCREMENTAL G RADIENT D ESCENT (CRAIG)
Revised:  C ORESETS FOR I NCREMENTAL G RADIENT D ESCENT (CRAIG)

Original: We proceed as follows:
Revised: We proceed as follows:

Original: First, we deﬁne an objective function L for selecting an optimal set S of size r that best approximates the gradient of the full training dataset V of size n .
Revised: First, we deﬁne an objective function L for selecting an optimal set S of size r that best approximates the gradient of the full training dataset V of size n .

Original: Then, we show that L can be turned into a submodular function F and thus S can be efﬁciently found using a greedy algorithm.
Revised: Then, we show that L can be turned into a submodular function F and thus S can be efﬁciently found using a greedy algorithm.

Original: Crucially, we also show that the approximation error between the estimated and the true gradient can be efﬁciently minimized in a way that is independent of the actual optimization procedure and thus
Revised: Crucially, we also show that the approximation error between the estimated and the true gradient can be efﬁciently minimized in a way that is independent of the actual optimization procedure and thus

Original: CRAIG can simply be used as a preprocessing step before the actual optimization starts.
Revised: CRAIG can simply be used as a preprocessing step before the actual optimization starts.

Original: Incremental gradient methods aim at estimating the full gradient (cid:80) i ∈
Revised: Incremental gradient methods aim at estimating the full gradient (cid:80) i ∈

Original: V ∇ f i ( x ) over V by iteratively making a step based on the gradient of every function f i .
Revised: V ∇ f i ( x ) over V by iteratively making a step based on the gradient of every function f i .

Original: Our key idea in CRAIG is that if we can ﬁnd a small subset S such that the weighted sum of the gradients of its elements closely approximates the full gradient over V , we can apply IG only to the set S (with stepsizes equal to the weight of the elements in S ), and we should still converge to the (approximately) optimal solution, but much faster.
Revised: Our key idea in CRAIG is that if we can ﬁnd a small subset S such that the weighted sum of the gradients of its elements closely approximates the full gradient over V , we can apply IG only to the set S (with stepsizes equal to the weight of the elements in S ), and we should still converge to the (approximately) optimal solution, but much faster.

Original: Speciﬁcally, our goal in CRAIG is to ﬁnd the smallest subset S ⊆ V and corresponding per-element stepsizes γ j > 0 that approximate the full gradient with an error at most (cid:15) > 0 for all the possible values of the optimization parameters x ∈ X .
Revised: Speciﬁcally, our goal in CRAIG is to ﬁnd the smallest subset S ⊆ V and corresponding per-element stepsizes γ j > 0 that approximate the full gradient with an error at most (cid:15) > 0 for all the possible values of the optimization parameters x ∈ X .

Original: [Equation]
Revised: [Equation]

Original: Given such an S ∗ and associated weights { γ } j , we are guaranteed that gradient updates on S ∗ will be similar to the gradient updates on V regardless of the value of x .
Revised: Given such an S ∗ and associated weights { γ } j , we are guaranteed that gradient updates on S ∗ will be similar to the gradient updates on V regardless of the value of x .

Original: Unfortunately, directly solving the above optimization problem is not feasible, due to two problems.
Revised: Unfortunately, directly solving the above optimization problem is not feasible, due to two problems.

Original: Problem 1: Eq.
Revised: Problem 1: Eq.

Original: (2) requires us to calculate the gradient of all the functions
Revised: (2) requires us to calculate the gradient of all the functions

Original: f i over the entire space
Revised: f i over the entire space

Original: X , which is too expensive and would not lead to overall speedup.
Revised: X , which is too expensive and would not lead to overall speedup.

Original: In other words, it would appear that solving for S ∗ is as difﬁcult as solving problem (1), as it involves calculating (cid:80) i ∈
Revised: In other words, it would appear that solving for S ∗ is as difﬁcult as solving problem (1), as it involves calculating (cid:80) i ∈

Original: V ∇ f i ( x ) for various x ∈ X .
Revised: V ∇ f i ( x ) for various x ∈ X .

Original: And Problem 2: even if calculating the normed difference between the gradients in
Revised: And Problem 2: even if calculating the normed difference between the gradients in

Original: Eq.
Revised: Eq.

Original: (2) would be fast, as we discuss later ﬁnding the optimal subset S ∗ in NP-hard.
Revised: (2) would be fast, as we discuss later ﬁnding the optimal subset S ∗ in NP-hard.

Original: In the following, we address the above two challenges and discuss how we can quickly ﬁnd a near-optimal subset S .
Revised: In the following, we address the above two challenges and discuss how we can quickly ﬁnd a near-optimal subset S .

Original: 3. U PPER - BOUND ON THE E STIMATION E RROR
Revised: 3. U PPER - BOUND ON THE E STIMATION E RROR

Original: We ﬁrst address Problem 1, i.e., how to quickly estimate the error/discrepancy of the weighted sum of gradients of functions f j associate with data points j ∈ S , vs the full gradient, for every x ∈ X .
Revised: We ﬁrst address Problem 1, i.e., how to quickly estimate the error/discrepancy of the weighted sum of gradients of functions f j associate with data points j ∈ S , vs the full gradient, for every x ∈ X .

Original: Let S be a subset of r data points.
Revised: Let S be a subset of r data points.

Original: Furthermore, assume that there is a mapping
Revised: Furthermore, assume that there is a mapping

Original: ς x : V → S that for every x ∈ X assigns every data point i ∈ V to one of the elements j in S , i.e., ς x ( i ) =
Revised: ς x : V → S that for every x ∈ X assigns every data point i ∈ V to one of the elements j in S , i.e., ς x ( i ) =

Original: j ∈ S .
Revised: j ∈ S .

Original: Let
Revised: Let

Original: C j = { i ∈ [ n ] | ς
Revised: C j = { i ∈ [ n ] | ς

Original: ( i ) =
Revised: ( i ) =

Original: j } ⊆
Revised: j } ⊆

Original: V be the set of data points that are assigned to j ∈ S , and γ j = | C
Revised: V be the set of data points that are assigned to j ∈ S , and γ j = | C

Original: j | be the number of such data points.
Revised: j | be the number of such data points.

Original: Hence, { C j } rj =1 form a partition of V .
Revised: Hence, { C j } rj =1 form a partition of V .

Original: Then, for any arbitrary (single) x ∈ X we can write (cid:88)
Revised: Then, for any arbitrary (single) x ∈ X we can write (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Subtracting and then taking the norm of the both sides, we get an upper bound on the error of estimating the full gradient with the weighted sum of the gradients of the functions f j for j ∈ S .
Revised: Subtracting and then taking the norm of the both sides, we get an upper bound on the error of estimating the full gradient with the weighted sum of the gradients of the functions f j for j ∈ S .

Original: I.e.,
Revised: I.e.,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the inequality follows from the triangle inequality.
Revised: where the inequality follows from the triangle inequality.

Original: The upper-bound in Eq.
Revised: The upper-bound in Eq.

Original: (5) is minimized when ς x assigns every i ∈ V to an element in S with most gradient similarity at x , or minimum
Revised: (5) is minimized when ς x assigns every i ∈ V to an element in S with most gradient similarity at x , or minimum

Original: Euclidean distance between the gradient vectors at x .
Revised: Euclidean distance between the gradient vectors at x .

Original: That is: ς x ( i ) ∈ argmin j ∈ S (cid:107)∇ f i ( x ) −∇ f j ( x ) (cid:107) .
Revised: That is: ς x ( i ) ∈ argmin j ∈ S (cid:107)∇ f i ( x ) −∇ f j ( x ) (cid:107) .

Original: Hence, min
Revised: Hence, min

Original: [Equation]
Revised: [Equation]

Original: The right hand side of Eq.
Revised: The right hand side of Eq.

Original: (6) is minimized when S is the set of r medoids (exemplars) for all the components in the gradient space.
Revised: (6) is minimized when S is the set of r medoids (exemplars) for all the components in the gradient space.

Original: So far, we considered upper-bounding the gradient estimation error at a particular x ∈ X .
Revised: So far, we considered upper-bounding the gradient estimation error at a particular x ∈ X .

Original: To bound the estimation error for all x ∈ X , we consider a worst-case approximation of the estimation error over the entire parameter space X .
Revised: To bound the estimation error for all x ∈ X , we consider a worst-case approximation of the estimation error over the entire parameter space X .

Original: Formally, we deﬁne a distance metric d ij between gradients of f i and f j as the maximum normed difference between ∇ f i ( x ) and ∇ f j ( x ) over all x ∈ X :
Revised: Formally, we deﬁne a distance metric d ij between gradients of f i and f j as the maximum normed difference between ∇ f i ( x ) and ∇ f j ( x ) over all x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: Thus, by solving the following minimization problem, we obtain the smallest weighted subset S ∗ that approximates the full gradient by an error of at most (cid:15) for all x ∈ X :
Revised: Thus, by solving the following minimization problem, we obtain the smallest weighted subset S ∗ that approximates the full gradient by an error of at most (cid:15) for all x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: such that L
Revised: such that L

Original: [Equation]
Revised: [Equation]

Original: Note that Eq.
Revised: Note that Eq.

Original: (8) requires that the gradient error is bounded over X .
Revised: (8) requires that the gradient error is bounded over X .

Original: However, we show (Appendix
Revised: However, we show (Appendix

Original: B) for several classes of convex problems, including linear regression, ridge regression, logistic regression, and regularized support vector machines (SVMs), the normed gradient difference between data points can be efﬁciently boundedly approximated by (Allen-Zhu et al., 2016; Hofmann et al., 2015):
Revised: B) for several classes of convex problems, including linear regression, ridge regression, logistic regression, and regularized support vector machines (SVMs), the normed gradient difference between data points can be efﬁciently boundedly approximated by (Allen-Zhu et al., 2016; Hofmann et al., 2015):

Original: [Equation]
Revised: [Equation]

Original: Note when (cid:107) x (cid:107) is bounded for all x ∈ X , i.e., max x ∈X O ( (cid:107) x (cid:107) ) < ∞ , upper-bounds on the Euclidean distances between the gradients can be pre-computed.
Revised: Note when (cid:107) x (cid:107) is bounded for all x ∈ X , i.e., max x ∈X O ( (cid:107) x (cid:107) ) < ∞ , upper-bounds on the Euclidean distances between the gradients can be pre-computed.

Original: This is crucial, because it means that estimation error of the full gradient can be efﬁciently bounded independent of the actual optimization problem (i.e., point x ).
Revised: This is crucial, because it means that estimation error of the full gradient can be efﬁciently bounded independent of the actual optimization problem (i.e., point x ).

Original: Thus, these upper-bounds can be computed only once as a pre-processing step before any training takes place, and then used to ﬁnd the subset S by solving the optimization problem (8).
Revised: Thus, these upper-bounds can be computed only once as a pre-processing step before any training takes place, and then used to ﬁnd the subset S by solving the optimization problem (8).

Original: We address upper-bounding the normed difference between gradients for deep models in Section 3.4.
Revised: We address upper-bounding the normed difference between gradients for deep models in Section 3.4.

Original: 3. T HE CRAIG A LGORITHM
Revised: 3. T HE CRAIG A LGORITHM

Original: Optimization problem (8) produces a subset S of elements with their associated weights { γ } j ∈ S or per-element stepsizes that closely approximates the full gradient.
Revised: Optimization problem (8) produces a subset S of elements with their associated weights { γ } j ∈ S or per-element stepsizes that closely approximates the full gradient.

Original: Here, we show how to efﬁciently approximately solve the above optimization problem in order to ﬁnd a near-optimal subset S .
Revised: Here, we show how to efﬁciently approximately solve the above optimization problem in order to ﬁnd a near-optimal subset S .

Original: The optimization problem (8) is NP-hard as it involves calculating the value of L ( S ) for all the 2 | V | subsets S ⊆ V .
Revised: The optimization problem (8) is NP-hard as it involves calculating the value of L ( S ) for all the 2 | V | subsets S ⊆ V .

Original: We show, however, to transform it into a submodular set cover problem , for which efﬁcient approximation algorithms exist.
Revised: We show, however, to transform it into a submodular set cover problem , for which efﬁcient approximation algorithms exist.

Original: Formally, F is submodular if F ( S ∪ { e } ) − f ( S ) ≥ F ( T ∪ { e } ) − F ( T ) , for any S ⊆ T
Revised: Formally, F is submodular if F ( S ∪ { e } ) − f ( S ) ≥ F ( T ∪ { e } ) − F ( T ) , for any S ⊆ T

Original: ⊆ V and e ∈ V \ T .
Revised: ⊆ V and e ∈ V \ T .

Original: We denote the marginal utility of an element s w.r.t.
Revised: We denote the marginal utility of an element s w.r.t.

Original: a subset S as F ( e | S )
Revised: a subset S as F ( e | S )

Original: =
Revised: =

Original: F ( S ∪ { e } ) − F ( S ) .
Revised: F ( S ∪ { e } ) − F ( S ) .

Original: Function F is called monotone if F ( e | S ) ≥ 0 for any e ∈ V \ S and S ⊆ V .
Revised: Function F is called monotone if F ( e | S ) ≥ 0 for any e ∈ V \ S and S ⊆ V .

Original: The submodular cover problem is deﬁned as ﬁnding the smallest set S that achieves utility ρ .
Revised: The submodular cover problem is deﬁned as ﬁnding the smallest set S that achieves utility ρ .

Original: Precisely,
Revised: Precisely,

Original: [Equation]
Revised: [Equation]

Original: such that F ( S ) ≥ ρ.
Revised: such that F ( S ) ≥ ρ.

Original: [Equation]
Revised: [Equation]

Original: Although ﬁnding S ∗ is NP-hard since it captures such well-known NP-hard problems as Minimum
Revised: Although ﬁnding S ∗ is NP-hard since it captures such well-known NP-hard problems as Minimum

Original: Vertex Cover, for many classes of submodular functions (Nemhauser et al., 1978; Wolsey, 1982), a simple greedy algorithm is known to be very effective.
Revised: Vertex Cover, for many classes of submodular functions (Nemhauser et al., 1978; Wolsey, 1982), a simple greedy algorithm is known to be very effective.

Original: The greedy algorithm starts with the empty set S 0 = ∅ , and at each iteration i , it chooses an element e ∈ V that maximizes (cid:52) ( e | S i − 1 ) , i.e.,
Revised: The greedy algorithm starts with the empty set S 0 = ∅ , and at each iteration i , it chooses an element e ∈ V that maximizes (cid:52) ( e | S i − 1 ) , i.e.,

Original: S i
Revised: S i

Original: =
Revised: =

Original: S
Revised: S

Original: i − 1 ∪ { argmax e ∈ V F ( e | S
Revised: i − 1 ∪ { argmax e ∈ V F ( e | S

Original: i − 1 ) − F ( S i − 1 ) } .
Revised: i − 1 ) − F ( S i − 1 ) } .

Original: Greedy gives us a logarithmic approximation, i.e. | S | ≤ (cid:0) 1 + ln(max e F ( e |∅ )) (cid:1) | S ∗ | .
Revised: Greedy gives us a logarithmic approximation, i.e. | S | ≤ (cid:0) 1 + ln(max e F ( e |∅ )) (cid:1) | S ∗ | .

Original: The computational complexity of the greedy algorithm is O
Revised: The computational complexity of the greedy algorithm is O

Original: ( | V | · | S | ) .
Revised: ( | V | · | S | ) .

Original: However, its running time can be reduced to O ( | V | ) using stochastic algorithms (Mirzasoleiman et al., 2015a) and further improved using lazy evaluation (Minoux, 1978), and distributed implementations (Mirzasoleiman et al., 2015b; 2016).
Revised: However, its running time can be reduced to O ( | V | ) using stochastic algorithms (Mirzasoleiman et al., 2015a) and further improved using lazy evaluation (Minoux, 1978), and distributed implementations (Mirzasoleiman et al., 2015b; 2016).

Original: Given a subset S ⊆ V , the facility location function quantiﬁes the coverage of the whole data set
Revised: Given a subset S ⊆ V , the facility location function quantiﬁes the coverage of the whole data set

Original: V by the subset S by summing the similarities between every i ∈ V and its closest element j ∈ S .
Revised: V by the subset S by summing the similarities between every i ∈ V and its closest element j ∈ S .

Original: Formally, facility location is deﬁned as F fl ( S ) =
Revised: Formally, facility location is deﬁned as F fl ( S ) =

Original: (cid:80) i ∈ V max j ∈ S s i,j , where s i,j is the similarity between i, j ∈ V .
Revised: (cid:80) i ∈ V max j ∈ S s i,j , where s i,j is the similarity between i, j ∈ V .

Original: The facility location function has been used in a number of applications, including scene and documents summarization (Simon et al., 2007; Lin & Bilmes, 2012).
Revised: The facility location function has been used in a number of applications, including scene and documents summarization (Simon et al., 2007; Lin & Bilmes, 2012).

Original: By introducing an auxiliary element s 0 we can turn L ( S ) in Eq.
Revised: By introducing an auxiliary element s 0 we can turn L ( S ) in Eq.

Original: (8) into a monotone submodular facility location function,
Revised: (8) into a monotone submodular facility location function,

Original: [Equation]
Revised: [Equation]

Original: where L ( { s 0 } ) is a constant.
Revised: where L ( { s 0 } ) is a constant.

Original: In words, F measures the decrease in the estimation error associated with the set S versus the estimation error associated with just the auxiliary element.
Revised: In words, F measures the decrease in the estimation error associated with the set S versus the estimation error associated with just the auxiliary element.

Original: It is easy to see that for suitable choice of s 0 , maximizing F is equivalent to minimizing L .
Revised: It is easy to see that for suitable choice of s 0 , maximizing F is equivalent to minimizing L .

Original: Therefore, we apply the greedy algorithm to approximately solve the following problem to get the subset S deﬁned in 8:
Revised: Therefore, we apply the greedy algorithm to approximately solve the following problem to get the subset S deﬁned in 8:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: At every step, the greedy algorithm selects an element that reduces the upper bound on the estimation error the most.
Revised: At every step, the greedy algorithm selects an element that reduces the upper bound on the estimation error the most.

Original: In fact, the size of the smallest subset S that estimates the full gradient by an error of at most (cid:15) depends on the structural properties of the data.
Revised: In fact, the size of the smallest subset S that estimates the full gradient by an error of at most (cid:15) depends on the structural properties of the data.

Original: Intuitively, as long as the marginal gains of facility location are considerably large, we need more elements to improve our estimation of the full gradient.
Revised: Intuitively, as long as the marginal gains of facility location are considerably large, we need more elements to improve our estimation of the full gradient.

Original: Having found S , the weight γ j of every element j ∈ S is the number of components that are closest to it in the gradient space, and are used as stepsize of element j ∈ S during IG.
Revised: Having found S , the weight γ j of every element j ∈ S is the number of components that are closest to it in the gradient space, and are used as stepsize of element j ∈ S during IG.

Original: The pseudocode for CRAIG subset selection method is outlined in Algorithm 1.
Revised: The pseudocode for CRAIG subset selection method is outlined in Algorithm 1.

Original: 3. O RDERING ON THE ELEMENTS OF THE SUBSET
Revised: 3. O RDERING ON THE ELEMENTS OF THE SUBSET

Original: Notice that CRAIG creates subset S incrementally one element at a time, which produces a natural order σ to the elements in S .
Revised: Notice that CRAIG creates subset S incrementally one element at a time, which produces a natural order σ to the elements in S .

Original: Adding the element with largest marginal gain j ∈ argmax e ∈ V F ( e | S i − 1 ) improves our estimation from the full gradient by an amount bounded by the marginal gain.
Revised: Adding the element with largest marginal gain j ∈ argmax e ∈ V F ( e | S i − 1 ) improves our estimation from the full gradient by an amount bounded by the marginal gain.

Original: Formally, at every step i , we have F ( S i ) ≥ (1 − e − i/ | S | ) F ( S ∗ ) , and hence
Revised: Formally, at every step i , we have F ( S i ) ≥ (1 − e − i/ | S | ) F ( S ∗ ) , and hence

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the ﬁrst elements of the ordering contribute the most to provide a close approximation of the full gradient and the rest of the elements further reﬁne the approximation.
Revised: Intuitively, the ﬁrst elements of the ordering contribute the most to provide a close approximation of the full gradient and the rest of the elements further reﬁne the approximation.

Original: Hence, the ﬁrst incremental gradient updates gets us close to x ∗ , and the rest of the updates further reﬁne the solution.
Revised: Hence, the ﬁrst incremental gradient updates gets us close to x ∗ , and the rest of the updates further reﬁne the solution.

Original: We show experimentally that processing data points in the order of S leads to faster convergence than when we consider elements in S in a random order.
Revised: We show experimentally that processing data points in the order of S leads to faster convergence than when we consider elements in S in a random order.

Original: We defer the formal proof to future work.
Revised: We defer the formal proof to future work.

Original: 3. A PPLICATION OF CRAIG TO D EEP N ETWORKS
Revised: 3. A PPLICATION OF CRAIG TO D EEP N ETWORKS

Original: Incremental gradient methods, including SGD with momentum (Qian, 1999), Adam (Kingma &
Revised: Incremental gradient methods, including SGD with momentum (Qian, 1999), Adam (Kingma &

Original: Ba, 2014) and Adagrad (Duchi et al., 2011) are widely used to train deep networks.
Revised: Ba, 2014) and Adagrad (Duchi et al., 2011) are widely used to train deep networks.

Original: As discussed,
Revised: As discussed,

Original: CRAIG selects a subset that closely approximates the full gradient, and hence can be also applied for speeding up training deep networks.
Revised: CRAIG selects a subset that closely approximates the full gradient, and hence can be also applied for speeding up training deep networks.

Original: The challenge here is that we cannot use inequality (9) to bound the normed difference between gradients for all x ∈ X and ﬁnd the subset as a preprocessing step.
Revised: The challenge here is that we cannot use inequality (9) to bound the normed difference between gradients for all x ∈ X and ﬁnd the subset as a preprocessing step.

Original: However, it has been shown that for neural networks, the variation of the gradient norms is mostly captured by the gradient of the loss w.r.t.
Revised: However, it has been shown that for neural networks, the variation of the gradient norms is mostly captured by the gradient of the loss w.r.t.

Original: the last layer (see Section 3.2 of (Katharopoulos & Fleuret, 2018))
Revised: the last layer (see Section 3.2 of (Katharopoulos & Fleuret, 2018))

Original: that is often not expensive or only slightly more expensive than calculating the loss.
Revised: that is often not expensive or only slightly more expensive than calculating the loss.

Original: In many cases, where we have cross entropy loss with soft-max as the last layer, the gradient of the loss w.r.t.
Revised: In many cases, where we have cross entropy loss with soft-max as the last layer, the gradient of the loss w.r.t.

Original: the i -th input to the soft-max is simply p i − y i , where p i are logits (dimension p − 1 for p classes) and y is the one-hot encoded label.
Revised: the i -th input to the soft-max is simply p i − y i , where p i are logits (dimension p − 1 for p classes) and y is the one-hot encoded label.

Original: In this case, CRAIG does not need any backward pass or extra storage.
Revised: In this case, CRAIG does not need any backward pass or extra storage.

Original: That is, CRAIG can be applied at the beginning of every epoch to ﬁnd a subset for that epoch.
Revised: That is, CRAIG can be applied at the beginning of every epoch to ﬁnd a subset for that epoch.

Original: Note that, although CRAIG needs an additional O ( | V | · | S | ) complexity (or O ( | V | ) using stochastic greedy) to ﬁnd the subset S at the beginning of every epoch, this complexity does not involve any (exact) gradient calculations and is negligible compared to the cost of backpropagations performed during the epoch.
Revised: Note that, although CRAIG needs an additional O ( | V | · | S | ) complexity (or O ( | V | ) using stochastic greedy) to ﬁnd the subset S at the beginning of every epoch, this complexity does not involve any (exact) gradient calculations and is negligible compared to the cost of backpropagations performed during the epoch.

Original: Hence, as we show in the experiments CRAIG is practical and salable.
Revised: Hence, as we show in the experiments CRAIG is practical and salable.

Original:  C ONVERGENCE R ATE A NALYSIS OF CRAIG
Revised:  C ONVERGENCE R ATE A NALYSIS OF CRAIG

Original: The idea of CRAIG is to selects a subset that closely approximates the full gradient, and hence can be applied to speed up most IG variants as we show in our experiments.
Revised: The idea of CRAIG is to selects a subset that closely approximates the full gradient, and hence can be applied to speed up most IG variants as we show in our experiments.

Original: Here, we brieﬂy introduce the original IG method, and then prove the convergence rate of IG applied to subsets found by CRAIG.
Revised: Here, we brieﬂy introduce the original IG method, and then prove the convergence rate of IG applied to subsets found by CRAIG.

Original: 4. I NCREMENTAL G RADIENT M ETHODS (IG)
Revised: 4. I NCREMENTAL G RADIENT M ETHODS (IG)

Original: Incremental gradient (IG) methods are core algorithms for solving Problem (1) and are widely used and studied.
Revised: Incremental gradient (IG) methods are core algorithms for solving Problem (1) and are widely used and studied.

Original: IG aims at approximating the standard gradient method by sequentially stepping along the gradient of the component functions
Revised: IG aims at approximating the standard gradient method by sequentially stepping along the gradient of the component functions

Original: f i in a cyclic order.
Revised: f i in a cyclic order.

Original: Starting from an initial point x 10 ∈ R d , it makes k passes over all the n components.
Revised: Starting from an initial point x 10 ∈ R d , it makes k passes over all the n components.

Original: At every epoch k ≥ 1 , it iteratively updates x ki − 1 based on the gradient of f i for i = 1 , · · · , n using stepsize α k > 0 .
Revised: At every epoch k ≥ 1 , it iteratively updates x ki − 1 based on the gradient of f i for i = 1 , · · · , n using stepsize α k > 0 .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: with the convention that x k + = x kn .
Revised: with the convention that x k + = x kn .

Original: Note that for a closed and convex subset X of R d , the results can be projected onto X , and the update rule becomes
Revised: Note that for a closed and convex subset X of R d , the results can be projected onto X , and the update rule becomes

Original: [Equation]
Revised: [Equation]

Original: where P X denotes projection on the set X ⊂ R d .
Revised: where P X denotes projection on the set X ⊂ R d .

Original: IG with diminishing stepsizes converges at rate O (1 / √ k ) for strongly convex sum function (Nedi´c & Bertsekas, 2001).
Revised: IG with diminishing stepsizes converges at rate O (1 / √ k ) for strongly convex sum function (Nedi´c & Bertsekas, 2001).

Original: If in addition to the strong convexity of the sum function, every component function f
Revised: If in addition to the strong convexity of the sum function, every component function f

Original: i is smooth, IG with diminishing stepsizes α k = Θ(1 /k s ) , s ∈
Revised: i is smooth, IG with diminishing stepsizes α k = Θ(1 /k s ) , s ∈

Original: (0 , 1] converges at rate
Revised: (0 , 1] converges at rate

Original: O (1 /k s )
Revised: O (1 /k s )

Original: (Gürbüzbalaban et al., 2015).
Revised: (Gürbüzbalaban et al., 2015).

Original: The convergence rate analysis of IG is valid regardless of order of processing the elements.
Revised: The convergence rate analysis of IG is valid regardless of order of processing the elements.

Original: However, in practice, the convergence rate of IG is known to be quite sensitive to the order of processing the functions (Bertsekas & Scientiﬁc, 2015; Gurbuzbalaban et al., 2017).
Revised: However, in practice, the convergence rate of IG is known to be quite sensitive to the order of processing the functions (Bertsekas & Scientiﬁc, 2015; Gurbuzbalaban et al., 2017).

Original: If problem-speciﬁc knowledge can be used to ﬁnd a favorable order σ (deﬁned as a permutation { σ 1 , · · · , σ n } of { 1 , 2 , ..., n } ), IG can be updated to process the functions according to this order, i.e.,
Revised: If problem-speciﬁc knowledge can be used to ﬁnd a favorable order σ (deﬁned as a permutation { σ 1 , · · · , σ n } of { 1 , 2 , ..., n } ), IG can be updated to process the functions according to this order, i.e.,

Original: [Equation]
Revised: [Equation]

Original: In general a favorable order is not known in advance, and a common approach is sampling the function indices with replacement from the set { 1 , 2 , · · · , n } and is called the Stochastic Gradient
Revised: In general a favorable order is not known in advance, and a common approach is sampling the function indices with replacement from the set { 1 , 2 , · · · , n } and is called the Stochastic Gradient

Original: Descent (SGD) method, a.k.a.
Revised: Descent (SGD) method, a.k.a.

Original: the Robbins-Monro algorithm (Robbins & Monro, 1951) (also see (Bottou, 1998; Bertsekas, 2015; Nemirovski et al., 2009; Shalev-Shwartz & Srebro, 2008)).
Revised: the Robbins-Monro algorithm (Robbins & Monro, 1951) (also see (Bottou, 1998; Bertsekas, 2015; Nemirovski et al., 2009; Shalev-Shwartz & Srebro, 2008)).

Original: 4. C ONVERGENCE R ATE OF IG ON S UBSETS F OUND BY CRAIG
Revised: 4. C ONVERGENCE R ATE OF IG ON S UBSETS F OUND BY CRAIG

Original: Next we analyze the convergence rate of IG applied to the weighted and ordered subset S found by
Revised: Next we analyze the convergence rate of IG applied to the weighted and ordered subset S found by

Original: CRAIG.
Revised: CRAIG.

Original: In particular, we show that (1) applying IG to S converges to a close neighborhood of the optimal solution and that (2) this convergence happens at the same rate (same number of epochs) as
Revised: In particular, we show that (1) applying IG to S converges to a close neighborhood of the optimal solution and that (2) this convergence happens at the same rate (same number of epochs) as

Original: IG on the full data.
Revised: IG on the full data.

Original: Formally, every step of IG on the subset becomes
Revised: Formally, every step of IG on the subset becomes

Original: [Equation]
Revised: [Equation]

Original: Here, σ is a permutation of { 1 , 2 , · · · , r } , and the per-element stepsize γ s i for every function f
Revised: Here, σ is a permutation of { 1 , 2 , · · · , r } , and the per-element stepsize γ s i for every function f

Original: s i is the weight of the element s i ∈ S and is ﬁxed for all epochs.
Revised: s i is the weight of the element s i ∈ S and is ﬁxed for all epochs.

Original: 4. C ONVERGENCE R ATE FOR S TRONGLY C ONVEX F UNCTIONS
Revised: 4. C ONVERGENCE R ATE FOR S TRONGLY C ONVEX F UNCTIONS

Original: We ﬁrst provide the convergence analysis for the case where the sum function in Problem (1) is strongly convex, i.e. ∀ x, y ∈
Revised: We ﬁrst provide the convergence analysis for the case where the sum function in Problem (1) is strongly convex, i.e. ∀ x, y ∈

Original: R
Revised: R

Original: d we have that f ( y ) ≥ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + µ 2 (cid:107) x − y (cid:107) 2 .
Revised: d we have that f ( y ) ≥ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + µ 2 (cid:107) x − y (cid:107) 2 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume that (cid:80) i ∈
Revised: Assume that (cid:80) i ∈

Original: V f i ( x ) is strongly convex, and S is a weighted subset of size r such that L ( S )
Revised: V f i ( x ) is strongly convex, and S is a weighted subset of size r such that L ( S )

Original: = (cid:80) i ∈
Revised: = (cid:80) i ∈

Original: V min j ∈ S d ij ≤ (cid:15) .
Revised: V min j ∈ S d ij ≤ (cid:15) .

Original: Then for the iterates { x k = x k 0 } generated by applying IG to S with per-epoch stepsize α k = α/k τ with α > 0 and τ ∈ [0 , 1] , we have
Revised: Then for the iterates { x k = x k 0 } generated by applying IG to S with per-epoch stepsize α k = α/k τ with α > 0 and τ ∈ [0 , 1] , we have

Original: (i) if τ = 1 , then (cid:107) x k − x ∗ (cid:107) 2 ≤ 2 (cid:15)R/µ + r 2 γ 2max C 2 /kµ , (ii) if 0 < τ < 1 , then (cid:107) x
Revised: (i) if τ = 1 , then (cid:107) x k − x ∗ (cid:107) 2 ≤ 2 (cid:15)R/µ + r 2 γ 2max C 2 /kµ , (ii) if 0 < τ < 1 , then (cid:107) x

Original: k − x ∗ (cid:107) 2 ≤ 2 (cid:15)R/µ ,
Revised: k − x ∗ (cid:107) 2 ≤ 2 (cid:15)R/µ ,

Original: for k → ∞
Revised: for k → ∞

Original: (iii) if τ = 0 , then (cid:107) x k − x ∗ (cid:107) 2 ≤ (1 − αµ )
Revised: (iii) if τ = 0 , then (cid:107) x k − x ∗ (cid:107) 2 ≤ (1 − αµ )

Original: k +1 (cid:107) x
Revised: k +1 (cid:107) x

Original: 0 − x ∗ (cid:107)
Revised: 0 − x ∗ (cid:107)

Original: 2 + 2 (cid:15)R/µ + αr 2 γ 2max C 2 /µ, where C is an upper-bound on the norm of the component function gradients, i.e. max i ∈ V sup x ∈X (cid:107)∇ f i ( x ) (cid:107) ≤ C , γ max = max j ∈ S
Revised: 2 + 2 (cid:15)R/µ + αr 2 γ 2max C 2 /µ, where C is an upper-bound on the norm of the component function gradients, i.e. max i ∈ V sup x ∈X (cid:107)∇ f i ( x ) (cid:107) ≤ C , γ max = max j ∈ S

Original: γ j is the largest per-element step size, and
Revised: γ j is the largest per-element step size, and

Original: R = min { d 0 , ( rγ max C + (cid:15) ) /µ } , where d 0 = (cid:107) x 0 − x ∗ (cid:107) is the initial distance to the optimum x ∗ .
Revised: R = min { d 0 , ( rγ max C + (cid:15) ) /µ } , where d 0 = (cid:107) x 0 − x ∗ (cid:107) is the initial distance to the optimum x ∗ .

Original: All the proofs can be found in the Appendix.
Revised: All the proofs can be found in the Appendix.

Original: The above theorem shows that IG on S converges at the same rate O (1 / √ k ) of IG on the entire data set V .
Revised: The above theorem shows that IG on S converges at the same rate O (1 / √ k ) of IG on the entire data set V .

Original: However, compared to IG on V , the | V | / | S | speedup of IG on S comes at the price of getting an extra error term, 2 (cid:15)R/µ .
Revised: However, compared to IG on V , the | V | / | S | speedup of IG on S comes at the price of getting an extra error term, 2 (cid:15)R/µ .

Original: 4. C ONVERGENCE R ATE FOR S MOOTH AND S TRONGLY C ONVEX F UNCTIONS
Revised: 4. C ONVERGENCE R ATE FOR S MOOTH AND S TRONGLY C ONVEX F UNCTIONS

Original: If in addition to strong convexity of the expected risk, each component function has a Lipschitz gradient, i.e. ∀ x ∈ X , i ∈ [ n ] we have (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ β i (cid:107) x − y (cid:107) , then we get the following results about the iterates generated by applying IG to the weighted subset S returned by CRAIG.
Revised: If in addition to strong convexity of the expected risk, each component function has a Lipschitz gradient, i.e. ∀ x ∈ X , i ∈ [ n ] we have (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ β i (cid:107) x − y (cid:107) , then we get the following results about the iterates generated by applying IG to the weighted subset S returned by CRAIG.

Original: set with 581,012 data points.
Revised: set with 581,012 data points.

Original: We compare performance of CRAIG (10% selected subset) vs. entire data set.
Revised: We compare performance of CRAIG (10% selected subset) vs. entire data set.

Original: We achieve the average speedup of 7x for achieving similar loss residual and error rate across the three optimization methods.
Revised: We achieve the average speedup of 7x for achieving similar loss residual and error rate across the three optimization methods.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume that (cid:80) i ∈
Revised: Assume that (cid:80) i ∈

Original: V f i ( x ) is strongly convex and let f i ( x ) , i = 1 , 2 , · · · , n be convex and twice continuously differentiable component functions with Lipschitz gradients on X .
Revised: V f i ( x ) is strongly convex and let f i ( x ) , i = 1 , 2 , · · · , n be convex and twice continuously differentiable component functions with Lipschitz gradients on X .

Original: Given a subset S such that L ( S ) =
Revised: Given a subset S such that L ( S ) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: V min j ∈ S d ij ≤ (cid:15) .
Revised: V min j ∈ S d ij ≤ (cid:15) .

Original: Then for the iterates { x k = x k 0 } generated by applying IG to S with per-epoch stepsize α k = α/k τ with α > 0 and τ ∈ [0 , 1] , we have
Revised: Then for the iterates { x k = x k 0 } generated by applying IG to S with per-epoch stepsize α k = α/k τ with α > 0 and τ ∈ [0 , 1] , we have

Original: (i) if τ = 1 , then (cid:107) x k − x ∗ (cid:107) ≤ 2 (cid:15)/µ + βCrγ 2max /kµ , (ii) if 0 < τ < 1 , then (cid:107) x k − x ∗ (cid:107) ≤ 2 (cid:15)/µ , for
Revised: (i) if τ = 1 , then (cid:107) x k − x ∗ (cid:107) ≤ 2 (cid:15)/µ + βCrγ 2max /kµ , (ii) if 0 < τ < 1 , then (cid:107) x k − x ∗ (cid:107) ≤ 2 (cid:15)/µ , for

Revised: →

Original: → ∞ (iii) if τ = 0 , then (cid:107) x k − x ∗ (cid:107) ≤ (1 − αµ ) k (cid:107) x 0 − x ∗ (cid:107) + 2 (cid:15)/µ + αβCrγ 2max /µ ,
Revised: (iii) if τ = 0 , then (cid:107) x k − x ∗ (cid:107) ≤ (1 − αµ ) k (cid:107) x 0 − x ∗ (cid:107) + 2 (cid:15)/µ + αβCrγ 2max /µ ,

Original: where β = (cid:80) ni
Revised: where β = (cid:80) ni

Original: =1 β
Revised: =1 β

Original: i is the sum of gradient Lipschitz constants of the component functions.
Revised: i is the sum of gradient Lipschitz constants of the component functions.

Original: The above theorem shows that for τ > 0 , IG applied to S converges to an (cid:15)/µ neighborhood of the optimal solution, with a rate of O (1 /k τ ) which is the same convergence rate for IG on the entire data set V .
Revised: The above theorem shows that for τ > 0 , IG applied to S converges to a 2 (cid:15)/µ neighborhood of the optimal solution, with a rate of O (1 /k τ ) which is the same convergence rate for IG on the entire data set V .

Original: As shown in our experiments, in real data sets small weighted subsets constructed by CRAIG provide a close approximation to the full gradient.
Revised: As shown in our experiments, in real data sets small weighted subsets constructed by CRAIG provide a close approximation to the full gradient.

Original: Hence, applying IG to the weighted subsets returned by CRAIG provides a solution of the same or higher quality compared to the solution obtained by applying IG to the whole data set, in a considerably shorter amount of time.
Revised: Hence, applying IG to the weighted subsets returned by CRAIG provides a solution of the same or higher quality compared to the solution obtained by applying IG to the whole data set, in a considerably shorter amount of time.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experimental evaluation we wish to address the following questions: (1) How do loss and accuracy of IG applied to the subsets returned by CRAIG compare to loss and accuracy of IG applied to the entire data; (2) How small is the size of the subsets that we can select with CRAIG and still get a comparable performance to IG applied to the entire data; (3) how does the ordering affect the performance of IG on the subset; and (4) how well does CRAIG scale to large data sets, and extend to non-convex problems.
Revised: In our experimental evaluation we wish to address the following questions: (1) How do loss and accuracy of IG applied to the subsets returned by CRAIG compare to loss and accuracy of IG applied to the entire data; (2) How small is the size of the subsets that we can select with CRAIG and still get a comparable performance to IG applied to the entire data; (3) how does the ordering affect the performance of IG on the subset; and (4) how well does CRAIG scale to large data sets, and extend to non-convex problems.

Original: To this end, we apply CRAIG to several convex and non-convex problems.
Revised: To this end, we apply CRAIG to several convex and non-convex problems.

Original: In our experiments, we report the run-time as the wall-clock time for subset selection with
Revised: In our experiments, we report the run-time as the wall-clock time for subset selection with

Original: CRAIG, plus minimizing the loss using IG or other optimizers with the speciﬁed learning rates.
Revised: CRAIG, plus minimizing the loss using IG or other optimizers with the speciﬁed learning rates.

Original: For the classiﬁcation problems, we separately select subsets from each class while maintaining the class ratios in the whole data, and apply IG to the union of the subsets.
Revised: For the classiﬁcation problems, we separately select subsets from each class while maintaining the class ratios in the whole data, and apply IG to the union of the subsets.

Original: 5. C ONVEX E XPERIMENTS
Revised: 5. C ONVEX E XPERIMENTS

Original: In our convex experiments, we apply CRAIG to IG, as well as variance reduction methods SVRG (Johnson & Zhang, 2013), and SAGA (Defazio et al., 2014), that try to reduce the variance of SGD either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory.
Revised: In our convex experiments, we apply CRAIG to IG, as well as variance reduction methods SVRG (Johnson & Zhang, 2013), and SAGA (Defazio et al., 2014), that try to reduce the variance of SGD either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory.

Original: We apply L2-regularized logistic regression: f i ( x ) =
Revised: We apply L2-regularized logistic regression: f i ( x ) =

Original: ln(1+exp( − x T a i y i ))
Revised: ln(1+exp( − x T a i y i ))

Original: +0 .
Revised: +0 .

Original: 5 λx T x to classify the following three datasets from LIBSVM: (1) covtype.binary including 581,012 data points of 54 dimensions, (2) SensIT including 78,823 training and 19,705 test data points of dimension 100, and (3) Ijcnn1 including 49,990 training and 91,701 test data points of 22 dimensions.
Revised: 5 λx T x to classify the following three datasets from LIBSVM: (1) covtype.binary including 581,012 data points of 54 dimensions, (2) SensIT including 78,823 training and 19,705 test data points of dimension 100, and (3) Ijcnn1 including 49,990 training and 91,701 test data points of 22 dimensions.

Original: As covtype do not come with labeled test data, we randomly split the training data into halves to make the training/test split (training and set sets are consistent for different methods).
Revised: As covtype do not come with labeled test data, we randomly split the training data into halves to make the training/test split (training and set sets are consistent for different methods).

Original: For the convex experiments, λ is set to 10 − 5 .
Revised: For the convex experiments, λ is set to 10 − 5 .

Original: Figure 1 compares training loss residual and test error rate of IG, SVRG, and SAGA on the subsets of size 10% of covtype selected by CRAIG (with corresponding per-element stepsizes) to that of IG,
Revised: Figure 1 compares training loss residual and test error rate of IG, SVRG, and SAGA on the subsets of size 10% of covtype selected by CRAIG (with corresponding per-element stepsizes) to that of IG,

Original: SVRG, and SAGA on the entire data set.
Revised: SVRG, and SAGA on the entire data set.

Original: We used a constant learning rate of α for SAGA and SVRG, and α/ √ k for IG, where α = 10 − 3 for Covtype and Ijcnn, and 3 × 10 − 5 for SensIT.
Revised: We used a constant learning rate of α for SAGA and SVRG, and α/ √ k for IG, where α = 10 − 3 for Covtype and Ijcnn, and 3 × 10 − 5 for SensIT.

Original: It can be seen that subsets obtained by CRAIG achieve a similar loss and error rate as the entire data sets, but much faster.
Revised: It can be seen that subsets obtained by CRAIG achieve a similar loss and error rate as the entire data sets, but much faster.

Original: In particular, we obtained a speedup of 8.3x, 8.3x, 4.5x from applying IG, SVRG and SAGA on the subsets of size 10% obtained by CRAIG.
Revised: In particular, we obtained a speedup of 8.3x, 8.3x, 4.5x from applying IG, SVRG and SAGA on the subsets of size 10% obtained by CRAIG.

Original: Figure 2 top row, compares the speedup achieved by CRAIG to reach a similar loss residual as that of IG after 50 epochs for subsets of size 5% to 25%.
Revised: Figure 2 top row, compares the speedup achieved by CRAIG to reach a similar loss residual as that of IG after 50 epochs for subsets of size 5% to 25%.

Original: The bottom row compares the L2-norm of the distance to optimal solution (estimated by running IG for a long time) for IG applied to the subsets of various size obtained by CRAIG.
Revised: The bottom row compares the L2-norm of the distance to optimal solution (estimated by running IG for a long time) for IG applied to the subsets of various size obtained by CRAIG.

Original: We observe that while smaller subsets provide a larger speedup, IG on smaller subsets may converge to a point farther away from the optimal solution.
Revised: We observe that while smaller subsets provide a larger speedup, IG on smaller subsets may converge to a point farther away from the optimal solution.

Original: Finally, Figure 3 shows the loss residual vs time for IG when it processes the elements of the subsets according to the ordering obtained by CRAIG compared to random permutations of the same subsets.
Revised: Finally, Figure 3 shows the loss residual vs time for IG when it processes the elements of the subsets according to the ordering obtained by CRAIG compared to random permutations of the same subsets.

Original: We observe that the greedy ordering signiﬁcantly improves the rate of convergence of IG.
Revised: We observe that the greedy ordering signiﬁcantly improves the rate of convergence of IG.

Original: [Figure]
Revised: [Figure]

Original: 5. N ON - CONVEX E XPERIMENTS
Revised: 5. N ON - CONVEX E XPERIMENTS

Original: Our non-convex experiments involves applying CRAIG to train the following two neural networks:
Revised: Our non-convex experiments involves applying CRAIG to train the following two neural networks:

Original: (1) Our smaller network is a fully-connected hidden layer of 100 nodes and ten softmax output nodes; sigmoid activation and L2 regularization with λ = 0.0001 and mini-batches of size 10 on MNIST dataset of handwritten digits containing 60,000 training and 10,000 test images.
Revised: (1) Our smaller network is a fully-connected hidden layer of 100 nodes and ten softmax output nodes; sigmoid activation and L2 regularization with λ = 0.0001 and mini-batches of size 10 on MNIST dataset of handwritten digits containing 60,000 training and 10,000 test images.

Original: (2) Our large neural network is ResNet-56 for CIFAR10 with convolution, average pooling and dense layers with softmax outputs and L2 regularization with λ = 2 × 10 − 4 CIFAR 10 includes 50,000 training and 10,000 test images from 10 classes, and we used mini-batches of size 128.
Revised: (2) Our large neural network is ResNet-56 for CIFAR10 with convolution, average pooling and dense layers with softmax outputs and L2 regularization with λ = 2 × 10 − 4 CIFAR 10 includes 50,000 training and 10,000 test images from 10 classes, and we used mini-batches of size 128.

Original: Both MNIST and CIFAR10 data sets are normalized into [0, 1] by division with 255.
Revised: Both MNIST and CIFAR10 data sets are normalized into [0, 1] by division with 255.

Original: We apply CRAIG to several popular methods for training neural networks, including SGD, Nesterov
Revised: We apply CRAIG to several popular methods for training neural networks, including SGD, Nesterov

Original: Accelerated Gradient (NAG) method (Nesterov, 1983), Adagrad (Duchi et al., 2011), and Adaptive
Revised: Accelerated Gradient (NAG) method (Nesterov, 1983), Adagrad (Duchi et al., 2011), and Adaptive

Original: Moment Estimation (Adam) (Kingma & Ba, 2014).
Revised: Moment Estimation (Adam) (Kingma & Ba, 2014).

Original: Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations.
Revised: Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations.

Original: NAG adapts momentum updates to the slope of our error function and speed up SGD.
Revised: NAG adapts momentum updates to the slope of our error function and speed up SGD.

Original: Adagrad adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.
Revised: Adagrad adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.

Original: Adam computes individual adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients.
Revised: Adam computes individual adaptive learning rates for different parameters from estimates of ﬁrst and second moments of the gradients.

Original: Fig.
Revised: Fig.

Original: 4a shows loss, and error rate for training a 1-layer neural net on
Revised: 4a shows loss, and error rate for training a 1-layer neural net on

Original: MNIST.
Revised: MNIST.

Original: Fig.
Revised: Fig.

Original: 4b show similar quantities for training ResNet-56 on CIFAR10.
Revised: 4b show similar quantities for training ResNet-56 on CIFAR10.

Original: For both problems, we used a constant learning rate of 10 − 2 .
Revised: For both problems, we used a constant learning rate of 10 − 2 .

Original: Here, we apply CRAIG to select a subset of 30%-40% of the data at the beginning of every epoch and train only on the selected subset with the corresponding per-element stepsizes.
Revised: Here, we apply CRAIG to select a subset of 30%-40% of the data at the beginning of every epoch and train only on the selected subset with the corresponding per-element stepsizes.

Original: Interestingly, in addition to achieving a speedup of 2x to 3x for training neural networks, the subsets selected by CRAIG provide a better generalization performance compared to models trained on the entire data set.
Revised: Interestingly, in addition to achieving a speedup of 2x to 3x for training neural networks, the subsets selected by CRAIG provide a better generalization performance compared to models trained on the entire data set.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We developed a method, CRAIG, for selecting an ordered subset (coreset) of data points with their corresponding per-element stepsizes to speed up iterative gradient (IG) methods.
Revised: We developed a method, CRAIG, for selecting an ordered subset (coreset) of data points with their corresponding per-element stepsizes to speed up iterative gradient (IG) methods.

Original: In particular, we showed that weighted subsets that minimize the upper-bound on the estimation error of the full gradient, maximize a submodular facility location function.
Revised: In particular, we showed that weighted subsets that minimize the upper-bound on the estimation error of the full gradient, maximize a submodular facility location function.

Original: Hence, we can obtain an ordered subset of data points with their corresponding learning rates using a greedy algorithm.
Revised: Hence, we can obtain an ordered subset of data points with their corresponding learning rates using a greedy algorithm.

Original: We showed that
Revised: We showed that

Original: IG on subsets S returned by CRAIG converges at the same rate as IG on the entire data set V , while providing a | V | / | S | speedup.
Revised: IG on subsets S returned by CRAIG converges at the same rate as IG on the entire data set V , while providing a | V | / | S | speedup.

Original: In our set of experiments, we showed that various IG methods, including SAGA, SVRG, NAG, Adagrad and Adam runs up to 10x faster on convex and up to 3x on non-convex problems on subsets found by CRAIG while achieving practically the same trining loss and test error.
Revised: In our set of experiments, we showed that various IG methods, including SAGA, SVRG, NAG, Adagrad and Adam runs up to 10x faster on convex and up to 3x on non-convex problems on subsets found by CRAIG while achieving practically the same trining loss and test error.

Original: Finally, we empirically demonstrated the effect of the ordering found by the greedy algorithm on the convergence rate of IG methods.
Revised: Finally, we empirically demonstrated the effect of the ordering found by the greedy algorithm on the convergence rate of IG methods.

Original: A C ONVERGENCE R ATE A NALYSIS
Revised: A C ONVERGENCE R ATE A NALYSIS

Original: We ﬁrs proof the following Lemma which is an extension of the [Chung et al.
Revised: We ﬁrs proof the following Lemma which is an extension of the [Chung et al.

Original: , Lemma 4].
Revised: , Lemma 4].

Original: Lemma 3.
Revised: Lemma 3.

Original: Let u k ≥ 0 be a sequence of real numbers.
Revised: Let u k ≥ 0 be a sequence of real numbers.

Original: Assume there exist k 0 such that
Revised: Assume there exist k 0 such that

Original: [Equation]
Revised: [Equation]

Original: where e > 0 , d > 0 , c > 0 are given real numbers.
Revised: where e > 0 , d > 0 , c > 0 are given real numbers.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let c > p − 1 and v k =
Revised: Let c > p − 1 and v k =

Original: k
Revised: k

Original: p − 1 u k − d k ( c − p +1)
Revised: p − 1 u k − d k ( c − p +1)

Original: − e c − p +1 .
Revised: − e c − p +1 .

Original: Then, using Taylor approximation (1 + 1 k ) p
Revised: Then, using Taylor approximation (1 + 1 k ) p

Original: = (1 + pk ) + o ( 1 k ) we can write
Revised: = (1 + pk ) + o ( 1 k ) we can write

Original: [Equation]
Revised: [Equation]

Original: Note that for v k , we have
Revised: Note that for v k , we have

Original: [Equation]
Revised: [Equation]

Original: and (cid:16) d/
Revised: and (cid:16) d/

Original: [Equation]
Revised: [Equation]

Original: c − p
Revised: c − p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: →
Revised: →

Original: [Equation]
Revised: [Equation]

Original: Therefore, lim k →∞ v k ≤ 0 , and we get Eq.
Revised: Therefore, lim k →∞ v k ≤ 0 , and we get Eq.

Original: For p = 1 , we have u k ≤ ec .
Revised: For p = 1 , we have u k ≤ ec .

Original: Hence, u k converges into the region u ≤ ec , with ratio 1 − ck .
Revised: Hence, u k converges into the region u ≤ ec , with ratio 1 − ck .

Original: Moreover, for p − 1 ≥ c we have
Revised: Moreover, for p − 1 ≥ c we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for sufﬁciently large k .
Revised: for sufﬁciently large k .

Original: Summing over k , we obtain that v k is bounded for p − 1 > c (since the series (cid:80) ∞
Revised: Summing over k , we obtain that v k is bounded for p − 1 > c (since the series (cid:80) ∞

Original: k =1 (1 /k α ) converges for α > 1 ) and v k
Revised: k =1 (1 /k α ) converges for α > 1 ) and v k

Original: =
Revised: =

Original: O (log k ) for p =
Revised: O (log k ) for p =

Original: c + 1 (since (cid:80) ki
Revised: c + 1 (since (cid:80) ki

Original: =1 (1 /i ) =
Revised: =1 (1 /i ) =

Original: O (log k ) ).
Revised: O (log k ) ).

Original: In addition, based on [Chung et al.
Revised: In addition, based on [Chung et al.

Original: [Equation]
Revised: [Equation]

Original: Lemma 5] for u k ≥
Revised: Lemma 5] for u k ≥

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE R ATE FOR S TRONGLY C ONVEX F UNCTIONS P ROOF OF T HEOREM
Revised: A. C ONVERGENCE R ATE FOR S TRONGLY C ONVEX F UNCTIONS P ROOF OF T HEOREM

Original: We now provide the convergence rate for strongly convex functions building on the analysis of Nedi´c & Bertsekas (2001).
Revised: We now provide the convergence rate for strongly convex functions building on the analysis of Nedi´c & Bertsekas (2001).

Original: For non-smooth functions, gradients can be replaced by sub-gradients.
Revised: For non-smooth functions, gradients can be replaced by sub-gradients.

Original: Let x k = x k 0 .
Revised: Let x k = x k 0 .

Original: For every IG update on subset S we have
Revised: For every IG update on subset S we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Adding the above inequalities over elements of S we get
Revised: Adding the above inequalities over elements of S we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using strong convexity we can write
Revised: Using strong convexity we can write

Original: [Equation]
Revised: [Equation]

Original: Using Cauchy–Schwarz inequality
Revised: Using Cauchy–Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we know |
Revised: we know |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From reverse triangle inequality, and the facts that S is chosen in a way that (cid:107) (cid:80) i ∈ V ∇ f i ( x ∗ ) − (cid:80) j ∈ S γ j ∇ f j ( x ∗ ) (cid:107) ≤ (cid:15) , and that (cid:80) i ∈
Revised: From reverse triangle inequality, and the facts that S is chosen in a way that (cid:107) (cid:80) i ∈ V ∇ f i ( x ∗ ) − (cid:80) j ∈ S γ j ∇ f j ( x ∗ ) (cid:107) ≤ (cid:15) , and that (cid:80) i ∈

Original: V ∇ f i ( x ∗ ) = 

Revised: V ∇ f i ( x ∗ ) =

Original: we have (cid:107) (cid:80) j ∈ S γ j ∇ f j ( x ∗ ) (cid:107) ≤ (cid:107) (cid:80) i ∈
Revised: we have (cid:107) (cid:80) j ∈ S γ j ∇ f j ( x ∗ ) (cid:107) ≤ (cid:107) (cid:80) i ∈

Original: V ∇ f i ( x ∗ ) (cid:107) + (cid:15) = (cid:15) .
Revised: V ∇ f i ( x ∗ ) (cid:107) + (cid:15) = (cid:15) .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: For a continuously differentiable function, the following condition is implied by strong convexity condition
Revised: For a continuously differentiable function, the following condition is implied by strong convexity condition

Original: [Equation]
Revised: [Equation]

Original: Assuming gradients have a bounded norm max
Revised: Assuming gradients have a bounded norm max

Original: [Equation]
Revised: [Equation]

Original: and the fact that
Revised: and the fact that

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Thus for initial distance (cid:107) x 0 − x ∗ (cid:107) =
Revised: Thus for initial distance (cid:107) x 0 − x ∗ (cid:107) =

Original: d 0 , we have
Revised: d 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Putting Eq.
Revised: Putting Eq.

Original: 42 to Eq.
Revised: 42 to Eq.

Original: 46 together we get
Revised: 46 together we get

Original: [Equation]
Revised: [Equation]

Original: Now, from convexity of every f j for j ∈ S we have that
Revised: Now, from convexity of every f j for j ∈ S we have that

Original: [Equation]
Revised: [Equation]

Original: In addition
Revised: In addition

Original: [Equation]
Revised: [Equation]

Original: incremental updates gives us
Revised: incremental updates gives us

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we get 2 α k (cid:88)
Revised: we get 2 α k (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where γ max is the size of the largest cluster, and C is the upperbound on the gradients.
Revised: where γ max is the size of the largest cluster, and C is the upperbound on the gradients.

Original: For 0 < s ≤ 1 , the theorem follows by applying Lemma 3 to Eq.
Revised: For 0 < s ≤ 1 , the theorem follows by applying Lemma 3 to Eq.

Original: 52, with c = µ , e = 2 (cid:15)R/µ , and d = r 2 γ 2max C 2 .
Revised: 52, with c = µ , e = 2 (cid:15)R/µ , and d = r 2 γ 2max C 2 .

Original: For s = 0 , where we have a constant step size α k = α ≤ 1 µ , we get
Revised: For s = 0 , where we have a constant step size α k = α ≤ 1 µ , we get

Original: [Equation]
Revised: [Equation]

Original: Since (cid:80) kj =0 (1 − αµ ) j ≤ 1 αµ , we get
Revised: Since (cid:80) kj =0 (1 − αµ ) j ≤ 1 αµ , we get

Original: [Equation]
Revised: [Equation]

Original: (54) and therefore,
Revised: (54) and therefore,

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE S TRONGLY C ONVEX AND S MOOTH C OMPONENT F UNCTIONS P ROOF OF T HEOREM
Revised: A. C ONVERGENCE S TRONGLY C ONVEX AND S MOOTH C OMPONENT F UNCTIONS P ROOF OF T HEOREM

Original: IG updates for cycle k on subset S can be written as
Revised: IG updates for cycle k on subset S can be written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Building on the analysis of Gürbüzbalaban et al.
Revised: Building on the analysis of Gürbüzbalaban et al.

Original: (2015), for convex and twice continuously differentiable function, we can write
Revised: (2015), for convex and twice continuously differentiable function, we can write

Original: [Equation]
Revised: [Equation]

Original: where A rk = (cid:82) 1 0 ∇ 2 f ( x ∗ + τ ( x k − x ∗ )) dτ is average of the Hessian matrices corresponding to the r (weighted) elements of S on the interval [ x k , x ∗ ] .
Revised: where A rk = (cid:82) 1 0 ∇ 2 f ( x ∗ + τ ( x k − x ∗ )) dτ is average of the Hessian matrices corresponding to the r (weighted) elements of S on the interval [ x k , x ∗ ] .

Original: From Eq.
Revised: From Eq.

Original: 58 we have
Revised: 58 we have

Original: [Equation]
Revised: [Equation]

Original: where A k is average of the Hessian matrices corresponding to all the n component functions on the interval [ x k , x ∗ ] .
Revised: where A k is average of the Hessian matrices corresponding to all the n component functions on the interval [ x k , x ∗ ] .

Original: Taking norm of both sides and noting that (cid:80) i ∈
Revised: Taking norm of both sides and noting that (cid:80) i ∈

Original: V f i ( x ∗ ) = 0 and hence (cid:107)
Revised: V f i ( x ∗ ) = 0 and hence (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ∈ S
Revised: ∈ S

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ (cid:15) , we get
Revised: ≤ (cid:15) , we get

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is the estimation error of the full gradient by the weighted gradients of the elements of the subset S , and we used (cid:107) (cid:80) i ∈
Revised: where (cid:15) is the estimation error of the full gradient by the weighted gradients of the elements of the subset S , and we used (cid:107) (cid:80) i ∈

Original: V ∇ f i ( x k )
Revised: V ∇ f i ( x k )

Original: [Equation]
Revised: [Equation]

Original: Substituting Eq.
Revised: Substituting Eq.

Original: 58 into Eq.
Revised: 58 into Eq.

Original: 56 we obtain
Revised: 56 we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking norms of both sides, we get
Revised: Taking norms of both sides, we get

Original: [Equation]
Revised: [Equation]

Original: Now, we have
Revised: Now, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: 62, we obtain
Revised: 62, we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From strong convexity of (cid:80) i ∈
Revised: From strong convexity of (cid:80) i ∈

Original: V f i ( x ) , and gradient smoothness of each component f i ( x ) we have
Revised: V f i ( x ) , and gradient smoothness of each component f i ( x ) we have

Original: [Equation]
Revised: [Equation]

Original: where β = (cid:80) i ∈ V
Revised: where β = (cid:80) i ∈ V

Original: β i
Revised: β i

Original: In addition, from the gradient smoothness of the components we can write
Revised: In addition, from the gradient smoothness of the components we can write

Original: [Equation]
Revised: [Equation]

Original: where in the last line we used | S |
Revised: where in the last line we used | S |

Original: = r .
Revised: = r .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For 0 < s ≤ 1 , the theorem follows by applying Lemma 3 to Eq.
Revised: For 0 < s ≤ 1 , the theorem follows by applying Lemma 3 to Eq.

Original: 73 with c = µ , e = 2 (cid:15) , d = βCrγ 2max .
Revised: 73 with c = µ , e = 2 (cid:15) , d = βCrγ 2max .

Original: For s = 0 , where we have a constant step size α k = α ≤ 1 β , we get
Revised: For s = 0 , where we have a constant step size α k = α ≤ 1 β , we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the inequality in Eq.
Revised: where the inequality in Eq.

Original: 76 follows since (cid:80) ki =

Revised: 76 follows since (cid:80) ki =

Original: (1 − αµ ) i ≤ 1 αµ .
Revised: (1 − αµ ) i ≤ 1 αµ .

Original: B N ORM OF THE D IFFERENCE B ETWEEN G RADIENTS
Revised: B N ORM OF THE D IFFERENCE B ETWEEN G RADIENTS

Original: For ridge regression f i ( x ) = 12 ( (cid:104) a
Revised: For ridge regression f i ( x ) = 12 ( (cid:104) a

Original: i , x (cid:105) − y i ) 2 + λ 2 (cid:107) x (cid:107) 2 , we have ∇ f i ( x )
Revised: i , x (cid:105) − y i ) 2 + λ 2 (cid:107) x (cid:107) 2 , we have ∇ f i ( x )

Original: =
Revised: =

Original: a i ( (cid:104) a i , x (cid:105) − y i )
Revised: a i ( (cid:104) a i , x (cid:105) − y i )

Original: + λx .
Revised: + λx .

Original: Therefore, (cid:107)∇
Revised: Therefore, (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: For (cid:107) a i (cid:107) ≤ 1 , and y i =
Revised: For (cid:107) a i (cid:107) ≤ 1 , and y i =

Original: y j we get (cid:107)∇
Revised: y j we get (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ( (cid:107) x (cid:107) )
Revised: ( (cid:107) x (cid:107) )

Original: For reguralized logistic regression with y ∈ {− 1 , 1 } , we have ∇ f i ( x ) =
Revised: For reguralized logistic regression with y ∈ {− 1 , 1 } , we have ∇ f i ( x ) =

Original: y i / (1 + e y
Revised: y i / (1 + e y

Original: i (cid:104) a i ,x (cid:105) ) .
Revised: i (cid:104) a i ,x (cid:105) ) .

Original: For y i =
Revised: For y i =

Original: y j we get (cid:107)∇ f i ( x ) − ∇ f j ( x ) (cid:107) =
Revised: y j we get (cid:107)∇ f i ( x ) − ∇ f j ( x ) (cid:107) =

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) −

Revised: (cid:107) −

Original: + e
Revised: + e

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ,x (cid:105) (cid:107) a j (cid:107) .
Revised: ,x (cid:105) (cid:107) a j (cid:107) .

Original: For (cid:107) a i (cid:107) ≤ 1 , using Taylor approximation e x ≤ 1 + x , and noting that 1 1+
Revised: For (cid:107) a i (cid:107) ≤ 1 , using Taylor approximation e x ≤ 1 + x , and noting that 1 1+

Original: [Equation]
Revised: [Equation]

Original: −(cid:104)
Revised: −(cid:104)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:105) ≤ 1 we get (cid:107)∇ f
Revised: (cid:105) ≤ 1 we get (cid:107)∇ f

Original: [Equation]
Revised: [Equation]

Original: − ∇
Revised: − ∇

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: For classiﬁcation, we require y i =
Revised: For classiﬁcation, we require y i =

Original: y j , hence we can select subsets from each class and then merge the results.
Revised: y j , hence we can select subsets from each class and then merge the results.

Original: On the other hand, in ridge regression we also need | y i − y
Revised: On the other hand, in ridge regression we also need | y i − y

Original: j | to be small.
Revised: j | to be small.

Original: Similar results can be deduced for other loss functions including square loss, smoothed hinge loss, etc.
Revised: Similar results can be deduced for other loss functions including square loss, smoothed hinge loss, etc.

Original: Assuming (cid:107) x (cid:107) is bounded for all x ∈ X , upper-bounds on the euclidean distances between the gradients can be pre-computed.
Revised: Assuming (cid:107) x (cid:107) is bounded for all x ∈ X , upper-bounds on the euclidean distances between the gradients can be pre-computed.

Original: B. A DDITIONAL E XPERIMENTS
Revised: C A DDITIONAL E XPERIMENTS