Original: S CHEDULED L EARNING WITH D ECLINING D IVERSITY AND I NCREMENTAL D IFFICULTY
Revised: S CHEDULED L EARNING WITH D ECLINING D IVERSITY AND I NCREMENTAL D IFFICULTY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study how to adaptively select training subsets for different stages of iterative machine learning.
Revised: We study how to adaptively select training subsets for different stages of iterative machine learning.

Original: We introduce minimax curriculum learning (MCL), which trains a model on a diverse few samples at ﬁrst, and then later on a larger training set containing concentrated hard samples, thereby avoiding wasted efforts on redundant samples in early stages and on disperse outliers in later stages.
Revised: We introduce minimax curriculum learning (MCL), which trains a model on a diverse few samples at ﬁrst, and then later on a larger training set containing concentrated hard samples, thereby avoiding wasted efforts on redundant samples in early stages and on disperse outliers in later stages.

Original: At each stage, model weights and training sets are updated by solving a minimax optimization, whose objective is composed of a loss (reﬂecting the hardness of the training set) and a submodular regularization (measuring its diversity).
Revised: At each stage, model weights and training sets are updated by solving a minimax optimization, whose objective is composed of a loss (reﬂecting the hardness of the training set) and a submodular regularization (measuring its diversity).

Original: MCL repeatedly solves a sequence of such optimizations with decreasing diversity and increasing training set size.
Revised: MCL repeatedly solves a sequence of such optimizations with decreasing diversity and increasing training set size.

Original: Unlike the expensive alternative minimization used in previous work, we reduce MCL to minimization of a surrogate function that can be handled by submodular maximization and optimized by gradient methods.
Revised: Unlike the expensive alternative minimization used in previous work, we reduce MCL to minimization of a surrogate function that can be handled by submodular maximization and optimized by gradient methods.

Original: We show that MCL achieves better performance by using fewer labeled samples for both shallow and deep models.
Revised: We show that MCL achieves better performance by using fewer labeled samples for both shallow and deep models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inspired by the interaction between teacher and student in human education, recent studies (Khan et al., 2011b; Basu & Christensen, 2013; Spitkovsky et al., 2009) support that learning algorithms can be improved by updating a model on a designed sequence of training sets, i.e., a curriculum.
Revised: Inspired by the interaction between teacher and student in human education, recent studies (Khan et al., 2011b; Basu & Christensen, 2013; Spitkovsky et al., 2009) support that learning algorithms can be improved by updating a model on a designed sequence of training sets, i e., a curriculum.

Original: This problem is addressed in curriculum learning (CL) (Bengio et al., 2009), where the sequence is designed by a human expert or heuristic before training begins.
Revised: This problem is addressed in curriculum learning (CL) (Bengio et al., 2009) where the sequence is designed by a human expert or heuristic before training begins.

Original: Instead of relying on a teacher to provide the curriculum, self-paced learning (SPL) (Kumar et al., 2010; Tang et al., 2012a; Supancic III & Ramanan, 2013; Tang et al., 2012b) learns the curriculum during the training process, by letting the student (i.e., the algorithm) determine which samples to learn from based on their hardness.
Revised: Instead of relying on a teacher to provide the curriculum, self-paced learning (SPL) (Kumar et al., 2010; Tang et al., 2012a; Supancic III & Ramanan, 2013; Tang et al., 2012b) learns the curriculum during the training process, by letting the student (i e., the algorithm) determine which samples to learn from based on their hardness.

Original: Given a training set D = { ( x 1 , y 1 ) , . . .
Revised: Given a training set D = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } of n samples and loss function L ( y i , f ( x i , w )) , where x i ∈
Revised: , ( x n , y n ) } of n samples and loss function L ( y i , f ( x i , w )) , where x i ∈

Original: R m represents feature vector for the i th sample, y i is its label, and f ( x i , w ) is the predicted label provided by model with weight w , SPL (Kumar et al., 2010) aims at solving the following min-min optimization.
Revised: R m represents feature vector for the i th sample, y i is its label, and f ( x i , w ) is the predicted label provided by model with weight w , SPL (Kumar et al., 2010) aims at solving the following min-min optimization.

Original: [Equation]
Revised: [Equation]

Original: It jointly learns model weights w and sample weights v , which are 0 - 1 indicators of selected samples, by alternating minimization.
Revised: It jointly learns model weights w and sample weights v , which are 0 - 1 indicators of selected samples, by alternating minimization.

Original: Fixing w , minimization w.r.t.
Revised: Fixing w , minimization w.r.t.

Original: v equals selecting samples with loss
Revised: v equals selecting samples with loss

Original: L ( y i , f ( x i , w )) < λ , where λ controls the learning amount (“learning pace” in SPL) and works as a threshold to the hardness of enrolled samples.
Revised: L ( y i , f ( x i , w )) < λ , where λ controls the learning amount (“learning pace” in SPL) and works as a threshold to the hardness of enrolled samples.

Original: More samples will be selected with a larger λ .
Revised: More samples will be selected with a larger λ .

Original: Self-paced curriculum learning (Jiang et al., 2015) introduces a blending of “teacher mode” in CL and “student mode” in SPL, where the teacher can deﬁne a region of v by attaching a linear constraint a T v ≤ c to Eq.
Revised: Self-paced curriculum learning (Jiang et al., 2015) introduces a blending of “teacher mode” in CL and “student mode” in SPL, where the teacher can deﬁne a region of v by attaching a linear constraint a T v ≤ c to Eq.

Original: Evidence in previous work (Khan et al., 2011b; Tang et al., 2012b; Basu &
Revised: Evidence in previous work (Khan et al., 2011b; Tang et al., 2012b; Basu &

Original: Christensen, 2013; Bengio, 2014) shows that solving a series of Eq.
Revised: Christensen, 2013; Bengio, 2014) shows that solving a series of Eq.

Original: (1) with increasing λ can avoid bad local minima and reduce generalization error.
Revised: (1) with increasing λ can avoid bad local minima and reduce generalization error.

Original: Selection of training samples has also been studied in other learning settings, often with different motivations.
Revised: Selection of training samples has also been studied in other learning settings, often with different motivations.

Original: In active learning (AL) (Settles, 2010) and experimental design (Montgomery, 2006), the learner can actively query labels of samples from an unlabeled pool during the training process, and the goal is to reduce annotation costs.
Revised: In active learning (AL) (Settles, 2010) and experimental design (Montgomery, 2006), the learner can actively query labels of samples from an unlabeled pool during the training process, and the goal is to reduce annotation costs.

Original: Their curriculum design aims to achieve the same performance by using fewer labeled samples and ruling out redundant uninformative ones.
Revised: Their curriculum design aims to achieve the same performance by using fewer labeled samples and ruling out redundant uninformative ones.

Original: In machine teaching (Khan et al., 2011a; Zhu, 2015; Patil et al., 2014), the teacher designs the optimal training set so it can take the minimal efforts (e.g., the smallest number of samples) to guide the learning algorithm to ﬁnd a target model.
Revised: In machine teaching (Khan et al., 2011a; Zhu, 2015; Patil et al., 2014), the teacher designs the optimal training set so it can take the minimal efforts (e.g., the smallest number of samples) to guide the learning algorithm to ﬁnd a target model.

Original: In boosting (Schapire, 1990; Freund & Schapire, 1997), the goal is to learn an ensemble of weak classiﬁers sequentially.
Revised: In boosting (Schapire, 1990; Freund & Schapire, 1997), the goal is to learn an ensemble of weak classiﬁers sequentially.

Original: It assigns large weights to samples with large loss or that are misclassiﬁed by the model in previous steps.
Revised: It assigns large weights to samples with large loss or that are misclassiﬁed by the model in previous steps.

Original: Then the weighted error is minimized.
Revised: Then the weighted error is minimized.

Original: Since the weights are not 0 - 1 indicators, it needs labels of all samples.
Revised: Since the weights are not 0 - 1 indicators, it needs labels of all samples.

Original: Comparing to CL and SPL, the selection criteria of the above methods do not change for different learning stages.
Revised: Comparing to CL and SPL, the selection criteria of the above methods do not change for different learning stages.

Original: CL and SPL use a continuation scheme (Allgower & Georg, 2003), which handles a hard task by solving a sequence of tasks from easy to hard, where the solution to each task is the “warm start” for the next harder task.
Revised: CL and SPL use a continuation scheme (Allgower & Georg, 2003), which handles a hard task by solving a sequence of tasks from easy to hard, where the solution to each task is the “warm start” for the next harder task.

Original: In particular, they update a classiﬁer on a sequence of training sets from easy to hard.
Revised: In particular, they update a classiﬁer on a sequence of training sets from easy to hard.

Original: It has been declared in (Bengio et al., 2013; Bengio, 2014) that such continuation scheme can reduce the impact of local minima when applied to neural nets.
Revised: It has been declared in (Bengio et al., 2013; Bengio, 2014) that such continuation scheme can reduce the impact of local minima when applied to neural nets.

Original: For example, after each round of alternating minimization in SPL (Eq.
Revised: For example, after each round of alternating minimization in SPL (Eq.

Original: (1)), the threshold λ to loss L ( y i , f ( x i , w ))
Revised: (1)), the threshold λ to loss L ( y i , f ( x i , w ))

Original: (which measures the hardness) increases and enforces selection of more samples with larger loss.
Revised: (which measures the hardness) increases and enforces selection of more samples with larger loss.

Original: The accretion of hardness leads to an increased entropy of the training set, which can also be achieved by increasing diversity.
Revised: The accretion of hardness leads to an increased entropy of the training set, which can also be achieved by increasing diversity.

Original: SPL with diversity (SPLD) (Jiang et al., 2014) adds to Eq.
Revised: SPL with diversity (SPLD) (Jiang et al., 2014) adds to Eq.

Original: (1) a negative group sparse regularization −(cid:107) v (cid:107) 2 , 1 (cid:44) − (cid:80) bj =1 (cid:107) v ( j ) (cid:107) 2 (all samples are divided into b groups and v ( j ) is the weight vector for the j th group), which favours samples from different groups.
Revised: (1) a negative group sparse regularization −(cid:107) v (cid:107) 2 , 1 (cid:44) − (cid:80) bj =1 (cid:107) v ( j ) (cid:107) 2 (all samples are divided into b groups and v ( j ) is the weight vector for the j th group), which favours samples from different groups.

Original: Active learning and boosting, by contrast, always favor samples that are difﬁcult to predict, since they are the most informative to learn.
Revised: Active learning and boosting, by contrast, always favor samples that are difﬁcult to predict, since they are the most informative to learn.

Original: For example, uncertainty sampling (Culotta & McCallum, 2005; Scheffer et al., 2001; Dagan & Engelson, 1995; Dasgupta & Hsu, 2008) select samples that are most uncertain, while query by committee (Seung et al., 1992; Dagan & Engelson, 1995; Abe &
Revised: For example, uncertainty sampling (Culotta & McCallum, 2005; Scheffer et al., 2001; Dagan & Engelson, 1995; Dasgupta & Hsu, 2008) select samples that are most uncertain, while query by committee (Seung et al., 1992; Dagan & Engelson, 1995; Abe &

Original: Mamitsuka, 1998) selects the ones that multiple models most disagree on.
Revised: Mamitsuka, 1998) selects the ones that multiple models most disagree on.

Original: Boosting assigns large weights to misclassiﬁed samples with large loss.
Revised: Boosting assigns large weights to misclassiﬁed samples with large loss.

Original: Recently, diversity modeling was introduced to AL (Wei et al., 2015).
Revised: Recently, diversity modeling was introduced to AL (Wei et al., 2015).

Original: It uses submodular maximization to select diverse training batches from the most uncertain samples.
Revised: It uses submodular maximization to select diverse training batches from the most uncertain samples.

Original: However, changing diversity during the learning process has not been investigated.
Revised: However, changing diversity during the learning process has not been investigated.

Original: It is advantageous to gradually change both hardness and diversity of the training set over different learning stages.
Revised: It is advantageous to gradually change both hardness and diversity of the training set over different learning stages.

Original: But increasing both, as in SPLD, might not help to select the most informative samples.
Revised: But increasing both, as in SPLD, might not help to select the most informative samples.

Original: In early stages of SPLD (Jiang et al., 2014), selected samples tend to be grouped into a few easy regions due to small diversity.
Revised: In early stages of SPLD (Jiang et al., 2014), selected samples tend to be grouped into a few easy regions due to small diversity.

Original: However, since the prediction of the model is already precise in these regions, selecting more from them cannot yield much improvement (due to redundancy).
Revised: However, since the prediction of the model is already precise in these regions, selecting more from them cannot yield much improvement (due to redundancy).

Original: In its later stages, difﬁcult samples are more favored but large diversity enforces the selected ones to be dispersed over the input space.
Revised: In its later stages, difﬁcult samples are more favored but large diversity enforces the selected ones to be dispersed over the input space.

Original: However, hard samples usually gather around the margin of local decision boundaries where the prediction has high variance.
Revised: However, hard samples usually gather around the margin of local decision boundaries where the prediction has high variance.

Original: To effectively reduce the loss in these regions, we need more training samples from the same region.
Revised: To effectively reduce the loss in these regions, we need more training samples from the same region.

Original: Moreover, diverse hard samples might be outliers, especially in later stages when the model is already “mature.”
Revised: Moreover, diverse hard samples might be outliers, especially in later stages when the model is already “mature.”

Original: Selecting them into the curriculum might make the training unstable.
Revised: Selecting them into the curriculum might make the training unstable.

Original: Evidence for changing hardness and diversity in a curriculum can also be found in human education.
Revised: Evidence for changing hardness and diversity in a curriculum can also be found in human education.

Original: For example, courses in primary and middle school usually cover a broad range of basic knowledge/skills and fundamentals of many subjects, while in college and graduate school students mainly focus on advanced topics in their majors.
Revised: For example, courses in primary and middle school usually cover a broad range of basic knowledge/skills and fundamentals of many subjects, while in college and graduate school students mainly focus on advanced topics in their majors.

Original: In addition, recent studies of bilingualism (Bialystok et al., 2012; Li et al., 2014; Mechelli et al., 2004; Kov´acs & Mehler, 2009) show that learning two or multiple languages in childhood is beneﬁcial for future brain development.
Revised: In addition, recent studies of bilingualism (Bialystok et al., 2012; Li et al., 2014; Mechelli et al., 2004; Kov´acs & Mehler, 2009) show that learning two or multiple languages in childhood is beneﬁcial for future brain development.

Original: These indicate that large diversity can be helpful during early learning stages.
Revised: These indicate that large diversity can be helpful during early learning stages.

Original: 1. O UR A PPROACH
Revised: 1. O UR A PPROACH

Original: Motivated by these observations, we introduce a new form of curriculum learning called “minimax curriculum learning (MCL).”
Revised: Motivated by these observations, we introduce a new form of curriculum learning called “minimax curriculum learning (MCL).”

Original: It increases the hardness but reduces the diversity of the curriculum during training.
Revised: It increases the hardness but reduces the diversity of the curriculum during training.

Original: This is accomplished by solving a sequence of minimax optimizations of the following form with a fading weight λ and growing k .
Revised: This is accomplished by solving a sequence of minimax optimizations of the following form with a fading weight λ and growing k .

Original: [Equation]
Revised: [Equation]

Original: The objective is composed of the loss on a subset A of samples evaluating the hardness of A , and a normalized monotone non-decreasing submodular function F : 2 V → R + 1 of subset A measuring its diversity, where V is the ground set of all available samples.
Revised: The objective is composed of the loss on a subset A of samples evaluating the hardness of A , and a normalized monotone non-decreasing submodular function F : 2 V → R + 1 of subset A measuring its diversity, where V is the ground set of all available samples.

Original: Larger loss implies that the subset A is harder to learn, while a larger F ( A ) indicates more diversity.
Revised: Larger loss implies that the subset A is harder to learn, while a larger F ( A ) indicates more diversity.

Original: The weight λ controls the trade-off between diversity and hardness, while k is the “learning amount” controlling the size of A .
Revised: The weight λ controls the trade-off between diversity and hardness, while k is the “learning amount” controlling the size of A .

Original: In practice, we select k clusters rather than k samples to avoid the annotation costs and computation of loss on all samples (details are given in Section 2.3).
Revised: In practice, we select k clusters rather than k samples to avoid the annotation costs and computation of loss on all samples (details are given in Section 2.3).

Original: The submodular function F ( · ) can be chosen from a large family (e.g., including but not limited to facility location and set cover functions).
Revised: The submodular function F ( · ) can be chosen from a large family (e.g., including but not limited to facility location and set cover functions).

Original: All have the following diminishing returns property: given a ﬁnite ground set V , and any A ⊆ B ⊆ V and a v / ∈ B ,
Revised: All have the following diminishing returns property: given a ﬁnite ground set V , and any A ⊆ B ⊆ V and a v / ∈ B ,

Original: [Equation]
Revised: [Equation]

Original: This implies v is more important to the smaller set A than to the larger set B .
Revised: This implies v is more important to the smaller set A than to the larger set B .

Original: The marginal gain of v conditioned on A is f ( v | A ) (cid:44) f
Revised: The marginal gain of v conditioned on A is f ( v | A ) (cid:44) f

Original: ( v ∪ A )
Revised: ( v ∪ A )

Original: − f
Revised: − f

Original: ( A ) and reﬂects the importance of v to A .
Revised: ( A ) and reﬂects the importance of v to A .

Original: Due to this property, submodular functions (Fujishige, 2005) have been widely used for diversity models (Batra et al., 2012;
Revised: Due to this property, submodular functions (Fujishige, 2005) have been widely used for diversity models (Batra et al., 2012;

Original: Prasad et al., 2014; Gillenwater et al., 2012; Iyer & Bilmes, 2015).
Revised: Prasad et al., 2014; Gillenwater et al., 2012; Iyer & Bilmes, 2015).

Original: In MCL, we gradually reduce λ and augment k to increase the hardness and decrease the diversity of
Revised: In MCL, we gradually reduce λ and augment k to increase the hardness and decrease the diversity of

Original: A , and alternatively update A and w in each iteration.
Revised: A , and alternatively update A and w in each iteration.

Original: At early stages, λ is large and k is small, so the maximization tends to select a few diverse samples with large F ( A ) .
Revised: At early stages, λ is large and k is small, so the maximization tends to select a few diverse samples with large F ( A ) .

Original: At later stages, λ decreases and k increases, so the maximization prefers larger but less diverse training set composed of more concentrated hard samples.
Revised: At later stages, λ decreases and k increases, so the maximization prefers larger but less diverse training set composed of more concentrated hard samples.

Original: This scheduling of λ and k helps to select the most informative samples for different learning stages, and saves annotation/computation on redundant ones.
Revised: This scheduling of λ and k helps to select the most informative samples for different learning stages, and saves annotation/computation on redundant ones.

Original: At early stages, rather than wasting efforts on many samples the model is already accurate on, a few diverse and representative samples are sufﬁciently informative to improve the model.
Revised: At early stages, rather than wasting efforts on many samples the model is already accurate on, a few diverse and representative samples are sufﬁciently informative to improve the model.

Original: When the model becomes “mature” after some training, the most effective way to reduce loss is to train it on more samples from where it usually fails to predict, i.e., where hard samples are located.
Revised: When the model becomes “mature” after some training, the most effective way to reduce loss is to train it on more samples from where it usually fails to predict, i.e., where hard samples are located.

Original: The reduced diversity can rule out spurious hard samples, which are far away from the others and likely to be distractive outliers.
Revised: The reduced diversity can rule out spurious hard samples, which are far away from the others and likely to be distractive outliers.

Original: Although Eq.
Revised: Although Eq.

Original: (2) is a hybrid optimization involving both continuous variable w and discrete variable
Revised: (2) is a hybrid optimization involving both continuous variable w and discrete variable

Original: A , it can be reduced to minimization of a piecewise function, where each piece is deﬁned by a subset
Revised: A , it can be reduced to minimization of a piecewise function, where each piece is deﬁned by a subset

Original: A achieving the maximum in an interval of w .
Revised: A achieving the maximum in an interval of w .

Original: It is convex when the loss is convex, so various off-the-shelf algorithms can be applied once A is known for each piece.
Revised: It is convex when the loss is convex, so various off-the-shelf algorithms can be applied once A is known for each piece.

Original: However, the number of feasible A is O (2 n ) , and enumerating them all to ﬁnd the maximum is intractable.
Revised: However, the number of feasible A is O (2 n ) , and enumerating them all to ﬁnd the maximum is intractable.

Original: Thanks to submodularity of the objective, fast approximate algorithms (Nemhauser et al., 1978; Minoux, 1978;
Revised: Thanks to submodularity of the objective, fast approximate algorithms (Nemhauser et al., 1978; Minoux, 1978;

Original: Leskovec et al., 2007; Mirzasoleiman et al., 2015) exist to ﬁnd a sub-optimal A .
Revised: Leskovec et al., 2007; Mirzasoleiman et al., 2015) exist to ﬁnd a sub-optimal A .

Original: We instead minimize a surrogate of the piecewise function deﬁned by a sub-optimal A in each interval of w .
Revised: We instead minimize a surrogate of the piecewise function deﬁned by a sub-optimal A in each interval of w .

Original: It is unknown at ﬁrst but can be tracked by submodular maximization between gradient descent steps of w .
Revised: It is unknown at ﬁrst but can be tracked by submodular maximization between gradient descent steps of w .

Original:  M INIMAX C URRICULUM L EARNING
Revised:  M INIMAX C URRICULUM L EARNING

Original: The minimax problem in Eq.
Revised: The minimax problem in Eq.

Original: (2) can be explained as a two-person zero-sum game between a teacher (the maximizer) and a student (the minimizer): the teacher chooses training set A based on the student’s feedback of hardness (i.e., loss achieved by current model w ) and how diverse according to the teacher’s expertise, while the student updates w to reduce the loss on training set A given by the teacher.
Revised: (2) can be explained as a two-person zero-sum game between a teacher (the maximizer) and a student (the minimizer): the teacher chooses training set A based on the student’s feedback of hardness (i.e., loss achieved by current model w ) and how diverse according to the teacher’s expertise, while the student updates w to reduce the loss on training set A given by the teacher.

Original: Similar teacher-student interaction also exists in real life.
Revised: Similar teacher-student interaction also exists in real life.

Original: In addition, the teacher usually introduces basic concepts at the beginning and asks easy questions from diverse topics to get sufﬁcient feedback from the student, and then trains the student with more practices from the topics that the student ﬁnds difﬁcult.
Revised: In addition, the teacher usually introduces basic concepts at the beginning and asks easy questions from diverse topics to get sufﬁcient feedback from the student, and then trains the student with more practices from the topics that the student ﬁnds difﬁcult.

Original: This minimax formulation is essentially different from the min-min formulation often used in previous
Revised: This minimax formulation is essentially different from the min-min formulation often used in previous

Original: CL and SPL (Kumar et al., 2010; Jiang et al., 2014; 2015).
Revised: CL and SPL (Kumar et al., 2010; Jiang et al., 2014; 2015).

Original: Minimizing the worst case loss is a widely used strategy in machine learning (Lanckriet et al., 2003; Farnia & Tse, 2016) to achieve better generalization performance and model robustness, especially when strong assumptions cannot be made to the data distribution.
Revised: Minimizing the worst case loss is a widely used strategy in machine learning (Lanckriet et al., 2003; Farnia & Tse, 2016) to achieve better generalization performance and model robustness, especially when strong assumptions cannot be made to the data distribution.

Original: With submodular regularization, MCL can further avoid overﬁtting on a few outliers having large loss.
Revised: With submodular regularization, MCL can further avoid overﬁtting on a few outliers having large loss.

Original: Moreover, comparing to min-min formulation that alternately solves two minimizations per iteration (Alternative Convex Search (ACS)
Revised: Moreover, comparing to min-min formulation that alternately solves two minimizations per iteration (Alternative Convex Search (ACS)

Original: (Bazaraa et al., 1993)) until converge for each ﬁxed λ , only one minimization is required to solve Eq.
Revised: (Bazaraa et al., 1993)) until converge for each ﬁxed λ , only one minimization is required to solve Eq.

Original: Furthermore, SPL grows the training set up to the ground set to avoid overﬁtting on an easy subset.
Revised: Furthermore, SPL grows the training set up to the ground set to avoid overﬁtting on an easy subset.

Original: In MCL, however, the training set’s size also increases but can grow much slowly (since it is diverse), and may end with a much smaller subset.
Revised: In MCL, however, the training set’s size also increases but can grow much slowly (since it is diverse), and may end with a much smaller subset.

Original: In addition, introducing any diversity regularization of v to SPL usually leads to the loss of bi-convexity and requires re-design of the algorithm.
Revised: In addition, introducing any diversity regularization of v to SPL usually leads to the loss of bi-convexity and requires re-design of the algorithm.

Original: Any convexity and convergence analysis, however, always hold for any convex loss and submodular regularization in MCL.
Revised: Any convexity and convergence analysis, however, always hold for any convex loss and submodular regularization in MCL.

Original: The goal of this section is to solve the minimax problem in Eq.
Revised: The goal of this section is to solve the minimax problem in Eq.

Original: (2), which equals to a minimization min w ∈ R m g ( w ) of the following objective g ( w ) .
Revised: (2), which equals to a minimization min w ∈ R m g ( w ) of the following objective g ( w ) .

Original: [Equation]
Revised: [Equation]

Original: In each region of w , an analytic form of g ( w ) can be derived from A achieving the maximum by enumerating all possible subsets.
Revised: In each region of w , an analytic form of g ( w ) can be derived from A achieving the maximum by enumerating all possible subsets.

Original: Different regions of w are associated with different A .
Revised: Different regions of w are associated with different A .

Original: So g ( w ) is piecewise convex
Revised: So g ( w ) is piecewise convex

Original: if the loss function L
Revised: if the loss function L

Original: ( y i , f ( x i , w )) is convex w.r.t.
Revised: ( y i , f ( x i , w )) is convex w.r.t.

Original: w .
Revised: w .

Original: Unfortunately, the number of feasible subsets A is O (2 n ) so enumerating them all is intractable.
Revised: Unfortunately, the number of feasible subsets A is O (2 n ) so enumerating them all is intractable.

Original: However, there exists efﬁcient algorithms such as the greedy procedure and its variants (Nemhauser et al., 1978; Minoux, 1978) that can ﬁnd a suboptimal solution to the maximization in Eq.
Revised: However, there exists efﬁcient algorithms such as the greedy procedure and its variants (Nemhauser et al., 1978; Minoux, 1978) that can ﬁnd a suboptimal solution to the maximization in Eq.

Original: (4) (with w ﬁxed), which is
Revised: (4) (with w ﬁxed), which is

Original: [Equation]
Revised: [Equation]

Original: The suboptimality is due to the submodularity of G ( A ) , since G ( A ) is a weighted sum of a modular function (cid:80)
Revised: The suboptimality is due to the submodularity of G ( A ) , since G ( A ) is a weighted sum of a modular function (cid:80)

Original: ni ∈
Revised: ni ∈

Original: A L ( y i , f ( x i , w )) and a submodular function F ( A ) .
Revised: A L ( y i , f ( x i , w )) and a submodular function F ( A ) .

Original: , We assume L ( y i , f ( x i , w ))
Revised: , We assume L ( y i , f ( x i , w ))

Original: ≥w.l.o.g., so G ( A ) is also monotone non-decreasing.
Revised: ≥w.l.o.g., so G ( A ) is also monotone non-decreasing.

Original: For each region of w , the greedy procedure gives a sub-optimal solution ˆ A of Eq.
Revised: For each region of w , the greedy procedure gives a sub-optimal solution ˆ A of Eq.

Original: and deﬁnes a surrogate function ˆ g ( w ) of g ( w )
Revised: and deﬁnes a surrogate function ˆ g ( w ) of g ( w )

Original: [Equation]
Revised: [Equation]

Original: L ( y i [Equation] f ( [Equation]
Revised: that satisﬁes ˆ g ( w ) ∈

Original: + [Equation] (6) that satisﬁes ˆ g ( w ) ∈ [ α · g ( w ) , g ( w )] ( α is the approximation factor) in this region.
Revised: [ α · g ( w ) , g ( w )] ( α is the approximation factor) in this region.

Original: Similar to g ( w ) , ˆ g ( w ) is piecewise convex
Revised: Similar to g ( w ) , ˆ g ( w ) is piecewise convex

Original: if the loss function L
Revised: if the loss function L

Original: ( y i , f ( x i , w )) is convex w.r.t.
Revised: ( y i , f ( x i , w )) is convex w.r.t.

Original: w .
Revised: w .

Original: So different regions of w are associated with different ˆ A .
Revised: So different regions of w are associated with different ˆ A .

Original: We prove in Section 2.2 that minimizing ˆ g ( w ) gives an approximate solution of Eq.
Revised: We prove in Section 2.2 that minimizing ˆ g ( w ) gives an approximate solution of Eq.

Original: 1: input: π ( · , η ) , γ , p , ∆ 2: output: w3: initialize: w 0 , λ , k 4: while not converge do 5:for t ∈ { 0 , · · · , p } do 6:
Revised: do 5:

Original: G( A ) ← (cid:80) i ∈
Revised: G i ∈

Original: A L ( y i, f ( x i, w t ))
Revised: A L ( y i f ( x i w t ))

Original: + λF (A ) ;
Revised: + λF (

Original: 7: ˆ A ← S UBMODULAR M AX ( G, k ) ; 8: ∇ ˆ g ( wt ) =
Revised: ) ; 8: ∇ [Equation] [Equation] ; 9: w

Revised: [Equation]

Revised: ←

Original: ∂∂w (cid:80) i ∈
Revised: [Equation] [Equation] [Equation] ; 10: end

Original: ˆ A L ( y i , f ( x i , w t )) ; 9: w t +1 ← w t + π (cid:0) { w 1: t } , {∇ ˆ g ( w 1: t ) } , η (cid:1) ; 10: end for 11: w 0 ← w p , λ ← (1 − γ ) · λ , k ← k + ∆ ; 12: end while
Revised: 11: w [Equation] ← w p [Equation]

Revised: ←

Revised: [Equation]

Revised: ←

Revised: [Equation]

Original:  With ˆ g ( w ) given, our algorithm is simply gradient descent for minimizing ˆ g ( w ) , where many off-the-shelf methods can be invoked, e.g., SGD, momentum methods, Nesterov’s accelerated gradient (Nesterov, 2005), Adagrad (Duchi et al., 2011), etc.
Revised: + [Equation] ; 12: end while With ˆ g ( w ) given, our algorithm is simply gradient descent for minimizing ˆ g ( w ) , where many off-the-shelf methods can be invoked, e.g., SGD, momentum methods, Nesterov’s accelerated gradient (Nesterov, 2005), Adagrad (Duchi et al., 2011), etc.

Original: The key problem is how to obtain ˆ g ( w ) , which depends on suboptimal solutions in different regions of w .
Revised: The key problem is how to obtain ˆ g ( w ) , which depends on suboptimal solutions in different regions of w .

Original: However, it is not necessary to run submodular maximization for every region of w .
Revised: However, it is not necessary to run submodular maximization for every region of w .

Original: Since we use gradient descent methods, we only need to know ˆ g ( w ) for w on the optimization path.
Revised: Since we use gradient descent methods, we only need to know ˆ g ( w ) for w on the optimization path.

Original: At the begining of each iteration in our algorithm, we ﬁx w and use submodular maximization to achieve ˆ A , which deﬁnes ˆ g ( w ) in the current region of w , then any gradient descent method is applied to ˆ g ( w ) .
Revised: At the begining of each iteration in our algorithm, we ﬁx w and use submodular maximization to achieve ˆ A , which deﬁnes ˆ g ( w ) in the current region of w , then any gradient descent method is applied to ˆ g ( w ) .

Original: Let A ∗ represent the optimal solution of Eq.
Revised: Let A ∗ represent the optimal solution of Eq.

Original: (5), then ˆ A with approximation factor α satisﬁes
Revised: (5), then ˆ A with approximation factor α satisﬁes

Original: G ( ˆ A ) ≥ αG ( A ∗ ) .
Revised: G ( ˆ A ) ≥ αG ( A ∗ ) .

Original: The greedy algorithm produces an ˆ A with α = 1 − e − 1 (Nemhauser et al., 1978).
Revised: The greedy algorithm produces an ˆ A with α = 1 − e − 1 (Nemhauser et al., 1978).

Original: Given ˆ A , ˆ g ( w ) has gradient
Revised: Given ˆ A , ˆ g ( w ) has gradient

Original: [Equation]
Revised: [Equation]

Original: Then any gradient descent method can update w and thus minimize ˆ g ( w ) .
Revised: Then any gradient descent method can update w and thus minimize ˆ g ( w ) .

Original: We can treat ˆ A as one batch if k is small, and update w by w ← w − η ∇ ˆ g ( w ) with learning rate η .
Revised: We can treat ˆ A as one batch if k is small, and update w by w ← w − η ∇ ˆ g ( w ) with learning rate η .

Original: For large ˆ A , we use mini-batch SGD that applies the same update rule to every mini-batch of ˆ A .
Revised: For large ˆ A , we use mini-batch SGD that applies the same update rule to every mini-batch of ˆ A .

Original: More complex gradient descent rules π ( · , η ) can take all the historical gradients and w t in previous steps into account.
Revised: More complex gradient descent rules π ( · , η ) can take all the historical gradients and w t in previous steps into account.

Original: With superscript t to index the step,
Revised: With superscript t to index the step,

Original: [Equation]
Revised: [Equation]

Original: Hence, each iteration of the algorithm aims to minimize ˆ g ( w ) and update w based on a training set
Revised: Hence, each iteration of the algorithm aims to minimize ˆ g ( w ) and update w based on a training set

Original: ˆ A chosen by the submodular maximization.
Revised: ˆ A chosen by the submodular maximization.

Original: The whole algorithm (approximately) solves a sequence of
Revised: The whole algorithm (approximately) solves a sequence of

Original: Eq.
Revised: Eq.

Original: (2) with decreasing λ and augmenting k , where the solution w minimizing ˆ g ( w ) in one iteration is the “warm-start” for the next iteration.
Revised: (2) with decreasing λ and augmenting k , where the solution w minimizing ˆ g ( w ) in one iteration is the “warm-start” for the next iteration.

Original: This equals repeatedly updating the model w on a sequence of training sets ˆ A that changes from small (easy) and diverse to large set with clusters of hard samples.
Revised: This equals repeatedly updating the model w on a sequence of training sets ˆ A that changes from small (easy) and diverse to large set with clusters of hard samples.

Original: Algorithm 1 gives the details of MCL.
Revised: Algorithm 1 gives the details of MCL.

Original: Step 5-10 aims to solve optimization in Eq.
Revised: Step 5-10 aims to solve optimization in Eq.

Original: (2) with λ and k scheduled in Step 11.
Revised: (2) with λ and k scheduled in Step 11.

Original: Step 6-7 ﬁnds a suboptimal subset ˆ A by submodular maximization, which will be discussed in Section 2.1.
Revised: Step 6-7 ﬁnds a suboptimal subset ˆ A by submodular maximization, which will be discussed in Section 2.1.

Original: Step 8-9 updates w based on ˆ A by gradient descent π ( · , η ) with learning rate η .
Revised: Step 8-9 updates w based on ˆ A by gradient descent π ( · , η ) with learning rate η .

Original: We stop the optimization after p steps to avoid overﬁtting.
Revised: We stop the optimization after p steps to avoid overﬁtting.

Original: Then λ is reduced by factor γ ∈ [0 , 1] and k is increased by ∆ .
Revised: Then λ is reduced by factor γ ∈ [0 , 1] and k is increased by ∆ .

Original: We set p ≤ 50 due to the warm start in continuation scheme.
Revised: We set p ≤ 50 due to the warm start in continuation scheme.

Original: 2. S UBMODULAR M AXIMIZATION
Revised: 2. S UBMODULAR M AXIMIZATION

Original: We now introduce the submodular maximization algorithm used in Step 7 of Algorithm 1, whose goal is to maximize G ( A ) in Eq.
Revised: We now introduce the submodular maximization algorithm used in Step 7 of Algorithm 1, whose goal is to maximize G ( A ) in Eq.

Original: (5) and select training set ˆ A based on the hardness of samples to the current model and diversity.
Revised: (5) and select training set ˆ A based on the hardness of samples to the current model and diversity.

Original: Though the problem in Eq.
Revised: Though the problem in Eq.

Original: (5) is NP-hard, a near-optimal solution can be achieved by the greedy algorithm, which holds an approximation factor α = 1 − e − 1 (Nemhauser et al., 1978).
Revised: (5) is NP-hard, a near-optimal solution can be achieved by the greedy algorithm, which holds an approximation factor α = 1 − e − 1 (Nemhauser et al., 1978).

Original: It starts with A ← ∅ , and selects the next element with the largest marginal gain f ( v | A ) from V \ A , i.e., A ← A ∪ { v ∗ } where v ∗ ∈ argmax v ∈ V \
Revised: It starts with A ← ∅ , and selects the next element with the largest marginal gain f ( v | A ) from V \ A , i.e., A ← A ∪ { v ∗ } where v ∗ ∈ argmax v ∈ V \

Original: A f ( v | A ) , and this repeats until | A | = k .
Revised: A f ( v | A ) , and this repeats until | A | = k .

Original: It is simple to implement and usually outperforms other methods, e.g., those based on integer linear programming.
Revised: It is simple to implement and usually outperforms other methods, e.g., those based on integer linear programming.

Original: However, it requires O ( nk ) function evaluations for ground set size | V | = n .
Revised: However, it requires O ( nk ) function evaluations for ground set size | V | = n .

Original: The lazy, or accelerated, greedy algorithm (Minoux, 1978; Leskovec et al., 2007) reduces the number of function evaluations per step by lazily updating a priority queue of marginal gains over all elements.
Revised: The lazy, or accelerated, greedy algorithm (Minoux, 1978; Leskovec et al., 2007) reduces the number of function evaluations per step by lazily updating a priority queue of marginal gains over all elements.

Original: It has the same output and guarantee as the original greedy algorithm but signiﬁcantly reduces computation in practice.
Revised: It has the same output and guarantee as the original greedy algorithm but signiﬁcantly reduces computation in practice.

Original: Greedy and lazy greedy can guarantee a better approximation factor α better than 1 − e − 1 when the objective G ( A ) is close to modular, which is exactly the case for later stage of MCL when λ decreases to a small value.
Revised: Greedy and lazy greedy can guarantee a better approximation factor α better than 1 − e − 1 when the objective G ( A ) is close to modular, which is exactly the case for later stage of MCL when λ decreases to a small value.

Original: Speciﬁcally, the approximation factor is α =
Revised: Speciﬁcally, the approximation factor is α =

Original: (1 − e − κ G ) /κ
Revised: (1 − e − κ G ) /κ

Original: G (Conforti & Cornuejols, 1984), which depends on the curvature κ G ∈ [0 , 1] of G ( A ) (Fujishige, 2005) below describing how modular G ( A ) is.
Revised: G (Conforti & Cornuejols, 1984), which depends on the curvature κ G ∈ [0 , 1] of G ( A ) (Fujishige, 2005) below describing how modular G ( A ) is.

Original: [Equation]
Revised: [Equation]

Original: In the extreme case of κ G = 0 , G ( A ) is modular.
Revised: In the extreme case of κ G = 0 , G ( A ) is modular.

Original: As κ G increases towards 1 , G ( A ) becomes more submodular, and the approximation factor α reduces to 1 − e − 1 .
Revised: As κ G increases towards 1 , G ( A ) becomes more submodular, and the approximation factor α reduces to 1 − e − 1 .

Original: In MCL, κ G decreases with the weight λ of submodular regularization F ( A ) .
Revised: In MCL, κ G decreases with the weight λ of submodular regularization F ( A ) .

Original: This increases the approximation factor α and results in the surrogate function ˆ g ( w ) being closer to the true objective g ( w ) .
Revised: This increases the approximation factor α and results in the surrogate function ˆ g ( w ) being closer to the true objective g ( w ) .

Original: 2. C ONVERGENCE A NALYSIS
Revised: 2. C ONVERGENCE A NALYSIS

Original: From the optimization perspective, Algorithm 1 uses a continuation scheme to solve a sequence of minimax problems in Eq.
Revised: From the optimization perspective, Algorithm 1 uses a continuation scheme to solve a sequence of minimax problems in Eq.

Original: (2) with decreasing λ and increasing k .
Revised: (2) with decreasing λ and increasing k .

Original: Each problem is solved by
Revised: Each problem is solved by

Original: Step 5-10, and conveys its solution w p as a warm start w 0 to the next problem.
Revised: Step 5-10, and conveys its solution w p as a warm start w 0 to the next problem.

Original: Instead of directly minimizing the true objective g ( w ) in Eq.
Revised: Instead of directly minimizing the true objective g ( w ) in Eq.

Original: (4) that could be NP-hard to obtain, Step 5-10 minimizes a surrogate function ˆ g ( w ) ≤ g ( w ) by using gradient descent rule π ( · , η ) .
Revised: (4) that could be NP-hard to obtain, Step 5-10 minimizes a surrogate function ˆ g ( w ) ≤ g ( w ) by using gradient descent rule π ( · , η ) .

Original: It is interesting to study how close the solution ˆ w of applying gradient descent to ˆ g ( w ) until convergence approximates the real solution w ∗ of min w ∈ R m g ( w ) .
Revised: It is interesting to study how close the solution ˆ w of applying gradient descent to ˆ g ( w ) until convergence approximates the real solution w ∗ of min w ∈ R m g ( w ) .

Original: In the following, we study whether running the inner loop in
Revised: In the following, we study whether running the inner loop in

Original: Step 5-10 until convergence can converge to the global solution of the minimax problem proposed in
Revised: Step 5-10 until convergence can converge to the global solution of the minimax problem proposed in

Original: Eq.
Revised: Eq.

Original: Proposition 1.
Revised: Proposition 1.

Original: The maximum of multiple β -strongly convex functions is β -strongly convex as well.
Revised: The maximum of multiple β -strongly convex functions is β -strongly convex as well.

Original: The proof can be found in Appendix 4.1.
Revised: The proof can be found in Appendix 4.1.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Inner-loop convergence)
Revised: (Inner-loop convergence)

Original: For minimax problem in Eq.
Revised: For minimax problem in Eq.

Original: (2) with given ground set of all samples V and λ , if the loss function L ( y i , f
Revised: (2) with given ground set of all samples V and λ , if the loss function L ( y i , f

Original: ( x i , w )) is β -strongly convex and | V | ≥ k , running
Revised: ( x i , w )) is β -strongly convex and | V | ≥ k , running

Original: Step 6-9 in Algorithm 1 for iterations until convergence yields a solution ˆ w satisfying
Revised: Step 6-9 in Algorithm 1 for iterations until convergence yields a solution ˆ w satisfying

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where w ∗ is the optimal solution of the minimax problem in Eq.
Revised: where w ∗ is the optimal solution of the minimax problem in Eq.

Original: (2) , g ( w ∗ ) is the objective value achieved on w ∗ , and α is the approximation factor that submodular maximization can guarantee for
Revised: (2) , g ( w ∗ ) is the objective value achieved on w ∗ , and α is the approximation factor that submodular maximization can guarantee for

Original: G ( A ) .
Revised: G ( A ) .

Original: Proof.

Original: The objective g ( w ) of the minimax problem in Eq.

Original: (2) after eliminating A is given in Eq.

Original: Since G ( A ) in Eq.

Original: (5) is monotone non-decreasing submodular, the optimal subset A when deﬁning g ( w ) in Eq.

Original: (4) always has size k if | V | ≥ k .

Original: In addition, because the loss function L ( y i , f ( x i , w )) is β -strongly convex, g ( w ) in Eq.

Original: (4) is the maximum over multiple kβ -strongly convex functions with different A .

Original: According to Proposition 1, g ( w ) is also kβ -strongly convex, i.e.,

Original: [Equation]

Original: Since the convex function g ( w ) achieves minimum on w ∗ , it is valid to substitute ∇ g ( w ∗ ) = 0 ∈ ∂g ( w ∗ ) into Eq.

Original: After rearrangement, we have

Original: [Equation]

Original: In the following, we will prove g ( w ∗ ) ≥ α · g ( ˆ w ) , which together with Eq.

Original: (12) will lead to the ﬁnal bound showing how close ˆ w is to w ∗ .

Original: Note ˆ g ( w ) (Eq.

Original: (6)) is a piecewise function, each piece of whom is convex and associated with different ˆ A achieved by a submodular maximization algorithm of approximation factor α .

Original: Since ˆ A is not guaranteed to be a global maxima, unlike g ( w ) , the whole ˆ g ( w ) cannot be written as the maximum of multiple convex functions and thus can be non-convex.

Original: Therefore, gradient descent in

Original: Step 6-9 of Algorithm 1 can lead to either 1) ˆ w is a global minima of ˆ g ( w ) ; or 2) ˆ w is a local minima of ˆ g ( w ) .

Original: Saddle points and local maxima do not exist on ˆ g ( w ) because each piece of it is convex.

Original: 1) When ˆ w is a global minima of ˆ g ( w ) , we have

Original: [Equation]

Original: The ﬁrst inequality is due to g ( · ) ≥ ˆ g ( · ) .

Original: The second inequality is due to the global optimality of ˆ w .

Original: The third inequality is due to the approximation bound ˆ g ( · ) ≥ α · g ( · ) guaranteed by the submodular maximization in Step 7 of Algorithm 1.

Original: 2) When ˆ w is a local minima of ˆ g ( w ) , we have ∇ ˆ g ( ˆ w ) = 0 .

Original: Let h ( w ) to be the piece of ˆ g ( w ) where ˆ w is located, then ˆ w has to be a global minima of h ( w ) due to the convexity of h ( w ) .

Original: Let A denote the ground set of ˆ A on all pieces of ˆ g ( w ) , we deﬁne an auxiliary convex function ˜ g ( w ) as

Original: [Equation]

Original: It is convex because it is deﬁned as the maximum of multiple convex function.

Original: So we have

Original: [Equation] The ﬁrst inequality is due to the deﬁnition of A , and the second inequality is a result of A ⊆ V by comparing g ( w ) in Eq. (4) with ˜ g ( w ) in Eq.
Revised: The proof can be found in Appendix 4.2.

Original: Let ˜ w denote a global minima of ˜ g ( w ) , we have

Original: [Equation]

Original: The ﬁrst inequality is due to Eq.

Original: (15), the second inequality is due to the global optimality of ˜ w on ˜ g

Original: ( w ) , the third inequality is due to the deﬁnition of ˜ g

Original: ( w ) in Eq.

Original: (14) ( ˜ g ( w ) is the maximum of all pieces of ˆ g ( w ) and h ( w ) is one piece of them), the fourth inequality is due to the global optimaliy of ˆ w on h ( w ) , the last inequality is due to the approximation bound ˆ g ( · ) ≥ α · g ( · ) guaranteed by the submodular maximization in Step 7 of Algorithm 1.

Original: Therefore, in both cases we have g ( w ∗ ) ≥ α · g ( ˆ w ) .

Original: Applying it to Eq.

Original: (12) results in [Equation] [Equation] In Theorem 1, we analyze the upper bound for (cid:107) ˆ w − w ∗ (cid:107) 22 based on two assumptions: 1) the loss
Revised: In Theorem 1, we analyze the upper bound for (cid:107) ˆ w − w ∗ (cid:107) 22 based on two assumptions: 1) the loss

Original: L ( y i , f ( x i , w )) being β -strongly convex w.r.t.
Revised: L ( y i , f ( x i , w )) being β -strongly convex w.r.t.

Original: w ; and 2) ˆ w is achieved by running gradient descent in Step 6-9 until convergence.
Revised: w ; and 2) ˆ w is achieved by running gradient descent in Step 6-9 until convergence.

Original: In case the loss L ( y i , f ( x i , w )) is convex but not β -strongly convex, a commonly used trick to modify it to β -strongly convex is to add an (cid:96) 2 regularization ( β/ 2) (cid:107) w (cid:107) 22 .
Revised: In case the loss L ( y i , f ( x i , w )) is convex but not β -strongly convex, a commonly used trick to modify it to β -strongly convex is to add an (cid:96) 2 regularization ( β/ 2) (cid:107) w (cid:107) 22 .

Original: In addition, for non-convex L ( y i , f ( x i , w )) , it is possible to prove that with high probability, a noise perturbed SGD on ˆ g ( w ) can hit an (cid:15) -optimal local solution of g ( w ) in polynomial time steps.
Revised: In addition, for non-convex L ( y i , f ( x i , w )) , it is possible to prove that with high probability, a noise perturbed SGD on ˆ g ( w ) can hit an (cid:15) -optimal local solution of g ( w ) in polynomial time steps.

Original: We will leave this to our future works.
Revised: We will leave this to our future works.

Original: In our empirical study (Section 3), MCL achieves compelling performance when applied to deep neural nets when loss is usually non-convex.
Revised: In our empirical study (Section 3), MCL achieves compelling performance when applied to deep neural nets when loss is usually non-convex.

Original: In Step 5-10 of Algorithm 1, we stop gradient descent after p steps rather than waiting for convergence as in Assumption 2).
Revised: In Step 5-10 of Algorithm 1, we stop gradient descent after p steps rather than waiting for convergence as in Assumption 2).

Original: This is because in the continuation scheme, w p is sufﬁciently good as an 
Revised: This is because in the continuation scheme, w p is sufﬁciently good as an initialization for the next iteration, and ˆ g ( w ) is small enough after p steps due to the warm start from its previous iteration. We will remove this assumption in later analysis of overall convergence (i.e., Lemma 1 and Theorem 3).

Original: In later stages of MCL when λ is small, G ( A ) tends to be more modular, i.e., with small curvature κ G .
Revised: In later stages of MCL when λ is small, G ( A ) tends to be more modular, i.e., with small curvature κ G .

Original: As discussed in Section 2.1, when κ G is close to 0 , lazy greedy can have approximation factor of α
Revised: As discussed in Section 2.1, when κ G is close to 0 , lazy greedy can have approximation factor of α

Original: = (1 − e − κ G ) /κ
Revised: = (1 − e − κ G ) /κ

Original: G , larger than 1 − e − 1 and potentially close to 1 .
Revised: G , larger than 1 − e − 1 and potentially close to 1 .

Original: With g ( w ∗ ) upper bounded, the bound in Eq.
Revised: With g ( w ∗ ) upper bounded, the bound in Eq.

Original: (10) can be nearly 0 .
Revised: (10) can be nearly 0 .

Original: Hence, ˆ w obtained by our algorithm is sufﬁciently close to w ∗ .
Revised: Hence, ˆ w obtained by our algorithm is sufﬁciently close to w ∗ .

Original: Proposition 2.

Original: If x ∈ [0 , 1] , the following inequality holds true.

Original: x [Equation] The proof can be found in Appendix 4.2. In the following, we further study how the distance between the solution ˆ w T achieved by Algorithm 1 and the global optimal solution w ∗ T of the minimax problem in Eq.(2) changes with the number of iterations T in the outer-loop.
Revised: In the following, we further study how the distance between the solution ˆ w T achieved by Algorithm 1 and the global optimal solution w ∗ T of the minimax problem in Eq.(2) changes with the number of iterations T in the outer-loop.

Original: We will show that different scheduling strategy for λ and k will lead to different convergence rate.
Revised: We will show that different scheduling strategy for λ and k will lead to different convergence rate.

Original: This is essentially important to understand how the design of curriculum can improve the model performance.
Revised: This is essentially important to understand how the design of curriculum can improve the model performance.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Outer-loop convergence)
Revised: (Outer-loop convergence)

Original: If the loss function L ( y i , f ( x i , w )) is β -strongly convex, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if each inner-loop in Algorithm 1 runs Step 6-9 until convergence, then solution ˆ w T at the end of the
Revised: If the loss function L ( y i , f ( x i , w )) is β -strongly convex, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if each inner-loop in Algorithm 1 runs Step 6-9 until convergence, then solution ˆ w T at the end of the

Original: T th iteration of outer-loop fulﬁlls
Revised: T th iteration of outer-loop fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where w ∗ T is the optimal solution of the minimax problem in Eq.
Revised: where w ∗ T is the optimal solution of the minimax problem in Eq.

Original: (2) with λ used in the T th iteration of outer-loop.
Revised: (2) with λ used in the T th iteration of outer-loop.

Original: If k starts from k 0 and linearly increases by k ← k +∆ (as in Step 11 of Algorithm 1),
Revised: If k starts from k 0 and linearly increases by k ← k +∆ (as in Step 11 of Algorithm 1),

Original: [Equation]
Revised: [Equation]

Original: Otherwise, if k increases exponentially, i.e., k ←(1 + ∆) · k ,
Revised: Otherwise, if k increases exponentially, i.e., k ←

Original: [Equation]
Revised: [Equation]

Original: In above results, κ F is the curvature of submodular function F ( · ) , λ 0 and k 0 are the initial values for λ and k respectively, c 1 = min j ∈ V,t
Revised: In above results, κ F is the curvature of submodular function F ( · ) , λ 0 and k 0 are the initial values for λ and k respectively, c 1 = min j ∈ V,t

Original: [ L ( y i , f ( x i , ˆ w tT )) /F
Revised: [ L ( y i , f ( x i , ˆ w tT )) /F

Original: ( j )] , c 2 = max A ⊆ V, | A |≤ k F ( A ) , and [Equation] Proof.
Revised: ( j )] , c 2 = max A ⊆ V, | A |≤ k F ( A ) , and g ( w ∗∞ ) =

Original: Applying the inequality in Proposition 2 and the approximation factor of lazy greedy

Original: α = (1 − e − κ G ) /κ G to the right hand side of Eq. (10) from Theorem 1 yields [Equation]
Revised: min w ∈ R m max A ⊆ V, | A |≤ k (cid:80)

Original: where κ G is the curvature of submodular function G ( · ) deﬁned in Eq.

Original: According to the deﬁnition of curvature given in Eq.

Original: (9), and let κ F denotes the curvature of submodular function F ( · ) in Eq.

Original: (5), we have

Original: [Equation]

Original: where we use L ( j, t ) as a simpler representation of per-sample loss L ( y j , f ( x i , ˆ w t )) if not causing any confusion, and the last inequality is due to the deﬁnition of curvature κ F for submodular function

Original: F ( · ) . Substituting the above inequality about κ G into Eq. (22) results in [Equation]
Revised: i ∈

Original: We use subscript as the index for iterations in the outer-loop, e.g., ˆ w T is the model weights w after the T

Original: th iteration of outer-loop.

Original: If we decrease λ exponentially from λ = λ 0 and increase k linearly from k = k 0 , as Step 11 in Algorithm 1, we have

Original: [Equation]

Original: According to the deﬁnition of g ( · ) in Eq.

Original: (4), for g (  w ∗ T ) we have [Equation] where g
Revised: A L ( y i , f ( x i , w )) .

Original: [Equation]

Original: Substituting Eq.

Original: (26) to Eq.

Original: (25) yields

Original: [Equation] If we can tolerate more expensive computational cost for running submodular maximization with larger budget k , and increase k exponentially, i.e., k ← (1 + ∆) · k , we have [Equation] This completes the proof.
Revised: The proof can be found in Appendix 4.4.

Original: Remarks: Theorem 2 gives a upper bound proportional to λ/k .
Revised: Remarks: Theorem 2 gives a upper bound proportional to λ/k .

Original: Therefore, either increasing k exponentially or decreasing λ exponentially can result in a linear convergence rate.
Revised: Therefore, either increasing k exponentially or decreasing λ exponentially can result in a linear convergence rate.

Original: However, submodular maximization in Step 7 usually has an expensive time cost when k is large.
Revised: However, submodular maximization in Step 7 usually has an expensive time cost when k is large.

Original: Hence, in
Revised: Hence, in

Original: Algorithm 1, we choose to decrease λ exponentially but increase k linearly.
Revised: Algorithm 1, we choose to decrease λ exponentially but increase k linearly.

Original: Another interesting result revealed by Theorem 2 is that the constant factor is related to both the curvature κ F of submodular term and the strongly-convex constant β of the loss term.
Revised: Another interesting result revealed by Theorem 2 is that the constant factor is related to both the curvature κ F of submodular term and the strongly-convex constant β of the loss term.

Original: It also depends on c 1 , the minimal ratio between  loss and singular gain of a single sample.
Revised: It also depends on c 1 , the minimal ratio between per-sample loss and singular gain among all samples.

Original: We will see that the quantities κ F /β and cfrequently appear in our later analysis.
Revised: We will see that the quantities κ F /β and c 1 frequently appear in our later analysis.

Original: In addition, this quantity is potentially useful to analysis of more general convex-submodular hybrid optimization problems.
Revised: In addition, these two quantities are potentially interesting to analysis of more general convex-submodular hybrid optimization problems.

Original: Although we have the convergence analysis for both the inner-loop and the outer-loop of Algorithm 1, we still need an overall convergence result for the whole algorithm, without the assumption that running the inner-loop until convergence.
Revised: Although we have the convergence analysis for both the inner-loop and the outer-loop of Algorithm 1, we still need an overall convergence result for the whole algorithm, without the assumption that running the inner-loop until convergence.

Original: In particular, by indexing the itereations of inner-loop with superscript and the itereations of outer-loop with subscript, our goal is to investigate how close a solution ˆ w pT achieved by Algorithm 1 to a global solution w ∗ T of the minimax problem Eq.(1) for a sufﬁciently large T ≥ T , and how the distance between them changes with p and T under different scheduling for λ and k .
Revised: In particular, by indexing the itereations of inner-loop with superscript and the itereations of outer-loop with subscript, our goal is to investigate how close a solution ˆ w pT achieved by Algorithm 1 to a global solution w ∗T of the minimax problem Eq.(1) for a sufﬁciently large T ≥ T , and how the distance between them changes with p and T under different scheduling for λ and k .

Original: A larger T usually indicates a better performance but is also a harder target.
Revised: A larger T usually indicates a better performance but is also a harder target.

Original: Proposition 3.

Original: If x ∈ [0 , 1] , the following inequality holds true.

Original: [Equation]

Original: The proof can be found in Appendix 4.3.

Original: Lemma 1.
Revised: Lemma 1.

Original: If the loss function L ( y i , f ( x i , w )) is β -strongly convex and L -smooth, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if Step 9 in
Revised: If the loss function L ( y i , f ( x i , w )) is β -strongly convex and L -smooth, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if Step 9 in

Original: Algorithm 1 uses normal gradient descent algorithm with learning rate η = 1 / ( kL ) , running the inner-loop (Step 6-9) for p iterations yields a solution ˆ w p satisfying
Revised: Algorithm 1 uses normal gradient descent algorithm with learning rate η = 1 / ( kL ) , running the inner-loop (Step 6-9) for p iterations yields a solution ˆ w p satisfying

Original: [Equation]
Revised: [Equation]

Original:  where ˆ w is the solution after running the inner-loop (Step 6-9) until convergence.
Revised: (14) where ˆ w is the solution after running the inner-loop (Step 6-9) until convergence.

Original: The above result holds for any iteration of outer-loop.
Revised: The above result holds for any iteration of outer-loop.

Original: Proof.

Original: On the optimization path from ˆ w p to ˆ w , ˆ g ( · ) may switch to different convex pieces.

Original: Hence, we use h t ( · ) to denote the piece of ˆ g ( · ) on which ˆ w t is located, so ˆ g ( ˆ w t )

Original: =

Original: h t ( ˆ w t ) .

Original: Because h t ( · ) is kβ -strongly convex,

Original: [Equation]

Original: Recall the auxiliary convex function ˜ g ( · ) deﬁned in Eq.

Original: [Equation]

Original: by using the inequality shown in

Original: Eq.

Original: [Equation]

Original: we have

Original: [Equation]

Original: where the third inequality is due to the approximation factor α

Original: [Equation]

Original: and the fourth inequality is due to the kL -smoothness of ˆ g ( · ) .

Original: Eq.

Original: (33) results in

Original: [Equation]

Original: Let w =

Original: ˆ w t and substitute Eq.

Original: (34) to Eq.

Original: [Equation]

Original: due to ˆ g ( ˆ w t )

Original: =

Original: h t ( ˆ w t )

Original: [Equation]

Original: we have

Original: [Equation]

Original: The last step is achieved by setting learning rate η = 1 / ( kL ) .

Original: Recursively applying the above inequality from t =

Original: p − 1 to t = 1 ( p is the number of iterations for inner loop) yields

Original: [Equation]

Original: Since α = 1 − e − κG κ G , Proposition 3 leads to

Original: [Equation]

Original: where the last inequality comes from Eq.

Original: In addition

Original: [Equation]

Original: due to inequality (cid:0) 1 + xn (cid:1) n ≥ 1 + x for n > 1 and | x | ≤ n , we have

Original: [Equation]

Original: Applying Eq.

Original: (37) and Eq.

Original: (38) to Eq. (36) results in [Equation] This completes the proof.
Revised: The proof can be found in Appendix 4.6.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Overall convergence)
Revised: (Overall convergence)

Original: If the loss function L ( y i , f ( x i , w )) is β -strongly convex and
Revised: If the loss function L ( y i , f ( x i , w )) is β -strongly convex and

Original: L -smooth, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if Step 9 in Algorithm 1 uses normal gradient descent algorithm with learning rate η = 1 / ( kL ) , running the outer-loop for T iterations and each inner-loop (Step 6-9) for p iterations yields a solution ˆ w pT , whose distance to the optimal solution w ∗ T of Eq.
Revised: L -smooth, submodular function F ( · ) has curvature κ F , the ground set V of all samples has size | V | ≥ k , and if Step 9 in Algorithm 1 uses normal gradient descent algorithm with learning rate η = 1 / ( kL ) , running the outer-loop for T iterations and each inner-loop (Step 6-9) for p iterations yields a solution ˆ w pT , whose distance to the optimal solution w ∗T of Eq.

Original: (2) (with λ = λ 0 (1 − γ ) T ) for T ≥ T fulﬁlls
Revised: (2) (with λ = λ 0 (1 − γ ) T ) for T ≥ T fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: For λ starting from λ 0 and exponentially decreasing by λ ← (1 − γ ) · λ (as in Step 11 of Algorithm 1), c is deﬁned as
Revised: For λ starting from λ 0 and exponentially decreasing by λ ← (1 − γ ) · λ (as in Step 11 of Algorithm 1), c is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 and g ( w ∗∞ ) are deﬁned in Theorem 2.
Revised: where c 1 , c 2 and g ( w ∗∞ ) are deﬁned in Theorem 2.

Original: If k starts from k 0 and linearly increases by k ← k + ∆ (as in Step 11 of Algorithm 1),
Revised: If k starts from k 0 and linearly increases by k ← k + ∆ (as in Step 11 of Algorithm 1),

Original: [Equation]
Revised: [Equation]

Original: Otherwise, if k increases exponentially, i [Equation] e., k ← (1 + ∆) · k [Equation]
Revised: Otherwise, if k increases exponentially, i.e., k ← (1 + ∆) · k ,

Original: Proof.

Original: Triangle inequality results in

Original: [Equation]

Original: [Equation]

Original: In the right hand side of above inequality, the ﬁrst term can be upper bounded by the result in Lemma 1, while the second term can be upper bounded by the results in Theorem 2.

Original: Now we start to study an upper bound for the third term.

Original: Since the third term has two different subscript representing different iterations for the outer-loop, we use g T ( · ) to represent g ( · ) (deﬁned in Eq.

Original: corresponding to the

Original: T th iteration for the outer loop, i.e., when λ = λ 0 (1 − γ ) T .

Original: [Equation]

Original: where the ﬁrst inequality is due to kβ -strong convexity of g T ( · ) and the global optimality of w ∗T w.r.t.

Original: g T ( · ) , the second inequality is due to g T ( w ) ≥ g T ( w ) for any w given T ≥ T , the third inequality is due to the global optimality of w ∗ T w.r.t.

Original: g T ( · ) .

Original: Let A T and A T be the optimal A achieving the maximum of G ( A ) (deﬁned in Eq.

Original: when w = w ∗ T and w = w ∗T respectively, then

Original: [Equation]

Original: Due to the optimality of A T , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is a result of applying inequality (cid:0) 1 + xn (cid:1) n ≥ 1 + x for n > 1 and | x | ≤ n .

Original: Applying Eq.

Original: (48) to Eq. (45) gives us an upper bound for the third term, i.e., [Equation] By combining Eq.
Revised: The proof can be found in Appendix 4.7.

Original: (31) from Lemma 1 giving an upper bound for the ﬁrst term in Eq.

Original: (44), Eq.

Original: (19) from Theorem 2 giving an upper bound for the second term, and Eq.

Original: (49) giving an upper bound for the third term, after some re-arrangements, we have the following result.

Original: [Equation]

Original: [Equation]

Original: when λ starts λ = λ 0 and decreases exponentially, i.e., λ ← (1 − γ ) · λ , c is deﬁned as

Original: [Equation]

Original: By using the fact F ( A T ) ≤ c 2 = max A ⊆ V, | A |≤ k F ( A ) , c can be further replaced by its upper bound

Original: [Equation]

Original: In addition, Eq.

Original: (13) and Eq.

Original: (16) show in both cases ˆ g ( ˆ w T ) ≤ g ( w ∗ T ) , by using the inequality in

Original: Eq.

Original: (26), we have

Original: [Equation]

Original: Hence, the ﬁrst term in Eq.

Original: (52) can be further simpliﬁed by using the above inequality, so we have

Original: [Equation]

Original: If we increase k linearly from k = k 0 as Step 11 in Algorithm 1, we have

Original: [Equation]

Original: If we can tolerate more expensive computational cost for running submodular maximization with larger budget k , and increase k exponentially, i.e., k ← (1 + ∆) · k , we instead have

Original: [Equation]

Original: This completes the proof.

Original: Remarks:
Revised: Remarks:

Original: In Eq.
Revised: In Eq.

Original: (40) of Theorem 3, the upper bound decreases exponentially with power p , so it shows the linear convergence rate of inner-loop.
Revised: (15) of Theorem 3, the upper bound decreases exponentially with power p , so it shows the linear convergence rate of inner-loop.

Original: Moreover, a smaller c indicates a better upper bound.
Revised: Moreover, a smaller c indicates a better upper bound.

Original: Interestingly, Eq.
Revised: Interestingly, Eq.

Original: (41) shows that c is a weighted sum of g ( w ∗∞ ) and c 2 , which are respectively the optimal objective value for the minimax problem Eq.
Revised: (16) shows that c is a weighted sum of g ( w ∗∞ ) and c 2 , which are respectively the optimal objective value for the minimax problem Eq.

Original: (2) without the submodular term and the optimal objective value for maximizing the submodular term only.
Revised: (2) without the submodular term and the optimal objective value for maximizing the submodular term only.

Original: Hence, it relates the convergence bound to the solutions of the two extreme cases.
Revised: Hence, it relates the convergence bound to the solutions of the two extreme cases.

Original: The weight for g ( w ∗∞ ) is  constant and the weight for c 2 decreases with increasing T .
Revised: The weight for g ( w ∗∞ ) is a constant and the weight for c 2 decreases with more outer-loop iterations T .

Original: In addition, Eq.
Revised: In addition, Eq.

Original: (42) and Eq.
Revised: (17) and Eq.

Original: (43) show linear convergence rate of the outer-loop.
Revised: (18) show linear convergence rate of the outer-loop.

Original: Although Theorem 3 holds for normal gradient descent, the analysis can be easily extended to accelerated gradient descent with momentum and stochastic gradient descent by substituting their update rules for ˆ w t +1 into the reasoning of the ﬁrst term’s upper bound.
Revised: Although Theorem 3 holds for normal gradient descent, the analysis can be easily extended to accelerated gradient descent with momentum and stochastic gradient descent by substituting their update rules for ˆ w t +1 into the reasoning of the ﬁrst term’s upper bound.

Original: 2. A DDITIONAL E MPIRICAL I MPROVEMENTS
Revised: 2. A DDITIONAL E MPIRICAL I MPROVEMENTS

Original: Step 6-7 of Algorithm 1 require computing the loss on all the available samples, and each step of submodular maximization needs to evaluate the marginal gains of all the unselected samples.
Revised: Step 6-7 of Algorithm 1 require computing the loss on all the available samples, and each step of submodular maximization needs to evaluate the marginal gains of all the unselected samples.

Original: This may lead to expensive computation and annotation cost (the loss computation need to know the labels) in practice.
Revised: This may lead to expensive computation and annotation cost (the loss computation need to know the labels) in practice.

Original: Empirically we use two tricks to achieve improvements in efﬁciency.
Revised: Empirically we use two tricks to achieve improvements in efﬁciency.

Original: Firstly, instead of selecting individual samples into A , we select clusters.
Revised: Firstly, instead of selecting individual samples into A , we select clusters.

Original: In particular, we replace the per-sample loss L ( y i , f ( x i , w )) with per-cluster loss L (cid:0) Y ( i ) , f
Revised: In particular, we replace the per-sample loss L ( y i , f ( x i , w )) with per-cluster loss L (cid:0) Y ( i ) , f

Original: ( X ( i ) , w ) (cid:1) that sums up losses of all samples in the cluster ( X ( i ) is the i th cluster and Y ( i ) denotes the labels).
Revised: ( X ( i ) , w ) (cid:1) that sums up losses of all samples in the cluster ( X ( i ) is the i th cluster and Y ( i ) denotes the labels).

Original: To save annotation costs, we further approximate it by the loss on the sample closest to the cluster centroid, i.e.,
Revised: To save annotation costs, we further approximate it by the loss on the sample closest to the cluster centroid, i.e.,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where C ( i ) contains the indices of the samples in the cluster, and x ( i ) with label y
Revised: where C ( i ) contains the indices of the samples in the cluster, and x ( i ) with label y

Original: ( i ) is the sample closest to the centroid.
Revised: ( i ) is the sample closest to the centroid.

Original: In practice, the loss on x ( i ) is sufﬁciently representative to reﬂect the hardness of the cluster.
Revised: In practice, the loss on x ( i ) is sufﬁciently representative to reﬂect the hardness of the cluster.

Original: When computing F ( A ) reﬂecting the diversity of selected clusters, we use the cluster centroid to represent each cluster.
Revised: When computing F ( A ) reﬂecting the diversity of selected clusters, we use the cluster centroid to represent each cluster.

Original: In Step 8, the gradient is computed on all the samples in the selected clusters rather than on x ( i ) .
Revised: In Step 8, the gradient is computed on all the samples in the selected clusters rather than on x ( i ) .

Original: By using this method, we only need to annotate and compute the loss for samples in the selected clusters and the representative samples x
Revised: By using this method, we only need to annotate and compute the loss for samples in the selected clusters and the representative samples x

Original: ( i ) of other clusters.
Revised: ( i ) of other clusters.

Original: The size of ground set in submodular maximization is also reduced to the number of clusters.
Revised: The size of ground set in submodular maximization is also reduced to the number of clusters.

Original: We can further reduce the ground set to save computation during submodular maximization via pruning methods, which lead to zero loss (Wei et al., 2014a) or sufﬁciently small loss (Zhou et al., 2017) on objective G ( A ) .
Revised: We can further reduce the ground set to save computation during submodular maximization via pruning methods, which lead to zero loss (Wei et al., 2014a) or sufﬁciently small loss (Zhou et al., 2017) on objective G ( A ) .

Original: In MCL, as λ decreases and G ( A ) becomes close to modular, pruning method can rule out more elements.
Revised: In MCL, as λ decreases and G ( A ) becomes close to modular, pruning method can rule out more elements.

Original: We provide more details and discuss other speedup methods for our submodular maximization in 4.4.
Revised: We provide more details and discuss other speedup methods for our submodular maximization in 4.8.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: In this section, we apply different curriculum learning methods to train a logistic regression model on 20newsgroups (Lang, 1995), LeNet5 on MNIST (Lecun et al., 1998), and a convolutional neural nets (2Conv-Pool-2Conv-Pool-3DenseLayer)on CIFAR10 (Krizhevsky & Hinton, 2009).
Revised: In this section, we apply different curriculum learning methods to train a logistic regression model on 20newsgroups (Lang, 1995), LeNet5 on MNIST (Lecun et al., 1998), and a convolutional neural nets (2Conv-Pool-2Conv-Pool-3DenseLayer)on CIFAR10 (Krizhevsky & Hinton, 2009).

Original: Details on the datasets can be found in
Revised: Details on the datasets can be found in

Original: Table 3 of Appendix.
Revised: Table 3 of Appendix.

Original: We compare MCL and its variants to SPL (Kumar et al., 2010), SPLD (Jiang et al., 2014) and SGD with a random curriculum (i.e., with random batches).
Revised: We compare MCL and its variants to SPL (Kumar et al., 2010), SPLD (Jiang et al., 2014) and SGD with a random curriculum (i.e., with random batches).

Original: They all use mini-batch SGD π ( · , η ) with the same learning rate strategy to update the parameters w .
Revised: They all use mini-batch SGD π ( · , η ) with the same learning rate strategy to update the parameters w .

Original: They differ only in the curriculum, i.e., different training sequences.
Revised: They differ only in the curriculum, i.e., different training sequences.

Original: In SPL and SPLD, the training set starts from a ﬁxed size ( 4000 samples for 20newsgroups,samples for MNIST and CIFAR10), and increases by a factor µ = 0 .
Revised: In SPL and SPLD, the training set starts from a ﬁxed size ( 4000 samples for 20newsgroups,samples for MNIST and CIFAR10), and increases by a factor µ = 0 .

Original: 1 per round of alternating minimization.
Revised: 1 per round of alternating minimization.

Original: The model w is updated after ρ passes of the selected training set per round.
Revised: The model w is updated after ρ passes of the selected training set per round.

Original: In SPLD, we further have a weight for negative group sparsity starting from ξ and increasing by a factor of 0 .per round.
Revised: In SPLD, we further have a weight for negative group sparsity starting from ξ and increasing by a factor of 0 .per round.

Original: We tried 5 different combinations of { ρ, µ } and { ρ, ξ } for SPL and SPLD respectively.
Revised: We tried 5 different combinations of { ρ, µ } and { ρ, ξ } for SPL and SPLD respectively.

Original: The best combination with the smallest error rate is reported.
Revised: The best combination with the smallest error rate is reported.

Original: Although both SPL and SPLD can be reduced to SGD when λ = 0 , we do not include this special case because SGD is already a baseline.
Revised: Although both SPL and SPLD can be reduced to SGD when λ = 0 , we do not include this special case because SGD is already a baseline.

Original: For SGD with a random curriculum, results of 10 independent trials are reported.
Revised: For SGD with a random curriculum, results of 10 independent trials are reported.

Original: [Table]
Revised: [Table]

Original: In our MCL experiments, a feature based submodular function (Wei et al., 2014b) is used for regularization, i.e., F ( A )
Revised: In our MCL experiments, a feature based submodular function (Wei et al., 2014b) is used for regularization, i.e., F ( A )

Original: = (cid:80) u
Revised: = (cid:80) u

Original: ∈U ω u (cid:112) c u ( A ) , where U is a set of features.
Revised: ∈U ω u (cid:112) c u ( A ) , where U is a set of features.

Original: For a subset A of clusters, c u ( A ) = (cid:80) i ∈
Revised: For a subset A of clusters, c u ( A ) = (cid:80) i ∈

Original: A c u ( i ) , where c u ( i ) is the nonnegative score of centroid for cluster i .
Revised: A c u ( i ) , where c u ( i ) is the nonnegative score of centroid for cluster i .

Original: We use
Revised: We use

Original: TF-IDF features for 20newsgroup, and the input feature to the output layer (given by ReLU) for
Revised: TF-IDF features for 20newsgroup, and the input feature to the output layer (given by ReLU) for

Original: MNIST and CIFAR10.
Revised: MNIST and CIFAR10.

Original: The submodularity of F ( A ) holds because these vectors are nonnegative.
Revised: The submodularity of F ( A ) holds because these vectors are nonnegative.

Original: We consider four variants of MCL: 1) MCL with λ = 0 and ∆ = 0 , having neither submodular regularization that promotes diversity nor scheduling of k that increases hardness; 2) MCL with ∆ = 0 , without submodular regularization but with scheduling of k ; 3) MCL+random, which inserts one round that randomly samples r clusters as training set ˆ A after every q rounds of the outer loop in
Revised: We consider four variants of MCL: 1) MCL with λ = 0 and ∆ = 0 , having neither submodular regularization that promotes diversity nor scheduling of k that increases hardness; 2) MCL with ∆ = 0 , without submodular regularization but with scheduling of k ; 3) MCL+random, which inserts one round that randomly samples r clusters as training set ˆ A after every q rounds of the outer loop in

Original: Algorithm 1; 4) MCL+k, which has scheduling of λ and k , but does not use a random training set.
Revised: Algorithm 1; 4) MCL+k, which has scheduling of λ and k , but does not use a random training set.

Original: We tried 5 different combinations of { q, r } for MCL+random and 5 different ∆ for MCL+k, and report the one with the smallest test error.
Revised: We tried 5 different combinations of { q, r } for MCL+random and 5 different ∆ for MCL+k, and report the one with the smallest test error.

Original: Other parameters such as initial values for λ, k , γ , p and the total number of clusters are the same for different variants (exact values are given in Table 4 of Appendix. 
Revised: Other parameters such as initial values for λ, k , γ , p and the total number of clusters are the same for different variants (exact values are given in Table 4 of Appendix. [Figure]

Revised: (grey curves represents 10 random trials of SGD).

Original: In MCL, S UBMODULAR M AX is the only extra computation comparing to normal SGD.
Revised: In MCL, S UBMODULAR M AX is the only extra computation comparing to normal SGD.

Original: To show that its time cost is ignorable (but still brings a clear advantage as shown later), we report the total time cost of MCL+k and the time spent on S UBMODULAR M AX in Table 2.
Revised: To show that its time cost is ignorable (but still brings a clear advantage as shown later), we report the total time cost of MCL+k and the time spent on S UBMODULAR M AX in Table 2.

Original: We summarize the main results in Figure 1-4.
Revised: We summarize the main results in Figure 1-4.

Original: More results are given in the end of Appendix.
Revised: More results are given in the end of Appendix.

Original: In all ﬁgures, grey curves correspond to the 10 trials of SGD with a random curriculum.
Revised: In all ﬁgures, grey curves correspond to the 10 trials of SGD with a random curriculum.

Original: The legend gives the parameters used in different methods of the following formats: 1) SPL: ρ, µ ; 2) SPLD: ρ, ξ ; 3)
Revised: The legend gives the parameters used in different methods of the following formats: 1) SPL: ρ, µ ; 2) SPLD: ρ, ξ ; 3)

Original: MCL+random: q, r .
Revised: MCL+random: q, r .

Original:  Figure 1-2 show how the test error changes with the number of distinct training samples and the number of training batches, which reﬂects the training time.
Revised: [Figure] Figure 1-2 show how the test error changes with the number of distinct training samples and the number of training batches, which reﬂects the training time.

Original: The left plot in each ﬁgure implies the “sample complexity” of different methods, i.e., how many distinct training samples are neededto achieve the error rate. The right plot shows the convergence rate. [Figure] [Figure]
Revised: The left plot in each ﬁgure implies the “sample complexity” of different methods, i.e., how many distinct training samples are needed

Original: On all datasets, MCL and its variants outperform SPL and SPLD for both sample complexity and convergence rate.
Revised: On all datasets, MCL and its variants outperform SPL and SPLD for both sample complexity and convergence rate.

Original: They are slightly slower than SGD on convergence but can achieve much smaller error when using the same number of labeled samples.
Revised: They are slightly slower than SGD on convergence but can achieve much smaller error when using the same number of labeled samples.

Original: Moreover, when using the same learning rate strategy, they are more robust to overﬁtting, as shown in Figure 2.
Revised: Moreover, when using the same learning rate strategy, they are more robust to overﬁtting, as shown in Figure 2.

Original: In addition, they reduce the error faster than others in later stages, and MCL+k always achieves the lowest test error, as shown in
Revised: In addition, they reduce the error faster than others in later stages, and MCL+k always achieves the lowest test error, as shown in

Original: Table 1.

Original: Comparing Figure 1 with Figure 2-3, MCL has signiﬁcant advantages when applied to deep models.

Original: [Figure]
Revised: [Figure]

Original: Among the four variants of MCL, MCL+k achieves the fastest convergence speed and the smallest test error, followed by MCL+random, MCL( ∆ = 0 ) and MCL( λ = 0 , ∆ = 0 ) (the only exception is the test error on News20 between the last two variants).
Revised: Among the four variants of MCL, MCL+k achieves the fastest convergence speed and the smallest test error, followed by MCL+random, MCL( ∆ = 0 ) and MCL( λ = 0 , ∆ = 0 ) (the only exception is the test error on News20 between the last two variants).

Original: This indicates that the diversity introduced by submodular regularization does bring improvement, and changing both hardness and diversity leads to better performance.
Revised: This indicates that the diversity introduced by submodular regularization does bring improvement, and changing both hardness and diversity leads to better performance.

Original: The combination  but still cannot outperform MCL+k.
Revised: The combination of MCL and random curriculum speedups the convergence, but still cannot outperform MCL+k.

Original: We can obtain similar ranking on sample complexity, but the differences among variants are small. 
Revised: We can obtain similar ranking on sample complexity, but the differences among variants are small. [Figure] o f d [Figure]

Revised: t i n c t t r a i n i n g s a m p l e s

Original: [Figure]
Revised: [Figure]

Original: Figure 4 shows how the number of distinct training samples changes as training proceeds.
Revised: Figure 4 shows how the number of distinct training samples changes as training proceeds.

Original: It reﬂects the trade-off between “training on more new samples” vs. “training on fewer samples more often.”
Revised: It reﬂects the trade-off between “training on more new samples” vs. “training on fewer samples more often.”

Original: [Figure] and its variants usually require much fewer labeled samples than SGD but more than SPL and SPLD.
Revised: MCL and its variants usually require much fewer labeled samples than SGD but more than SPL and SPLD.

Original: Considering their advantages on the smaller error, better sample complexity, and faster convergence, MCL achieves a promising trade-off.
Revised: Considering their advantages on the smaller error, better sample complexity, and faster convergence, MCL achieves a promising trade-off.

Original:  A PPENDIX 4. P ROOF OF P ROPOSITION
Revised:  A PPENDIX 4. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Let g ( x ) = max
Revised: Let g ( x ) = max

Original: i g i ( x ) , where g i ( x ) is β -strongly convex for any i .
Revised: i g i ( x ) , where g i ( x ) is β -strongly convex for any i .

Original: According to a deﬁnition of strongly convex function given in Theorem 2.1.8 of (Nesterov, 2004), let λ ∈ [0 , 1] , we have 
Revised: According to a deﬁnition of strongly convex function given in Theorem 2.1.8 of (Nesterov, 2004), let λ ∈ [0 , 1] , we have g i [Equation] λx [Equation]

Revised: 1 − λ

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Original: The following proves that g ( x ) is also β -strongly convex:
Revised: The following proves that g ( x ) is also β -strongly convex:

Original: [Equation]
Revised: [Equation]

Revised: 4. P ROOF OF T HEOREM

Revised: Proof.

Revised: The objective g ( w ) of the minimax problem in Eq.

Revised: (2) after eliminating A is given in Eq.

Revised: Since G ( A ) in Eq.

Revised: (5) is monotone non-decreasing submodular, the optimal subset A when deﬁning g ( w ) in Eq.

Revised: (4) always has size k if | V | ≥ k .

Revised: In addition, because the loss function L ( y i , f ( x i , w )) is β -strongly convex, g ( w ) in Eq.

Revised: (4) is the maximum over multiple kβ -strongly convex functions with different A .

Revised: According to Proposition 1, g ( w ) is also kβ -strongly convex, i.e.,

Revised: [Equation]

Revised: Since the convex function g ( w ) achieves minimum on w ∗ , it is valid to substitute ∇ g ( w ∗ ) = 0 ∈ ∂g ( w ∗ ) into Eq.

Revised: After rearrangement, we have

Revised: [Equation]

Revised: In the following, we will prove g ( w ∗ ) ≥ α · g ( ˆ w ) , which together with Eq.

Revised: (21) will lead to the ﬁnal bound showing how close ˆ w is to w ∗ .

Revised: Note ˆ g ( w ) (Eq.

Revised: (6)) is a piecewise function, each piece of whom is convex and associated with different ˆ A achieved by a submodular maximization algorithm of approximation factor α .

Revised: Since ˆ A is not guaranteed to be a global maxima, unlike g ( w ) , the whole ˆ g ( w ) cannot be written as the maximum of multiple convex functions and thus can be non-convex.

Revised: Therefore, gradient descent in

Revised: Step 6-9 of Algorithm 1 can lead to either 1) ˆ w is a global minima of ˆ g ( w ) ; or 2) ˆ w is a local minima of ˆ g ( w ) .

Revised: Saddle points and local maxima do not exist on ˆ g ( w ) because each piece of it is convex.

Revised: 1) When ˆ w is a global minima of ˆ g ( w ) , we have

Revised: [Equation]

Revised: The ﬁrst inequality is due to g ( · ) ≥ ˆ g ( · ) .

Revised: The second inequality is due to the global optimality of ˆ w .

Revised: The third inequality is due to the approximation bound ˆ g ( · ) ≥ α · g ( · ) guaranteed by the submodular maximization in Step 7 of Algorithm 1.

Revised: When

Revised: ˆ

Revised: w is a local minima of ˆ g ( w ) , we have ∇

Revised: ˆ g ( ˆ w ) = 0 .

Revised: Let h ( w ) to be the piece of ˆ g ( w ) where ˆ w is located, then ˆ w has to be a global minima of h ( w ) due to the convexity of h ( w ) .

Revised: Let A denote the ground set of ˆ A on all pieces of ˆ g ( w ) , we deﬁne an auxiliary convex function ˜ g ( w ) as

Revised: [Equation]

Revised: It is convex because it is deﬁned as the maximum of multiple convex function.

Revised: So we have

Revised: [Equation]

Revised: [Equation]

Revised: The ﬁrst inequality is due to the deﬁnition of A , and the second inequality is a result of A ⊆ V by comparing g ( w ) in Eq.

Revised: (4) with ˜ g

Revised: ( w ) in Eq.

Revised: Let ˜ w denote a global minima of ˜ g ( w ) , we have

Revised: [Equation]

Revised: The ﬁrst inequality is due to Eq.

Revised: (24), the second inequality is due to the global optimality of ˜ w on ˜ g

Revised: ( w ) , the third inequality is due to the deﬁnition of ˜ g

Revised: ( w ) in Eq.

Revised: (23) ( ˜ g ( w ) is the maximum of all pieces of ˆ g ( w ) and h ( w ) is one piece of them), the fourth inequality is due to the global optimaliy of ˆ w on h ( w ) , the last inequality is due to the approximation bound ˆ g ( · ) ≥ α · g ( · ) guaranteed by the submodular maximization in Step 7 of Algorithm 1.

Revised: Therefore, in both cases we have g ( w ∗ ) ≥ α · g ( ˆ w ) .

Revised: Applying it to Eq.

Revised: (21) results in

Original:  4. PROOF OF P ROPOSITION
Revised: [Equation] 4. P ROPOSITION

Revised: Proposition 2.

Revised: If x ∈ [0 , 1] , the following inequality holds true.

Revised: x

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Due to two inequalities e x ≤ 1 + x + x 2 / 2 for x ≤ 0 and 1 − e − x ≥ x/ 2 for x ∈ [0 , 1] ,
Revised: Due to two inequalities e x ≤ 1 + x + x 2 / 2 for x ≤ 0 and 1 − e − x ≥ x/ 2 for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Revised: 4. P ROOF OF T HEOREM

Revised: Proof.

Revised: Applying the inequality in Proposition 2 and the approximation factor of lazy greedy

Revised: α = (1 − e − κ G ) /κ

Revised: G to the right hand side of Eq.

Revised: (10) from Theorem 1 yields

Revised: [Equation]

Revised: where κ G is the curvature of submodular function G ( · ) deﬁned in Eq.

Revised: According to the deﬁnition of curvature given in Eq.

Revised: [Equation]

Revised: and let κ F denotes the curvature of submodular function F ( · ) in Eq.

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: where we use L ( j, t ) as a simpler representation of per-sample loss L ( y j , f ( x i , ˆ w t )) if not causing any confusion

Revised: [Equation]

Revised: and the last inequality is due to the deﬁnition of curvature κ F for submodular function

Revised: F ( · ) .

Revised: Substituting the above inequality about κ G into Eq.

Revised: (29) results in

Revised: [Equation]

Revised: [Equation]

Revised: We use subscript as the index for iterations in the outer-loop, e.g., ˆ w T is the model weights w after the T

Revised: th iteration of outer-loop.

Revised: If we decrease λ exponentially from λ = λ 0 and increase k linearly from k = k 0 , as Step 11 in Algorithm 1, we have

Revised: [Equation]

Revised: [Equation]

Revised: According to the deﬁnition of g ( · ) in Eq.

Revised: (4), for g ( w ∗ T )

Revised: we have

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: Substituting Eq.

Revised: (33) to Eq.

Revised: (32) yields

Revised: [Equation]

Revised: If we can tolerate more expensive computational cost for running submodular maximization with larger budget k , and increase k exponentially,

Revised: i

Revised: [Equation]

Revised: e., k ← (1 + ∆) · k , we have

Revised: [Equation]

Revised: This completes the proof.

Original: 4. P ROOF OF P ROPOSITION
Revised: 4. P ROPOSITIONProposition

Revised: If x ∈ [0 , 1] , the following inequality holds true.

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Proposition 2 leads to
Revised: Proposition 2 leads to

Original: [Equation]
Revised: [Equation]

Original: By substituting inequality e x ≤ 1 + x + x 2 / 2 for x ≤ 0 into above inequality, we have
Revised: By substituting inequality e x ≤ 1 + x + x 2 / 2 for x ≤ 0 into above inequality [Equation] we have

Original: [Equation]
Revised: [Equation]

Revised: 4. P ROOF OF L EMMA

Revised: Proof.

Revised: On the optimization path from ˆ w p to ˆ w

Revised: [Equation]

Revised: ˆ g ( · ) may switch to different convex pieces.

Revised: Hence, we use h t ( · ) to denote the piece of ˆ g ( · ) on which ˆ w t is located, so ˆ g ( ˆ w t )

Revised: =

Revised: h t ( ˆ w t ) .

Revised: Because h t ( · ) is kβ -strongly convex,

Revised: [Equation]

Revised: Recall the auxiliary convex function ˜ g ( · ) deﬁned in Eq.

Revised: (23), by using the inequality shown in

Revised: Eq.

Revised: (25), we have

Revised: [Equation]

Revised: where the third inequality is due to the approximation factor α , and the fourth inequality is due to the kL -smoothness of ˆ g ( · ) .

Revised: Eq.

Revised: (41) results in

Revised: [Equation]

Revised: Let w =

Revised: ˆ w t and substitute Eq.

Revised: (42) to Eq.

Revised: [Equation]

Revised: due to ˆ g ( ˆ w t )

Revised: =

Revised: h t ( ˆ w t )

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: The last step is achieved by setting learning rate η = 1 / ( kL ) .

Revised: Recursively applying the above inequality from t =

Revised: p − 1 to t = 1 ( p is the number of iterations for inner loop) yields

Revised: [Equation]

Revised: [Equation]

Revised: Since α = 1 − e − κG κ G , Proposition 3 leads to

Revised: [Equation]

Revised: where the last inequality comes from Eq.

Revised: In addition, due to inequality (cid:0) 1 + xn (cid:1) n ≥ 1 + x for n > 1 and | x | ≤ n , we have

Revised: [Equation]

Revised: Applying Eq.

Revised: (45) and Eq.

Revised: (46) to Eq.

Revised: (44) results in

Revised: [Equation]

Revised: [Equation]

Revised: This completes the proof.

Revised: 4. P ROOF OF T HEOREM

Revised: Proof.

Revised: Triangle inequality results in

Revised: [Equation]

Revised: [Equation]

Revised: In the right hand side of above inequality, the ﬁrst term can be upper bounded by the result in Lemma 1, while the second term can be upper bounded by the results in Theorem 2.

Revised: Now we start to study an upper bound for the third term.

Revised: Since the third term has two different subscript representing different iterations for the outer-loop, we use g T ( · ) to represent g ( · ) (deﬁned in Eq.

Revised: corresponding to the

Revised: T th iteration for the outer loop, i.e., when λ = λ 0 (1 − γ ) T .

Revised: [Equation]

Revised: [Equation]

Revised: where the ﬁrst inequality is due to kβ -strong convexity of g T ( · ) and the global optimality of w ∗T w.r.t.

Revised: g T ( · ) , the second inequality is due to g T ( w ) ≥ g T ( w ) for any w given T ≥ T , the third inequality is due to the global optimality of w ∗ T w.r.t.

Revised: g T ( · ) .

Revised: Let A T and A T be the optimal A achieving the maximum of G ( A ) (deﬁned in Eq.

Revised: when w = w ∗ T and w = w ∗T respectively, then

Revised: [Equation]

Revised: Due to the optimality of A T , we have

Revised: [Equation]

Revised: where the last inequality is a result of applying inequality (cid:0) 1 + xn (cid:1) n ≥ 1 + x for n > 1 and | x | ≤ n .

Revised: Applying Eq.

Revised: (52) to Eq.

Revised: (49) gives us an upper bound for the third term, i.e.,

Revised: [Equation]

Revised: By combining Eq.

Revised: (14) from Lemma 1 giving an upper bound for the ﬁrst term in Eq.

Revised: (48), Eq.

Revised: (11) from Theorem 2 giving an upper bound for the second term, and Eq.

Revised: (53) giving an upper bound for the third term, after some re-arrangements, we have the following result.

Revised: [Equation]

Revised: when λ starts λ = λ 0 and decreases exponentially, i.e., λ ← (1 − γ ) · λ , c is deﬁned as

Revised: [Equation]

Revised: By using the fact F ( A T ) ≤ c 2 = max A ⊆ V, | A |≤ k F ( A ) , c can be further replaced by its upper bound

Revised: [Equation]

Revised: In addition, Eq.

Revised: (22) and Eq.

Revised: (25) show in both cases ˆ g ( ˆ w T ) ≤ g ( w ∗ T ) , by using the inequality in

Revised: Eq.

Revised: (33), we have

Revised: [Equation]

Revised: Hence, the ﬁrst term in Eq.

Revised: (56) can be further simpliﬁed by using the above inequality, so we have

Revised: [Equation]

Revised: If we increase k linearly from k = k 0 as Step 11 in Algorithm 1, we have

Revised: [Equation]

Revised: If we can tolerate more expensive computational cost for running submodular maximization with larger budget k , and increase k exponentially, i.e., k ← (1 + ∆) · k , we instead have

Revised: [Equation]

Revised: This completes the proof.

Original: 4. S UBMODULAR M AXIMIZATION USED IN O UR I MPLEMENTATION
Revised: 4. S UBMODULAR M AXIMIZATION USED IN O UR I MPLEMENTATION

Original: In our implementation, we use two strategies to accelerate the computation: 1) reduce the size of ground set by pruning methods before running any submodular maximization algorithm, we use (Wei et al., 2014a) here for its simplicity but running (Zhou et al., 2017) afterwards can reduce additional elements; 2) update A by a subgradient ascent step (which costs | V | function evaluations) of majorization-minimization (MM) scheme (Iyer et al., 2013) when a targeted approximation factor α has already been achieved on ˆ A from the previous iteration, otherwise the lazy greedy algorithm is applied. 
Revised: In our implementation, we use two strategies to accelerate the computation: 1) reduce the size of ground set by pruning methods before running any submodular maximization algorithm, we use (Wei et al., 2014a) here for its simplicity but running (Zhou et al., 2017) afterwards can reduce additional elements; 2) update A by a subgradient ascent step (which costs | V | function evaluations) of majorization-minimization (MM) scheme (Iyer et al., 2013) when a targeted approximation factor α has already been achieved on ˆ A from the previous iteration, otherwise the lazy greedy algorithm is applied. [Equation] ˆ A ) h

Revised: ˆ A (

Revised: ) − h ˆ A ( ˆ A

Original: Thecomplete algorithm is given in Algorithm 2.
Revised: The complete algorithm is given in Algorithm 2.

Original: Step 3 denotes the pruning procedure given in (Wei et al., 2014a).
Revised: Step 3 denotes the pruning procedure given in (Wei et al., 2014a).

Original: Step 4 compute an upper bound τ for max A ∈ V, | A |≤ k G ( A ) due to submodularity.
Revised: Step 4 compute an upper bound τ for max A ∈ V, | A |≤ k G ( A ) due to submodularity.

Original: Step 5-12 performs submodular maximization: if the targeted approximation factor α is already achieved on ˆ A we run subgradient ascent to further improve it, otherwise a lazy greedy algorithm is applied as in Step 11.
Revised: Step 5-12 performs submodular maximization: if the targeted approximation factor α is already achieved on ˆ A we run subgradient ascent to further improve it, otherwise a lazy greedy algorithm is applied as in Step 11.

Original: Note the modular maximization in Step 4 and Step 9 simply needs to ﬁnd the top k elements from V (cid:48) with the largest objective value.
Revised: Note the modular maximization in Step 4 and Step 9 simply needs to ﬁnd the top k elements from V (cid:48) with the largest objective value.

Original: Step 6-9 is identical to one step of MM scheme (Iyer et al., 2013) except Step 6 which arranges the elements V (cid:48)
Revised: Step 6-9 is identical to one step of MM scheme (Iyer et al., 2013) except Step 6 which arranges the elements V (cid:48)

Original: \ ˆ A in a non-decreasing order of
Revised: \ ˆ A in a non-decreasing order of

Original: G ( i | V (cid:48) \ i ) /G
Revised: G ( i | V (cid:48) \ i ) /G

Original: ( i ) .
Revised: ( i ) .

Original: The set of the ﬁrst i elements in permutation σ is denoted by S σi in Step 7.
Revised: The set of the ﬁrst i elements in permutation σ is denoted by S σi in Step 7.

Original: Step 6 places fewer elements before the ones with small G ( i | V (cid:48) \ i ) /G
Revised: Step 6 places fewer elements before the ones with small G ( i | V (cid:48) \ i ) /G

Original: ( i ) in σ , and intuitively tends to reduce the gap between L ( A ) and G ( A ) for any subset A .
Revised: ( i ) in σ , and intuitively tends to reduce the gap between L ( A ) and G ( A ) for any subset A .

Original: To see this, let A σ i − 1 (cid:44) { σ
Revised: To see this, let A σi − 1 (cid:44) { σ

Original: ( j ) ∈
Revised: ( j ) ∈

Original: A | j < i } τ = max A ∈ V, | A |≤ k (cid:88)

Original: i ∈ A (cid:2)

Original: L (cid:0) y i , f ( x i , w t ) (cid:1) + λF ( i ) (cid:3)

Original: ﬁrst

Original: Compute

Original: (cid:44)

Original: h

Original: ˆ A (

Original: ) − h ˆ A ( ˆ A

Original: ←

Original: 11: ˆ LazyGreedy G V (cid:48) , k ) ; 12: for any i ∈ A , the gap we want to reduce is
Revised: A | j < i } for any i ∈ A , the gap we want to reduce is

Original: [Equation]
Revised: [Equation]

Original: where the second term in the brackets is
Revised: where the second term in the brackets is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The elements σ ( i ) with small rank i associates with small S σi − 1 , which often results in (but does not guarantee)
Revised: The elements σ ( i ) with small rank i associates with small S σi − 1 , which often results in (but does not guarantee)

Original: small h
Revised: small h

Original: ˆ A ( σ ( i )) /G
Revised: ˆ A ( σ ( i )) /G

Original: ( σ ( i ))
Revised: ( σ ( i ))

Original: due to submodularity.
Revised: due to submodularity.

Original: According to Step 6, the elements with large rank in σ has large G ( σ ( i ) | V (
Revised: According to Step 6, the elements with large rank in σ has large G ( σ ( i ) | V (

Original: cid:48) \ σ ( i )) /G
Revised: cid:48) \ σ ( i )) /G

Original: ( σ ( i )) , the lower bound for h ˆ A ( σ ( i )) /G
Revised: ( σ ( i )) , the lower bound for h ˆ A ( σ ( i )) /G

Original: ( σ ( i )) in
Revised: ( σ ( i )) in

Original: Eq.
Revised: Eq.

Original: Together they indicate large h
Revised: Together they indicate large h

Original: ˆ A ( σ ( i )) /G
Revised: ˆ A ( σ ( i )) /G

Original: ( σ ( i )) and thus a small gap in Eq.
Revised: ( σ ( i )) and thus a small gap in Eq.

Original: It has been proved in (Iyer et al., 2013) that Step 6-9 produces a non-decreasing G ( ˆ A ) , i.e., if ˆ A + is the optimal ˆ A achieved in Line 9, we have
Revised: It has been proved in (Iyer et al., 2013) that Step 6-9 produces a non-decreasing G ( ˆ A ) , i.e., if ˆ A + is the optimal ˆ A achieved in Line 9, we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is due to that L ( · ) is the lower bound of G ( A ) ; the second inequality is due to the optimality of ˆ A + ; the equality is due to S σ k = ˆ A .
Revised: The ﬁrst inequality is due to that L ( · ) is the lower bound of G ( A ) ; the second inequality is due to the optimality of ˆ A + ; the equality is due to S σk = ˆ A .

Original: Lemma 2.
Revised: Lemma 2.

Original: Algorithm 2 outputs a solution ˆ A such that G ( ˆ A ) ≥ min { α, α LG }· max A ∈ V, | A |≤ k G ( A ) , where α LG is the approximation factor of lazy greedy.
Revised: Algorithm 2 outputs a solution ˆ A such that G ( ˆ A ) ≥ min { α, α LG }· max A ∈ V, | A |≤ k G ( A ) , where α LG is the approximation factor of lazy greedy.

Original: Proof.
Revised: Proof.

Original: Firstly, let A ∗ denote the optimal solution to Eq.
Revised: Firstly, let A ∗ denote the optimal solution to Eq.

Original: [Equation]
Revised: [Equation]

Original:  We prove that τ in Step 4 is an upper bound to G ( A ∗ ) as follows.
Revised: [Equation] [Table] [Table] We prove that τ in Step 4 is an upper bound to G ( A ∗ ) as follows.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is due to the fact that τ is the maximum of the objective in Step 4; the last inequality is due to submodularity, which guarantees F ( i ) ≥ F ( i | B ) for any B ⊆ V .
Revised: The ﬁrst inequality is due to the fact that τ is the maximum of the objective in Step 4; the last inequality is due to submodularity, which guarantees F ( i ) ≥ F ( i | B ) for any B ⊆ V .

Original: When G ( ˆ A ) ≥ α · τ (Step 5), as we discussed before, the subgradient ascent in Step 6-9 does not decrease the objective, so we have G ( ˆ A ) ≥ α · max A ∈ V, | A |≤ k G ( A ) for ˆ A obtained in Step 9.
Revised: When G ( ˆ A ) ≥ α · τ (Step 5), as we discussed before, the subgradient ascent in Step 6-9 does not decrease the objective, so we have G ( ˆ A ) ≥ α · max A ∈ V, | A |≤ k G ( A ) for ˆ A obtained in Step 9.

Original: Otherwise, we run lazy greedy on the reduced ground set V (cid:48) .
Revised: Otherwise, we run lazy greedy on the reduced ground set V (cid:48) .

Original: According to Lemma 1 in (Wei et al.,

Original: [Table]

Original: [Table] 2014a) , the ground set reduction in Step 3 does not change the objective value achieved by lazy greedy.
Revised: According to Lemma 1 in (Wei et al., 2014a), the ground set reduction in Step 3 does not change the objective value achieved by lazy greedy.

Original: So we have G ( ˆ A ) ≥ α LG · max A ∈ V, | A |≤ k G ( A ) for ˆ A obtained in Step 11.
Revised: So we have G ( ˆ A ) ≥ α LG · max A ∈ V, | A |≤ k G ( A ) for ˆ A obtained in Step 11.

Original: This completes the proof.
Revised: This completes the proof.

Original: Algorithm 2 simply combines three proposed submodular maximization techniques, but effectively accelerates the update of ˆ A in practice.
Revised: Algorithm 2 simply combines three proposed submodular maximization techniques, but effectively accelerates the update of ˆ A in practice.

Original: Firstly, as λ decreasing, G ( A ) turns to be more modular, and
Revised: Firstly, as λ decreasing, G ( A ) turns to be more modular, and

Original: G ( i | V \ i ) becomes closer to G ( i ) .
Revised: G ( i | V \ i ) becomes closer to G ( i ) .

Original: So the pruning in Step 3 becomes similar to sorting, and rules out more elements from V .
Revised: So the pruning in Step 3 becomes similar to sorting, and rules out more elements from V .

Original: Secondly, G ( A ) changes smoothly with w t between two iterations within
Revised: Secondly, G ( A ) changes smoothly with w t between two iterations within

Original: Algorithm 1.
Revised: Algorithm 1.

Original: When the learning rate η is small, ˆ A from the previous iteration can still achieve a large approximation factor, in which case running lazy greedy is not necessary, and Algorithm 2 thus uses faster subgradient ascent to update ˆ A .
Revised: When the learning rate η is small, ˆ A from the previous iteration can still achieve a large approximation factor, in which case running lazy greedy is not necessary, and Algorithm 2 thus uses faster subgradient ascent to update ˆ A .

Original: This mechanism of setting targeted approximation factor α provides a trade-off between α and computational cost: if α is large, we can obtain large approximation factor but the computation might be expensive since lazy greedy is used more frequently; if α is small, single subgradient ascent will be used more often and the computational cost reduces, but the approximation factor also decrease.
Revised: This mechanism of setting targeted approximation factor α provides a trade-off between α and computational cost: if α is large, we can obtain large approximation factor but the computation might be expensive since lazy greedy is used more frequently; if α is small, single subgradient ascent will be used more often and the computational cost reduces, but the approximation factor also decrease.

Original: Note if we set α = 1 , Algorithm 2 is forced to run lazy greedy to update ˆ A .
Revised: Note if we set α = 1 , Algorithm 2 is forced to run lazy greedy to update ˆ A .

Original: In experiments, we set α = 0 .
Revised: In experiments, we set α = 0 .

Original: Though α = 1 might lead to smaller approximation error, empirically α = 0 .
Revised: Though α = 1 might lead to smaller approximation error, empirically α = 0 .

Original: 5 often results in less computation for Algorithm 2 while maintaining promising performance
Revised: 5 often results in less computation for Algorithm 2 while maintaining promising performance.

Original: [Figure]
Revised: [Figure]

Original: T r a i n i n g e rr

Original: [Figure]
Revised: [Figure]

Original: r r a t e ( % )

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]