Original: S TOCHASTIC AUC M AXIMIZATION WITH D EEP N EURAL N ETWORKS
Revised: S TOCHASTIC AUC M AXIMIZATION WITH D EEP N EURAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Stochastic AUC maximization has garnered an increasing interest due to better ﬁt to imbalanced data classiﬁcation.
Revised: Stochastic AUC maximization has garnered an increasing interest due to better ﬁt to imbalanced data classiﬁcation.

Original: However, existing works are limited to stochastic
Revised: However, existing works are limited to stochastic

Original: AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data.
Revised: AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data.

Original: In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model.
Revised: In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model.

Original: Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a non-convex concave min-max problem.
Revised: Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a non-convex concave min-max problem.

Original: The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well.
Revised: The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well.

Original: In particular, we propose to explore Polyak-Łojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme.
Revised: In particular, we propose to explore Polyak-Łojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme.

Original: An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate.
Revised: An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate.

Original: Our experimental results demonstrate the effectiveness of the proposed algorithms.
Revised: Our experimental results demonstrate the effectiveness of the proposed algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has been witnessed with tremendous success for various tasks, including computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: Deep learning has been witnessed with tremendous success for various tasks, including computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: He et al., 2016; Ren et al., 2015), speech recognition (Hinton et al., 2012; Mohamed et al., 2012; Graves, 2013), natural language processing (Bahdanau et al., 2014; Sutskever et al., 2014;
Revised: He et al., 2016; Ren et al., 2015), speech recognition (Hinton et al., 2012; Mohamed et al., 2012; Graves, 2013), natural language processing (Bahdanau et al., 2014; Sutskever et al., 2014;

Original: Devlin et al., 2018), etc.
Revised: Devlin et al., 2018), etc.

Original: From an optimization perspective, all of them are solving an empirical risk minimization problem in which the objective function is a surrogate loss of the prediction error made by a deep neural network in comparison with the ground-truth label.
Revised: From an optimization perspective, all of them are solving an empirical risk minimization problem in which the objective function is a surrogate loss of the prediction error made by a deep neural network in comparison with the ground-truth label.

Original: For example, for image classiﬁcation task, the objective function is often chosen as the cross entropy between the probability distribution calculated by forward propagation of a convolutional neural network and the vector encoding true label information (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: For example, for image classiﬁcation task, the objective function is often chosen as the cross entropy between the probability distribution calculated by forward propagation of a convolutional neural network and the vector encoding true label information (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: He et al., 2016), where the cross entropy is a surrogate loss of the misclassiﬁcation rate.
Revised: He et al., 2016), where the cross entropy is a surrogate loss of the misclassiﬁcation rate.

Original: However, when the data is imbalanced, this formulation is not reasonable since the data coming from minor class have little effect in this case and the model is almost determined by the data from the majority class.
Revised: However, when the data is imbalanced, this formulation is not reasonable since the data coming from minor class have little effect in this case and the model is almost determined by the data from the majority class.

Original: To address this issue, AUC maximization has been proposed as a new learning paradigm (Zhao et al., 2011).
Revised: To address this issue, AUC maximization has been proposed as a new learning paradigm (Zhao et al., 2011).

Original: Statistically, AUC (short for Area Under the ROC curve) is deﬁned as the probability that the prediction score of a positive example is higher than that of a negative example (Hanley & McNeil, 1982; 1983) .
Revised: Statistically, AUC (short for Area Under the ROC curve) is deﬁned as the probability that the prediction score of a positive example is higher than that of a negative example (Hanley & McNeil, 1982; 1983) .

Original: Compared with misclassiﬁcation rate and its corresponding surrogate loss, AUC is more suitable for imbalanced data setting (Elkan, 2001).
Revised: Compared with misclassiﬁcation rate and its corresponding surrogate loss, AUC is more suitable for imbalanced data setting (Elkan, 2001).

Original: Several online or stochastic algorithms for
Revised: Several online or stochastic algorithms for

Original: AUC maximization have been developed based on a convex surrogate loss (Zhao et al., 2011; Gao et al., 2013;
Revised: AUC maximization have been developed based on a convex surrogate loss (Zhao et al., 2011; Gao et al., 2013;

Original: Ying et al., 2016; Liu et al., 2018; Natole et al., 2018).
Revised: Ying et al., 2016; Liu et al., 2018; Natole et al., 2018).

Original: However, all of these works only consider learning a linear predictive model.
Revised: However, all of these works only consider learning a linear predictive model.

Original: This naturally motivates the following question
Revised: This naturally motivates the following question:

Original: : How to design stochastic algorithms with provable guarantees to solve the AUC maximization problem with a deep neural network as the predictive model?
Revised: How to design stochastic algorithms with provable guarantees to solve the AUC maximization problem with a deep neural network as the predictive model?

Original: In this paper, we make some efforts to answer this question.
Revised: In this paper, we make some efforts to answer this question.

Original: We design two algorithms with stateof-the-art complexities for this problem.
Revised: We design two algorithms with stateof-the-art complexities for this problem.

Original: Based on a surrogated loss of AUC and inspired by the min-max reformulation in (Ying et al., 2016), we cast the problem into a non-convex concave minmax stochastic optimization problem, where it is nonconvex in the primal variable and concave in the dual variable.
Revised: Based on a surrogated loss of AUC and inspired by the min-max reformulation in (Ying et al., 2016), we cast the problem into a non-convex concave minmax stochastic optimization problem, where it is nonconvex in the primal variable and concave in the dual variable.

Original: This allows us to leverage the inexact proximal point algorithmic framework proposed in (Raﬁque et al., 2018) to solve stochastic AUC maximization with a deep neural network.
Revised: This allows us to leverage the inexact proximal point algorithmic framework proposed in (Raﬁque et al., 2018) to solve stochastic AUC maximization with a deep neural network.

Original: However, their algorithms are limited for stochastic AUC maximization with a deep neural network due to three reasons.
Revised: However, their algorithms are limited for stochastic AUC maximization with a deep neural network due to three reasons.

Original: First, their algorithms are general and do not utilize the underlying favorable property of the the objective function induced by an overparameterized deep neural network, which prevents them from designing better algorithms with faster convergence.
Revised: First, their algorithms are general and do not utilize the underlying favorable property of the the objective function induced by an overparameterized deep neural network, which prevents them from designing better algorithms with faster convergence.

Original: Second, these algorithms use a polynomially decaying step size scheme instead of the widely used geometrically decaying step size scheme in deep neural network training.
Revised: Second, these algorithms use a polynomially decaying step size scheme instead of the widely used geometrically decaying step size scheme in deep neural network training.

Original: Third, the algorithm in (Raﬁque et al., 2018) with the best attainable complexity only applies to the ﬁnite-sum setting, which needs to go through all data at the end of each stage and are not applicable to the pure stochastic setting.
Revised: Third, the algorithm in (Raﬁque et al., 2018) with the best attainable complexity only applies to the ﬁnite-sum setting, which needs to go through all data at the end of each stage and are not applicable to the pure stochastic setting.

Original: To address these limitations, we propose to leverage the Polyak-Łojasiewicz (PL) condition of the objective function for AUC maximization with a deep neural network.
Revised: To address these limitations, we propose to leverage the Polyak-Łojasiewicz (PL) condition of the objective function for AUC maximization with a deep neural network.

Original: The PL condition (or its equivalent condition) has been proved for a class of linear and non-linear neural networks (Hardt &
Revised: The PL condition (or its equivalent condition) has been proved for a class of linear and non-linear neural networks (Hardt &

Original: Ma, 2016; Charles & Papailiopoulos, 2017; Zhou & Liang, 2017).
Revised: Ma, 2016; Charles & Papailiopoulos, 2017; Zhou & Liang, 2017).

Original: It is the key to recent developments that prove that (stochastic) gradient descent can ﬁnd a global minimum for an overparameterized deep neural network (Allen-Zhu et al., 2018; Du et al., 2018b).
Revised: It is the key to recent developments that prove that (stochastic) gradient descent can ﬁnd a global minimum for an overparameterized deep neural network (Allen-Zhu et al., 2018; Du et al., 2018b).

Original: It is also observed in practice for learning deep neural networks (Li & Yuan, 2017; Kleinberg et al., 2018).
Revised: It is also observed in practice for learning deep neural networks (Li & Yuan, 2017; Kleinberg et al., 2018).

Original: From an optimization perspective, the PL condition has been considered extensively for designing faster optimization algorithms in the literature (Karimi et al., 2016; Reddi et al., 2016;
Revised: From an optimization perspective, the PL condition has been considered extensively for designing faster optimization algorithms in the literature (Karimi et al., 2016; Reddi et al., 2016;

Original: Lei et al., 2017).
Revised: Lei et al., 2017).

Original: However, there still remains a big gap between existing algorithms that focus on solving a minimization problem and the considered min-max problem of AUC maximization.
Revised: However, there still remains a big gap between existing algorithms that focus on solving a minimization problem and the considered min-max problem of AUC maximization.

Original: It is a non-trivial task to leverage the PL condition of a non-convex minimization objective for developing faster primal-dual stochastic algorithms to solve its equivalent non-convex concave min-max problem.
Revised: It is a non-trivial task to leverage the PL condition of a non-convex minimization objective for developing faster primal-dual stochastic algorithms to solve its equivalent non-convex concave min-max problem.

Original: The main theoretical contributions in this paper are to solve this issue.
Revised: The main theoretical contributions in this paper are to solve this issue.

Original: Our contributions are:
Revised: Our contributions are:

Original: • We propose a stochastic algorithm named Proximal Primal-Dual Stochastic Gradient (PPD-SG) for solving a min-max formulation of AUC maximization under the PL condition of the surrogated AUC objective with a deep neural network.
Revised: • We propose a stochastic algorithm named Proximal Primal-Dual Stochastic Gradient (PPD-SG) for solving a min-max formulation of AUC maximization under the PL condition of the surrogated AUC objective with a deep neural network.

Original: We establish a convergence rate in the order of O (1 /(cid:15) ) , which is faster than that achieved by simply applying the result in (Raﬁque et al., 2018) to the considered problem under the PL condition, i
Revised: We establish a convergence rate in the order of O (1 /(cid:15) ) , which is faster than that achieved by simply applying the result in (Raﬁque et al., 2018) to the considered problem under the PL condition, i

Original: e., O (1 /(cid:15) 3 ) and O ( n/(cid:15) ) with n being the size of training set.
Revised: e., O (1 /(cid:15) 3 ) and O ( n/(cid:15) ) with n being the size of training set.

Original: • In addition, we propose an AdaGrad-style primal-dual algorithm named Proximal Primal-Dual Adagrad (PPD-Adagrad), and show that it enjoys better adaptive complexity when the growth of cumulative stochastic gradient is slow.
Revised: • In addition, we propose an AdaGrad-style primal-dual algorithm named Proximal Primal-Dual Adagrad (PPD-Adagrad), and show that it enjoys better adaptive complexity when the growth of cumulative stochastic gradient is slow.

Original: This is the ﬁrst time an adaptive convergence of a stochastic AdaGrad-style algorithm is established for solving non-convex concave min-max problems.
Revised: This is the ﬁrst time an adaptive convergence of a stochastic AdaGrad-style algorithm is established for solving non-convex concave min-max problems.

Original: • We evaluate the proposed algorithms on several large-scale benchmark datasets.
Revised: • We evaluate the proposed algorithms on several large-scale benchmark datasets.

Original: The experimental results show that our algorithms have superior performance than other baselines.
Revised: The experimental results show that our algorithms have superior performance than other baselines.

Revised: To the best of our knowledge

Original: To the best of our knowledge, this is the ﬁrst work incorporating PL condition into stochastic AUC maximization with a deep neural network as the predictive model, and more generally into solving a non-convex concave min-max problem
Revised: , this is the ﬁrst work incorporating PL condition into stochastic AUC maximization with a deep neural network as the predictive model and more generally into solving a non-convex concave min-max problem.

Original: Our results achieve the state-of-the-art iteration complexity for non-convex concave min-max problems.
Revised: Our results achieve the state-of-the-art iteration complexity for non-convex concave min-max problems

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Stochastic AUC Maximization.
Revised: Stochastic AUC Maximization.

Original: Stochastic AUC maximization in the classical online setting is challenging due to its pairwise nature.
Revised: Stochastic AUC maximization in the classical online setting is challenging due to its pairwise nature.

Original: There are several studies trying to update the model each time based on a new sampled/received training data.
Revised: There are several studies trying to update the model each time based on a new sampled/received training data.

Original: Instead of storing all examples in the memory,
Revised: Instead of storing all examples in the memory,

Original: Zhao et al.
Revised: Zhao et al.

Original: (2011) employ reservoir sampling technique to maintain representative samples in a buffer, based on which their algorithms update the model.
Revised: (2011) employ reservoir sampling technique to maintain representative samples in a buffer, based on which their algorithms update the model.

Original: To get optimal regret bound, their buffer size needs to be O ( √ n ) , where n is the number of received training examples.
Revised: To get optimal regret bound, their buffer size needs to be O ( √ n ) , where n is the number of received training examples.

Original: Gao et al.
Revised: Gao et al.

Original: (2013) design a new algorithm which is not buffer-based.
Revised: (2013) design a new algorithm which is not buffer-based.

Original: Instead, their algorithm needs to maintain the ﬁrst-order and second-order statistics of the received data to compute the stochastic gradient, which is prohibitive for high dimensional data.
Revised: Instead, their algorithm needs to maintain the ﬁrst-order and second-order statistics of the received data to compute the stochastic gradient, which is prohibitive for high dimensional data.

Original: Based on a novel saddle-point reformulation of a surrogate loss of AUC proposed by (Ying et al., 2016), there are several studies (Ying et al., 2016; Liu et al., 2018;
Revised: Based on a novel saddle-point reformulation of a surrogate loss of AUC proposed by (Ying et al., 2016), there are several studies (Ying et al., 2016; Liu et al., 2018;

Original: Natole et al., 2018) trying to design stochastic primal-dual algorithms.
Revised: Natole et al., 2018) trying to design stochastic primal-dual algorithms.

Original: Ying et al.
Revised: Ying et al.

Original: (2016) employ the classical primal-dual stochastic gradient (Nemirovski et al., 2009) and obtain (cid:101)
Revised: (2016) employ the classical primal-dual stochastic gradient (Nemirovski et al., 2009) and obtain (cid:101)

Original: O (1 / √ t ) convergence rate.
Revised: O (1 / √ t ) convergence rate.

Original: Natole et al.
Revised: Natole et al.

Original: (2018) add a strongly convex regularizer, invoke composite mirror descent (Duchi et al., 2010) and achieve (cid:101)
Revised: (2018) add a strongly convex regularizer, invoke composite mirror descent (Duchi et al., 2010) and achieve (cid:101)

Original: O (1 /t ) convergence rate.
Revised: O (1 /t ) convergence rate.

Original: Liu et al.
Revised: Liu et al.

Original: (2018) leverage the structure of the formulation, design a multi-stage algorithm and achieve (cid:101)
Revised: (2018) leverage the structure of the formulation, design a multi-stage algorithm and achieve (cid:101)

Original: O (1 /t ) convergence rate without strong convexity assumptions.
Revised: O (1 /t ) convergence rate without strong convexity assumptions.

Original: However, all of them only consider learning a linear model, which results in a convex objective function.
Revised: However, all of them only consider learning a linear model, which results in a convex objective function.

Original: Non-Convex Min-max Optimization.
Revised: Non-Convex Min-max Optimization.

Original: Stochastic optimization of non-convex min-max problems have received increasing interests recently (Raﬁque et al., 2018; Lin et al., 2018;
Revised: Stochastic optimization of non-convex min-max problems have received increasing interests recently (Raﬁque et al., 2018; Lin et al., 2018;

Original: Sanjabi et al., 2018;
Revised: Sanjabi et al., 2018;

Original: Lu et al., 2019; Jin et al., 2019).
Revised: Lu et al., 2019; Jin et al., 2019).

Original: When the objective function is weakly convex in the primal variable and is concave in the dual variable, Raﬁque et al.
Revised: When the objective function is weakly convex in the primal variable and is concave in the dual variable, Raﬁque et al.

Original: (2018) design a proximal guided algorithm in spirit of the inexact proximal point method (Rockafellar, 1976), which solves a sequence of convexconcave subproblems constructed by adding a quadratic proximal term in the primal variable with a periodically updated reference point.
Revised: (2018) design a proximal guided algorithm in spirit of the inexact proximal point method (Rockafellar, 1976), which solves a sequence of convexconcave subproblems constructed by adding a quadratic proximal term in the primal variable with a periodically updated reference point.

Original: Due to the potential non-smoothness of objective function, they show the convergence to a nearly-stationary point for the equivalent minimization problem.
Revised: Due to the potential non-smoothness of objective function, they show the convergence to a nearly-stationary point for the equivalent minimization problem.

Original: In the same vein as (Raﬁque et al., 2018), Lu et al.
Revised: In the same vein as (Raﬁque et al., 2018), Lu et al.

Original: (2019) design an algorithm by adopting the block alternating minimization/maximization strategy and show the convergence in terms of the proximal gradient.
Revised: (2019) design an algorithm by adopting the block alternating minimization/maximization strategy and show the convergence in terms of the proximal gradient.

Original: When the objective is weakly convex and weakly concave, Lin et al.
Revised: When the objective is weakly convex and weakly concave, Lin et al.

Original: (2018) propose a proximal algorithm which solves a strongly monotone variational inequality in each epoch and establish its convergence to stationary point.
Revised: (2018) propose a proximal algorithm which solves a strongly monotone variational inequality in each epoch and establish its convergence to stationary point.

Original: Sanjabi et al.
Revised: Sanjabi et al.

Original: (2018) consider non-convex non-concave min-max games where the inner maximization problem satisﬁes a PL condition, based on which they design a multi-step deterministic gradient descent ascent with convergence to a stationary point.
Revised: (2018) consider non-convex non-concave min-max games where the inner maximization problem satisﬁes a PL condition, based on which they design a multi-step deterministic gradient descent ascent with convergence to a stationary point.

Original: It is notable that our work is different in that (i) we explore the PL condition for the outer minimization problem instead of the inner maximization problem; (ii) we focus on designing stochastic algorithms instead of deterministic algorithms.
Revised: It is notable that our work is different in that (i) we explore the PL condition for the outer minimization problem instead of the inner maximization problem; (ii) we focus on designing stochastic algorithms instead of deterministic algorithms.

Original: Leveraging PL Condition for Minimization.
Revised: Leveraging PL Condition for Minimization.

Original: PL condition is ﬁrst introduced by Polyak (Polyak, 1963), which shows that gradient descent is able to enjoy linear convergence to a global minimum under this condition.
Revised: PL condition is ﬁrst introduced by Polyak (Polyak, 1963), which shows that gradient descent is able to enjoy linear convergence to a global minimum under this condition.

Original: Karimi et al.
Revised: Karimi et al.

Original: (2016) show that stochastic gradient descent, randomized coordinate descent, greedy coordinate descent are able to converge to a global minimum with faster rates under the PL condition.
Revised: (2016) show that stochastic gradient descent, randomized coordinate descent, greedy coordinate descent are able to converge to a global minimum with faster rates under the PL condition.

Original: If the objective function has a ﬁnite-sum structure and satisﬁes PL condition, there are several non-convex SVRG-style algorithms (Reddi et al., 2016; Lei et al., 2017; Nguyen et al., 2017; Zhou et al., 2018; Li & Li, 2018; Wang et al., 2018), which are guaranteed to converge to a global minimum with a linear convergence rate.
Revised: If the objective function has a ﬁnite-sum structure and satisﬁes PL condition, there are several non-convex SVRG-style algorithms (Reddi et al., 2016; Lei et al., 2017; Nguyen et al., 2017; Zhou et al., 2018; Li & Li, 2018; Wang et al., 2018), which are guaranteed to converge to a global minimum with a linear convergence rate.

Original: However, the stochastic algorithms in these works are developed for a minimization problem, and hence is not applicable to the min-max formulation for stochastic AUC maximization.
Revised: However, the stochastic algorithms in these works are developed for a minimization problem, and hence is not applicable to the min-max formulation for stochastic AUC maximization.

Original: To the best of our knowledge, Liu et al.
Revised: To the best of our knowledge, Liu et al.

Original: (2018) is the only work that leverages an equivalent condition to the PL condition (namely quadratic growth condition) to develop a stochastic primal-dual algorithm for AUC maximization with a fast rate.
Revised: (2018) is the only work that leverages an equivalent condition to the PL condition (namely quadratic growth condition) to develop a stochastic primal-dual algorithm for AUC maximization with a fast rate.

Original: However, as mentioned before their algorithm and analysis rely on the convexity of the objective function, which does not hold for AUC maximization with a deep neural network.
Revised: However, as mentioned before their algorithm and analysis rely on the convexity of the objective function, which does not hold for AUC maximization with a deep neural network.

Original: Finally, we notice that PL condition is the key to many recent works in deep learning for showing there is no spurious local minima or for showing global convergence of gradient descent and stochastic gradient descent methods (Hardt & Ma, 2016; Li & Yuan, 2017; Arora et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018b;a; Li & Liang, 2018; Allen-Zhu et al., 2018; Zou et al., 2018; Zou & Gu, 2019).
Revised: Finally, we notice that PL condition is the key to many recent works in deep learning for showing there is no spurious local minima or for showing global convergence of gradient descent and stochastic gradient descent methods (Hardt & Ma, 2016; Li & Yuan, 2017; Arora et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018b;a; Li & Liang, 2018; Allen-Zhu et al., 2018; Zou et al., 2018; Zou & Gu, 2019).

Original: Using the square loss, it has also been proved that the PL condition holds globally or locally for deep linear residual network (Hardt & Ma, 2016), deep linear network, one hidden layer neural network with Leaky ReLU activation (Charles & Papailiopoulos, 2017; Zhou & Liang, 2017).
Revised: Using the square loss, it has also been proved that the PL condition holds globally or locally for deep linear residual network (Hardt & Ma, 2016), deep linear network, one hidden layer neural network with Leaky ReLU activation (Charles & Papailiopoulos, 2017; Zhou & Liang, 2017).

Original: Several studies (Li & Yuan, 2017; Arora et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018b; Li & Liang, 2018) consider the trajectory of (stochastic) gradient descent on learning neural networks, and their analysis imply the PL condition in a certain form.
Revised: Several studies (Li & Yuan, 2017; Arora et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018b; Li & Liang, 2018) consider the trajectory of (stochastic) gradient descent on learning neural networks, and their analysis imply the PL condition in a certain form.

Original: For example, Du et al.
Revised: For example, Du et al.

Original: (2018b) show that when the width of a two layer neural network is sufﬁciently large, a global optimum would lie in the ball centered at the initial solution, in which PL condition holds.
Revised: (2018b) show that when the width of a two layer neural network is sufﬁciently large, a global optimum would lie in the ball centered at the initial solution, in which PL condition holds.

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: (2018) extends this insight further to overparameterized deep neural networks with ReLU activation, and show that
Revised: (2018) extends this insight further to overparameterized deep neural networks with ReLU activation, and show that

Original: the
Revised: the

Original: PL condition holds for a global minimum around a random initial solution.
Revised: PL condition holds for a global minimum around a random initial solution.

Original:  P RELIMINARIES AND N OTATIONS
Revised:  P RELIMINARIES AND N OTATIONS

Original: Let (cid:107) · (cid:107) denote the Euclidean norm.
Revised: Let (cid:107) · (cid:107) denote the Euclidean norm.

Original: A function f ( x ) is ρ -weakly convex if f ( x ) + ρ 2 (cid:107) x (cid:107) 2 is convex, where ρ is the so-called weak-convexity parameter.
Revised: A function f ( x ) is ρ -weakly convex if f ( x ) + ρ 2 (cid:107) x (cid:107) 2 is convex, where ρ is the so-called weak-convexity parameter.

Original: A function f ( x ) satisﬁes PL condition with parameter µ > 0 if f ( x ) − f ( x ∗ ) ≤ 1 2 µ (cid:107)∇ f ( x ) (cid:107) 2 , where x ∗ stands for the optimal solution of f .
Revised: A function f ( x ) satisﬁes PL condition with parameter µ > 0 if f ( x ) − f ( x ∗ ) ≤ 12 µ (cid:107)∇ f ( x ) (cid:107) 2 , where x ∗ stands for the optimal solution of f .

Original: Let z
Revised: Let z

Original: =
Revised: =

Original: ( x , y ) ∼ P denote a random data following an unknown distribution P , where x ∈ X represents the feature vector and y ∈ Y =
Revised: ( x , y ) ∼ P denote a random data following an unknown distribution P , where x ∈ X represents the feature vector and y ∈ Y =

Original: {− 1 , +1 } represents the label.
Revised: {− 1 , +1 } represents the label.

Original: Denote by Z = X
Revised: Denote by Z = X

Original: × Y and by p =
Revised: × Y and by p =

Original: Pr ( y = 1) =
Revised: Pr ( y = 1) =

Original: E y (cid:2) I
Revised: E y (cid:2) I

Original: [ y =1] (cid:3) , where I ( · ) is the indicator function.
Revised: [ y =1] (cid:3) , where I ( · ) is the indicator function.

Original: The area under the curve (AUC) on a population level for a scoring function h : X → R is deﬁned as
Revised: The area under the curve (AUC) on a population level for a scoring function h : X → R is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where z = ( x , y ) and z (cid:48) =
Revised: , where z = ( x , y ) and z (cid:48) =

Original: ( x (cid:48) , y (cid:48) ) are drawn independently from P .
Revised: ( x (cid:48) , y (cid:48) ) are drawn independently from P .

Original: By employing the squared loss as the surrogate for the indicator function that is a common choice used by previous studies (Ying et al., 2016; Gao et al., 2013), the AUC maximization problem can be formulated as
Revised: By employing the squared loss as the surrogate for the indicator function that is a common choice used by previous studies (Ying et al., 2016; Gao et al., 2013), the AUC maximization problem can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where H denotes a hypothesis class.
Revised: where H denotes a hypothesis class.

Original: All previous works of AUC maximization assume h ( x ) = w (cid:62) x for simplicity.
Revised: All previous works of AUC maximization assume h ( x ) = w (cid:62) x for simplicity.

Original: Instead, we consider learning a general nonlinear model parameterized by w , i.e. h ( w ; x ) , which is not necessarily linear or convex in terms of w (e.g., h ( w ; x ) can be a score function deﬁned by a neural network with weights denoted by w ).
Revised: Instead, we consider learning a general nonlinear model parameterized by w , i.e. h ( w ; x ) , which is not necessarily linear or convex in terms of w (e.g., h ( w ; x ) can be a score function deﬁned by a neural network with weights denoted by w ).

Original: Hence, the corresponding optimization problem becomes
Revised: Hence, the corresponding optimization problem becomes

Original: [Equation]
Revised: [Equation]

Original: The following proposition converts the original optimization problem (1) into a saddle-point problem, which is similar to Theorem 1 in (Ying et al., 2016).
Revised: The following proposition converts the original optimization problem (1) into a saddle-point problem, which is similar to Theorem 1 in (Ying et al., 2016).

Original: For completeness, the proof is included in the supplement.
Revised: For completeness, the proof is included in the supplement.

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimization problem (1) is equivalent to
Revised: The optimization problem (1) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where z = ( x , y ) ∼ P , and
Revised: where z = ( x , y ) ∼ P , and

Original: [Equation]
Revised: [Equation]

Original: Remark : It is notable that the min-max formulation (2) is more favorable than the original formulation (1) for developing a stochastic algorithm that updates the model parameters based on one example or a mini-batch of samples.
Revised: Remark : It is notable that the min-max formulation (2) is more favorable than the original formulation (1) for developing a stochastic algorithm that updates the model parameters based on one example or a mini-batch of samples.

Original: For stochastic optimization of (1), one has to carefully sample both positive and negative examples, which is not allowed in an online setting.
Revised: For stochastic optimization of (1), one has to carefully sample both positive and negative examples, which is not allowed in an online setting.

Original: It is notable that in the classical batch-learning setting, p becomes the ratio of positive training examples and the expectation in (2) becomes average over n individual functions.
Revised: It is notable that in the classical batch-learning setting, p becomes the ratio of positive training examples and the expectation in (2) becomes average over n individual functions.

Original: However, our algorithms are applicable to both batch-learning setting and online learning setting.
Revised: However, our algorithms are applicable to both batch-learning setting and online learning setting.

Original: Deﬁne v = ( w (cid:62) , a, b ) (cid:62) , φ ( v ) = max α f ( v , α ) .
Revised: Deﬁne v = ( w (cid:62) , a, b ) (cid:62) , φ ( v ) = max α f ( v , α ) .

Original: It is clear that min w P ( w ) =
Revised: It is clear that min w P ( w ) =

Original: min v φ ( v ) and
Revised: min v φ ( v ) and

Original: P ( w ) ≤ φ ( v ) for any v = ( w (cid:62) , a, b ) (cid:62) .
Revised: P ( w ) ≤ φ ( v ) for any v = ( w (cid:62) , a, b ) (cid:62) .

Original: The following assumption is made throughout the paper.
Revised: The following assumption is made throughout the paper.

Original: Assumption 1.
Revised: Assumption 1.

Original: [Equation]
Revised: (1) µ ( φ ( v ) − φ ( v ∗ ))

Original:  φ ( v ) (cid:107) 2 , where µ > 0 and v ∗ is the optimal solution of φ .
Revised: ≤ 12 (cid:107)∇ φ ( v ) (cid:107) 2 , where µ > 0 and v ∗ is the optimal solution of φ .

Original: (2) h ( w ; x ) is ˜ L -Lipschitz continuous in terms of w for all x .
Revised: (2) h ( w ; x ) is ˜ L -Lipschitz continuous in terms of w for all x .

Original: (3) φ ( v ) is L -smooth.
Revised: (3) φ ( v ) is L -smooth.

Original: Var [ h ( w ; x ) | y = − 1] + Var [ h ( w ; x ) | y = 1] ≤ σ 2 .
Revised: Var [ h ( w ; x ) | y = − 1] + Var [ h ( w ; x ) | y = 1] ≤ σ 2 .

Original: Given a initial solution ¯ v 0 , there exists ∆ 0 > 0 such that φ (¯ v 0 ) − φ ( v ∗ ) ≤ ∆ 0 , where v ∗ is the global minimum of φ .
Revised: Given a initial solution ¯ v 0 , there exists ∆ 0 > 0 such that φ (¯ v 0 ) − φ ( v ∗ ) ≤ ∆ 0 , where v ∗ is the global minimum of φ .

Original: Remark :
Revised: Remark :

Original: The ﬁrst condition is inspired by a PL condition on the objective function P ( w ) for learning a deep neural network.
Revised: The ﬁrst condition is inspired by a PL condition on the objective function P ( w ) for learning a deep neural network.

Original: The following lemma establishes the connection.
Revised: The following lemma establishes the connection.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose (cid:107)∇ w h ( w ; x ) (cid:107) ≤ ˜ L for all w and x .
Revised: Suppose (cid:107)∇ w h ( w ; x ) (cid:107) ≤ ˜ L for all w and x .

Original: If P ( w ) satisﬁes PL condition, i.e. there exists µ (cid:48) > 0 , such that µ (cid:48)
Revised: If P ( w ) satisﬁes PL condition, i.e. there exists µ (cid:48) > 0 , such that µ (cid:48)

Original: ( P ( w ) − min w P ( w )) ≤ 12 (cid:107)∇ w P ( w ) (cid:107) 2 , then we have µ ( φ ( v ) − φ ( v ∗ )) ≤ 12 (cid:107)∇ φ ( v ) (cid:107) 2 , where µ =
Revised: ( P ( w ) − min w P ( w )) ≤ 12 (cid:107)∇ w P ( w ) (cid:107) 2 , then we have µ ( φ ( v ) − φ ( v ∗ )) ≤ 12 (cid:107)∇ φ ( v ) (cid:107) 2 , where µ =

Original: [Equation]
Revised: [Equation]

Original: Remark : The PL condition of P ( w ) could be proved for learning a neural network similar to existing studies, which is not the main focus of this paper.
Revised: Remark : The PL condition of P ( w ) could be proved for learning a neural network similar to existing studies, which is not the main focus of this paper.

Original: Nevertheless, In Appendix A.7, we provide an example for AUC maximization with one-hidden layer neural network.
Revised: Nevertheless, In Appendix A.7, we provide an example for AUC maximization with one-hidden layer neural network.

Original: Warmup.
Revised: Warmup.

Original: We ﬁrst discuss the algorithms and their convergence results of (Raﬁque et al., 2018) applied to the considered min-max problem.
Revised: We ﬁrst discuss the algorithms and their convergence results of (Raﬁque et al., 2018) applied to the considered min-max problem.

Original: They have algorithms for problems in batch-learning setting and online learning setting.
Revised: They have algorithms for problems in batch-learning setting and online learning setting.

Original: Since the algorithms for the batch-learning setting have complexities scaling with n , we will concentrate on the algorithm for the online learning setting.
Revised: Since the algorithms for the batch-learning setting have complexities scaling with n , we will concentrate on the algorithm for the online learning setting.

Original: The algorithm is presented in Algorithm 1, which is a direct application of Algorithm 2 of (Raﬁque et al., 2018) to an online setting.
Revised: The algorithm is presented in Algorithm 1, which is a direct application of Algorithm 2 of (Raﬁque et al., 2018) to an online setting.

Original: Since their analysis requires the domain of the primal and the dual variable to be bounded, hence we add a ball constraint on the primal variable and the dual variable as well.
Revised: Since their analysis requires the domain of the primal and the dual variable to be bounded, hence we add a ball constraint on the primal variable and the dual variable as well.

Original: As long as R 1 and R 2 is sufﬁciently large, they should not affect the solution.
Revised: As long as R 1 and R 2 is sufﬁciently large, they should not affect the solution.

Original: The convergence result of
Revised: The convergence result of

Original: Algorithm 1 is stated below.
Revised: Algorithm 1 is stated below.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Raﬁque et al., 2018)
Revised: (Raﬁque et al., 2018)

Original: Suppose f ( v , α ) is ρ -weakly convex in v and concave in α .
Revised: Suppose f ( v , α ) is ρ -weakly convex in v and concave in α .

Original: Let γ = 1 / 2 ρ , and deﬁne ˆ v τ = arg min v φ ( v )
Revised: Let γ = 1 / 2 ρ , and deﬁne ˆ v τ = arg min v φ ( v )

Original: +
Revised: +

Original: 12 γ (cid:107) v − ¯ v τ (cid:107) 2 .
Revised: 12 γ (cid:107) v − ¯ v τ (cid:107) 2 .

Original: Algorithm 1 with T k
Revised: Algorithm 1 with T k

Original: = ckand K
Revised: = ckand K

Original: = (cid:101) O ( (cid:15) − 2 ) ensures that E (cid:2) dist 2 (0 , ∂φ (ˆ v τ ))
Revised: = (cid:101) O ( (cid:15) − 2 ) ensures that E (cid:2) dist 2 (0 , ∂φ (ˆ v τ ))

Original: (cid:3) ≤ 1 γ 2 E (cid:107) ˆ v τ − ¯ v τ (cid:107) 2 ≤ (cid:15) 2 .
Revised: (cid:3) ≤ 1 γ 2 E (cid:107) ˆ v τ − ¯ v τ (cid:107) 2 ≤ (cid:15) 2 .

Original: The total iteration complexity is (cid:101) O
Revised: The total iteration complexity is (cid:101) O

Original: ( (cid:15) − 6 ) .
Revised: ( (cid:15) − 6 ) .

Original: Remark: Under the condition φ ( v ) is smooth and the returned solution is within the added bounded ball constraint, the above result implies E [ (cid:107)∇ φ (¯ v τ ) (cid:107) 2 ≤ (cid:15) ] with a complexity of (cid:101)
Revised: Remark: Under the condition φ ( v ) is smooth and the returned solution is within the added bounded ball constraint, the above result implies E [ (cid:107)∇ φ (¯ v τ ) (cid:107) 2 ≤ (cid:15) ] with a complexity of (cid:101)

Original: O
Revised: O

Original: (1 /(cid:15) 3 ) .
Revised: (1 /(cid:15) 3 ) .

Original: It further implies that with a complexity of (cid:101) O
Revised: It further implies that with a complexity of (cid:101) O

Original: (1 / ( µ 3 (cid:15) 3 ))
Revised: (1 / ( µ 3 (cid:15) 3 ))

Original: we have E [ φ (¯ v τ ) − min v φ ( v )] ≤ (cid:15) under the assumed PL condition.
Revised: we have E [ φ (¯ v τ ) − min v φ ( v )] ≤ (cid:15) under the assumed PL condition.

Original: We can see that this complexity result under the PL condition of φ ( v ) is worse than the typical complexity result of stochastic gradient descent method under the PL condition (i.e., O (1 /(cid:15) ) )
Revised: We can see that this complexity result under the PL condition of φ ( v ) is worse than the typical complexity result of stochastic gradient descent method under the PL condition (i.e., O (1 /(cid:15) ) )

Original: (Karimi et al., 2016).
Revised: (Karimi et al., 2016).

Original: It remains an open problem how to design a stochastic primal-dual algorithm for solving min v max α F ( v , α ) in order to achieve a complexity of O (1 /(cid:15) ) in terms of minimizing φ ( v ) .
Revised: It remains an open problem how to design a stochastic primal-dual algorithm for solving min v max α F ( v , α ) in order to achieve a complexity of O (1 /(cid:15) ) in terms of minimizing φ ( v ) .

Original: A naive idea is to solve the inner maximization problem of α ﬁrst and the use SGD on the primal variable v .
Revised: A naive idea is to solve the inner maximization problem of α ﬁrst and the use SGD on the primal variable v .

Original: However, this is not viable since exact maximization over α is a non-trivial task.
Revised: However, this is not viable since exact maximization over α is a non-trivial task.

Original:  A LGORITHMS AND T HEORETICAL A NALYSIS
Revised:  A LGORITHMS AND T HEORETICAL A NALYSIS

Original: In this section, we present two primal-dual algorithms for solving the min-max optimization problem (2) with corresponding theoretical convergence results.
Revised: In this section, we present two primal-dual algorithms for solving the min-max optimization problem (2) with corresponding theoretical convergence results.

Original: For simplicity, we ﬁrst assume the positive ratio p is known in advance, which is true in the batch-learning setting.
Revised: For simplicity, we ﬁrst assume the positive ratio p is known in advance, which is true in the batch-learning setting.

Original: Handling the unknown p in an online learning setting is a simple extension, which will be discussed in Section 4.3.
Revised: Handling the unknown p in an online learning setting is a simple extension, which will be discussed in Section 4.3.

Original: The proposed algorithms follow the same proximal point framework proposed in (Raﬁque et al., 2018), i.e., wesolve the following convex-concave problems approximately and iteratively: [Equation]
Revised: The proposed algorithms follow the same proximal point framework proposed in (Raﬁque et al., 2018), i.e., we

Original: where γ < 1 /L to ensure that the new objective function becomes convex and concave, and v 0 is periodically updated.

Original: 4. P ROXIMAL P RIMAL -D UAL S TOCHASTIC G RADIENT

Original: Our ﬁrst algorithm named Proximal Primal-Dual Stochastic Gradient (PPD-SG) is presented in

Original: Algorithm 2.

Original: Similar to Algorithm 1, it has a nested loop, where the inner loop is to approximately solve a regularized min-max optimization problem (3) using stochastic primal-dual gradient method, and the outer loop updates the reference point and learning rate.

Original: One key difference is that PPD-SG uses a geometrically decaying step size scheme, while Algorithm 1 uses a polynomially decaying step size scheme.

Original: Another key difference is that at the end of k -th outer loop, we update the dual variable ¯ α k in Step 12, which is motivated by its closed-form solution given ¯ v k .

Original: In particular, the

Original: 1: Initialize ¯ v 0 = 0 ∈ R d
Revised: 1: Initialize ¯ v 0 = 0 ∈ R d

Original: +2 , ¯ α 0 = 0 , the global index j =2: for k = 1 , . .
Revised: +2 , ¯ α 0 = 0 , the global index j =2: for k = 1 , . .

Original: , K do 3: v k 0 =
Revised: , K do 3: v k 0 =

Original: ¯ v k − 1 , α k 

Revised: ¯ v k − 1 , α k

Original: = ¯
Revised: = ¯

Original: α k − 1 , η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) 4: for t = 1 , . . .
Revised: α k − 1 , η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) 4: for t = 1 , . . .

Original: , T k do 5:
Revised: , T k do 5:

Original: Receive z j = ( x j , y j ) from P , ˆ g v = ∇ v F ( v kt − 1 , α kt − 1 ; z j ) , ˆ g α = ∇ α F ( v kt − 1 , α kt − 1 ; z j ) 6: v kt = v kt − 1 − η k (cid:16) ˆ g v + 1 γ ( v kt − 1 − v k 0 ) (cid:17) 7: α kt = α kt − 1 + η k ˆ g α 8: j = j +9: end for 10: ¯ v k = 1 T k (cid:80) T k t =1 v kt 11:
Revised: Receive z j = ( x j , y j ) from P , ˆ g v = ∇ v F ( v kt − 1 , α kt − 1 ; z j ) , ˆ g α = ∇ α F ( v kt − 1 , α kt − 1 ; z j ) 6: v kt = v kt − 1 − η k (cid:16) ˆ g v + 1 γ ( v kt − 1 − v k 0 ) (cid:17) 7: α kt = α kt − 1 + η k ˆ g α 8: j = j +9: end for 10: ¯ v k = 1 T k (cid:80) T k t =1 v kt 11:

Original: Draw a minibatch { z j , . . .
Revised: Draw a minibatch { z j , . . .

Original: , z j + m k − 1 } of size m k 12: ¯ α k = (cid:80) j + mk − 1 i =
Revised: , z j + m k − 1 } of size m k 12: ¯ α k = (cid:80) j + mk − 1 i =

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: i =

Original: j

Original: I
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: i =

Original: [Equation] − [Equation]
Revised: j

Original: mk
Revised: I yj =1 13: j = j + m k 14: end for 15: return ¯ v K , ¯ α K solve the following convex-concave problems approximately and iteratively:

Original: [Equation]
Revised: [Equation]

Original:  1 i = j [Equation] [Equation] =
Revised: where γ < 1 /L to ensure that the new objective function becomes convex and concave, and v 0 is periodically updated.

Original: [Equation] mk −i = j
Revised: 4. P ROXIMAL P RIMAL -D UAL S TOCHASTIC G RADIENT

Original: I yj =1 13: j = j + m k 14: end for 15: return ¯ v K , ¯ α K given ¯ v k , the dual solution that optimizes the inner maximization problem is given by:
Revised: Our ﬁrst algorithm named Proximal Primal-Dual Stochastic Gradient (PPD-SG) is presented in Algorithm 2.

Original: [Equation]
Revised: Similar to Algorithm 1, it has a nested loop, where the inner loop is to approximately solve a regularized min-max optimization problem (3) using stochastic primal-dual gradient method, and the outer loop updates the reference point and learning rate. One key difference is that PPD-SG uses a geometrically decaying step size scheme, while Algorithm 1 uses a polynomially decaying step size scheme. Another key difference is that at the end of k -th outer loop, we update the dual variable ¯ α k in Step 12, which is motivated by its closed-form solution given ¯ v k . In particular, the given ¯ v k , the dual solution that optimizes the inner maximization problem is given by:

Original: [Equation]
Revised: [Equation]

Original: In the algorithm, we only use a small number of samples in Step 11 to compute an estimation of the optimal α given ¯ v k .
Revised: In the algorithm, we only use a small number of samples in Step 11 to compute an estimation of the optimal α given ¯ v k .

Original: These differences are important for us to achieve lower iteration complexity of
Revised: These differences are important for us to achieve lower iteration complexity of

Original: PPD-SG.
Revised: PPD-SG.

Original: Next, we present our convergence results of PPD-SG.
Revised: Next, we present our convergence results of PPD-SG.

Original: Lemma 2 (One Epoch Analysis of Algorithm 2) .
Revised: Lemma 2 (One Epoch Analysis of Algorithm 2) .

Original: Suppose Assumption 1 and there exists G >such that (cid:107) ˆ g kt (cid:107) 2 ≤ G , where ˆ gkt =
Revised: Suppose Assumption 1 and there exists G >such that (cid:107) ˆ g kt (cid:107) 2 ≤ G , where ˆ g

Original: [Equation]
Revised: [Equation]

Original: ∇

Original: [Equation]
Revised: (cid:62) .

Original: Deﬁne φ k ( v ) =
Revised: Deﬁne φ k ( v ) =

Original: φ ( v ) + 12 γ (cid:107) v − ¯ v k − 1 (cid:107) 2 , s k = arg min v ∈ R d +2 φ k ( v ) .
Revised: φ ( v ) + 12 γ (cid:107) v − ¯ v k − 1 (cid:107) 2 , s k = arg min v ∈ R d +2 φ k ( v ) .

Original: Choosing m k ≥ [Equation]
Revised: Choosing m k ≥ σ 2 η 2 k

Original:  then we have
Revised: G 2 T k , then we have

Original: [Equation]
Revised: [Equation]

Original: where E k − 1 stands for the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.
Revised: where E k − 1 stands for the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the same conditions in Lemma 2 hold.
Revised: Suppose the same conditions in Lemma 2 hold.

Original: Set η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) and T k = max(2 , 8˜
Revised: Set η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) and T k = max(2 , 8 ˜ L 2 )

Original: L 2 ) Lηexp (cid:16) ( k − 1) µ/L 5+ µ/L (cid:17) , m k = σ 2 L G 2 η 0 max(2 , 8˜ L 2 ) exp (cid:16) ( k − 1) µ/L 5+ µ/L (cid:17) , γ = 12 L in Algorithm 2. To return ¯ v K such that E
Revised: Lη

Original: [-[ φ (¯ v K ) − φ ( v ∗ )] ≤ (cid:15) , it sufﬁces to choose K ≥ (cid:16) 
 Lµ + 1 (cid:17) max (cid:16) log 2∆ 0 (cid:15) , log K + log 24 G 2 η 0 5 (cid:15) (cid:17) .-]
Revised: [-[ φ (¯ v K ) − φ ( v ∗ )] ≤ (cid:15) , it sufﬁces to choose K ≥ (cid:16) 
 Lµ + 1 (cid:17) max (cid:16) log 2∆ 0 (cid:15) , log K + log 24 G 2 η 0 5 (cid:15) (cid:17) .-][Equation]

Original: As a result, the number of iterations is at most (cid:101) O (cid:16) LG 2 µ 2 (cid:15) (cid:17) , and the required number of samples is at most (cid:101) O (cid:16) L 3 σ 2 µ 2 (cid:15) (cid:17) , where (cid:101) O ( · ) hides logarithmic factor of L, µ, (cid:15), G, σ .
Revised: in Algorithm 2, where (cid:101) O ( · ) hides logarithmic factor of L, µ, (cid:15), G, σ .

Original: Remark :
Revised: Remark :

Original: The above complexity result is similar to that of (Karimi et al., 2016) for solving nonconvex minimization problem under the PL condition up to a logarithmic factor.
Revised: The above complexity result is similar to that of (Karimi et al., 2016) for solving nonconvex minimization problem under the PL condition up to a logarithmic factor.

Original: Compared with the complexity result of Algorithm 1 discussed earlier, i.e., (cid:101) O
Revised: Compared with the complexity result of Algorithm 1 discussed earlier, i.e., (cid:101) O

Original: (1 / ( µ 3 (cid:15) 3 )) , the above complexity in the order of (cid:101)
Revised: (1 / ( µ 3 (cid:15) 3 )) , the above complexity in the 1: for t = 1 , . . . , T k do [Table]

Original: O
Revised: Receive z j = ( x j , y j ) from P , ˆ g v = ∇ v F ( v kt , α kt ; z j ) , ˆ g α = ∇ α F ( v kt , α kt ; z j ) [Table] [Table] =

Revised: [Table]

Revised: ; −

Revised: [Table]

Revised: ∈

Revised: [Table]

Revised: [Table]

Revised: =

Revised: [Table]

Revised: [Table]

Revised: =

Revised: [Table]

Revised: H kt =

Revised: δI + diag ( s kt ) ,

Revised: [Table]

Revised: kt ( u ) = 12 (cid:104)

Revised: u − u k 0 , H kt ( u − u

Revised: [Table]

Original: (1 / ( µ 2 (cid:15) )) is much better - it not only improves the dependence on (cid:15) but also improves the dependence on µ .
Revised: 0 ) (cid:105) , where u k 0 = [ v k 0 ; α k 0 ] ∈ R d + [Table] : u kt +1 = arg min

Revised: [Table]

Revised: (cid:110)

Revised: η

Revised: [Table]

Revised: kt

Revised: [Table]

Revised: (cid:111)

Revised: [Table]

Revised: end

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: order of (cid:101) O

Revised: (1 / ( µ 2 (cid:15) )) is much better - it not only improves the dependence on (cid:15) but also improves the dependence on µ .

Original: 4. P ROXIMAL P RIMAL -D UAL A DAGRAD
Revised: 4. P ROXIMAL P RIMAL -D UAL A DAGRAD

Original: Our second algorithm named Proximal Primal-Dual Adagrad (PPD-Adagrad) is a AdaGrad-style algorithm.
Revised: Our second algorithm named Proximal Primal-Dual Adagrad (PPD-Adagrad) is a AdaGrad-style algorithm.

Original: Since it only differs from PPD-SG in the updates of the inner loop, we only present the inner loop in Algorithm 3.
Revised: Since it only differs from PPD-SG in the updates of the inner loop, we only present the inner loop in Algorithm 3.

Original: The updates in the inner loop are similar to the adaptive updates of traditional AdaGrad (Duchi et al., 2011).
Revised: The updates in the inner loop are similar to the adaptive updates of traditional AdaGrad (Duchi et al., 2011).

Original: We aim to achieve an adaptive convergence by using
Revised: We aim to achieve an adaptive convergence by using

Original: PPD-AdaGrad.
Revised: PPD-AdaGrad.

Original: The analysis of PPD-AdaGrad is inspired by the analysis of AdaGrad for non-convex minimization problems (Chen et al., 2019).
Revised: The analysis of PPD-AdaGrad is inspired by the analysis of AdaGrad for non-convex minimization problems (Chen et al., 2019).

Original: The key difference is that we have to carefully deal with

Original: 1: for t = 1 , . . .

Original: , T k do 2:

Original: Receive z j = ( x j , y j ) from P , ˆ g v = ∇ v F ( v kt , α kt ; z j ) , ˆ g α = ∇ α F ( v kt , α kt ; z j ) 3: ˆ g kt =

Original: [ˆ g v + 1 γ ( v kt − v k 0 ); − ˆ g α ] ∈ R d +3 , ˆ g k 1: t =

Original: [ˆ g k 1: t − 1 , ˆ g kt ] , s kt,i = (cid:13)(cid:13)

Original: ˆ g k 1: t,i (cid:13)(cid:13) 2 , 4:

Original: H kt =

Original: δI + diag ( s kt ) , ψ kt ( u ) =

Original: 12 (cid:104) u − u k 0 , H kt ( u − u k 0 ) (cid:105) , where u k 0 =

Original: [ v k 0 ; α k 0 ] ∈ R d + 5: u kt +1 = arg min u (cid:110) η k (cid:104) 1 t (cid:80) tτ

Original: =

Original: 1 ˆ g kτ , u (cid:105)

Original: + 1 t ψ kt ( u ) (cid:111) 6: end for

Original: : Update T −

Original: T − + (cid:80)

Original: + m −

Original: =

Original: j

Original: I [ y j = − 1] , T + = T + + (cid:80) j

Original: + m − 1 i =

Original: j

Original: I [ y j =1] 2: (cid:98) p

Original: = T + / ( T + + T − ) , ¯ y =

Original: ( j +2)¯ y + (cid:80) j

Original: + m − 1 i =

Original: j I [ yj =1] j + m + , (cid:92) [Equation] the primal-dual updates for the non-convex min-max problem.
Revised: The key difference is that we have to carefully deal with the primal-dual updates for the non-convex min-max problem.

Original: We summarize the convergence results of PPD-AdaGrad below.
Revised: We summarize the convergence results of PPD-AdaGrad below.

Original: Lemma 3 (One Epoch Analysis of Algorithm 3) .
Revised: Lemma 3 (One Epoch Analysis of Algorithm 3) .

Original: Suppose Assumption 1 and (cid:107) ˆ g kt (cid:107) ∞ ≤ δ hold.Deﬁne φ k ( v [Equation]
Revised: Suppose Assumption 1 and (cid:107) ˆ g kt (cid:107) ∞ ≤ δ hold.

Original: = [Equation] [Equation] = arg min v
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: d
Revised: arg

Original: [Equation]
Revised: [Equation]

Original: 2 φ k [Equation] v ) .
Revised: φ k ( v ) .

Original: Choosing m k ≥
Revised: Choosing m k ≥

Original: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original:   with M k > 0 , c > 0 , then we have
Revised:  with M k > 0 , c > 0 , then we have

Original: [Equation]
Revised: [Equation]

Original: where E k − 1 stands for the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.
Revised: where E k − 1 stands for the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.

Original: Theorem 3.
Revised: Theorem 3.

Original: Suppose the same conditions as in Lemma 3 hold.
Revised: Suppose the same conditions as in Lemma 3 hold.

Original: Set η k = η 0 exp (cid:16) − ( k − 1) 
Revised: Set η k = η 0 exp (cid:16) − ( k − 1) µ

Original: [Equation]
Revised: [Equation]

Original: L (cid:17) ,
Revised: L (cid:17) ,

Original: Mk = 4 cLη 0 exp (cid:16) ( k − 1) µ/L 5+ µ/L (cid:17) , m k
Revised: M

Original:  = σ 2 η 20 ( d +3) exp (cid:16) ( k − 1) µ/L 5+ µ/L (cid:17) , γ = 12 L and T k as in
Revised: [Equation] = [Equation] (cid:17) , γ = 12 L and T k as in

Original: Lemma 3 where c = 1 √ d +3 .
Revised: Lemma 3 where c = 1 √ d +3 .

Original: Suppose (cid:107) ˆ
Revised: Suppose (cid:107) ˆ

Original: g k 1: T k ,i (cid:107) 2 ≤ δ · T αk for ∀ k , where 0 ≤ α ≤ 12 .
Revised: g k 1: T k ,i (cid:107) 2 ≤ δ · T αk for ∀ k , where 0 ≤ α ≤ 12 .

Original: To return ¯ v K such that E
Revised: To return ¯ v K such that E

Original: [ φ (¯ v K ) − φ ( v ∗ )] ≤ (cid:15) , it sufﬁces to choose K ≥ (cid:16) 
 
Revised: [ φ (¯ v K ) − φ ( v ∗ )] ≤ (cid:15) , it sufﬁces to choose K ≥ (cid:16) 
 Lµ + 1 (cid:17) max [Equation] log

Original: Lµ + 1 (cid:17) max (cid:16) log 2∆ 0 (cid:15) , log K + log η 20 L 5 c 2 (cid:15) (cid:17) .
Revised: [Equation] (cid:15) , log K + log η [Equation] L 5 c 2 (cid:15) (cid:17) .

Original: The number of iterations is at most (cid:101) O (cid:18)(cid:16)
Revised: The number of iterations is at most (cid:101) O (cid:18)(cid:16)

Original: Lδ 2 d µ 2 (cid:15) (cid:17) 1 2(1 − α ) (cid:19) , and the required number of samples is at most (cid:101) O (cid:16) L 3 σ 2 µ 2 (cid:15) (cid:17) , where (cid:101) O ( · ) hides logarithmic factors of L, µ, (cid:15), δ .
Revised: Lδ 2 d µ 2 (cid:15) (cid:17) [Equation] ) (cid:19) , and the required number of samples is at most (cid:101) O (cid:16) L 3 σ 2 µ 2 (cid:15) (cid:17) , where (cid:101) O ( · ) hides logarithmic factors of L, µ, (cid:15), δ .

Original: Remark : When the cumulative growth of stochastic gradient is slow, i.e., α < 1 / 2 , the number of iterations is less than that in Theorem 2, which exhibits adaptive iteration complexity.
Revised: Remark : When the cumulative growth of stochastic gradient is slow, i.e., α < 1 / 2 , the number of iterations is less than that in Theorem 2, which exhibits adaptive iteration complexity.

Original: 4. E XTENSIONS
Revised: 4. E XTENSIONS

Original: Setting η k , T k , m k .
Revised: Setting η k , T k , m k .

Original: It is notable that the setting of η k , T k , m k depends on unknown parameters µ , L , etc., which are typically unknown.
Revised: It is notable that the setting of η k , T k , m k depends on unknown parameters µ , L , etc., which are typically unknown.

Original: One heuristic to address this issue is that we can decrease η k by a constant factor larger than 1 (e.g., 2 or 5 or 10), and similarly increase T k and m k by a constant factor.
Revised: One heuristic to address this issue is that we can decrease η k by a constant factor larger than 1 (e.g., 2 or 5 or 10), and similarly increase T k and m k by a constant factor.

Original: Another heuristic is to decrease the step size by a constant factor when the performance on a validation data saturates (Krizhevsky et al., 2012).
Revised: Another heuristic is to decrease the step size by a constant factor when the performance on a validation data saturates (Krizhevsky et al., 2012).

Original: Variants when p is unknown .
Revised: Variants when p is unknown .

Original: In the online learning setting when p is unknown, the stochastic gradients of f in both v and α are not directly available.
Revised: In the online learning setting when p is unknown, the stochastic gradients of f in both v and α are not directly available.

Original: To address this issue, we can keep unbiased estimators for both p and p (1 − p ) which are independent of the new arrived data, and update these estimators during the optimization procedure.
Revised: To address this issue, we can keep unbiased estimators for both p and p (1 − p ) which are independent of the new arrived data, and update these estimators during the optimization procedure.

Original: All values depending on p and p (1 − p ) (i.e., F, g v , g α ) are estimated by substituting p and p (1 − p ) by (cid:98) p and (cid:92) p (1 − p ) (i.e., ˆ F, ˆ g v , ˆ g α ) respectively.
Revised: All values depending on p and p (1 − p ) (i.e., F, g v , g α ) are estimated by substituting p and p (1 − p ) by (cid:98) p and (cid:92) p (1 − p ) (i.e., ˆ F, ˆ g v , ˆ g α ) respectively.

Original: The approach for keeping unbiased estimator ˆ p and (cid:92) p (1 − p ) during the optimization is described in
Revised: The approach for keeping unbiased estimator ˆ p and (cid:92) p (1 − p ) during the optimization is described in

Original: Algorithm 4, where j is the global index, and m is the number of examples received.
Revised: Algorithm 4, where j is the global index, and m is the number of examples received.

Original: Extensions to multi-class problems.
Revised: Extensions to multi-class problems.

Original: In the previous analysis, we only consider the binary classiﬁcation problem.
Revised: In the previous analysis, we only consider the binary classiﬁcation problem.

Original: We can extend it to the multi-class setting.
Revised: We can extend it to the multi-class setting.

Original: To this end, we ﬁrst introduce the deﬁnition of AUC in this setting according to (Hand & Till, 2001).
Revised: To this end, we ﬁrst introduce the deﬁnition of AUC in this setting according to (Hand & Till, 2001).

Original: Suppose there are c classes, we have c scoring functions for each class, namely h ( w 1 ; x ) , . .
Revised: Suppose there are c classes, we have c scoring functions for each class, namely h ( w 1 ; x ) , . .

Original: , h ( w c ; x ) .
Revised: , h ( w c ; x ) .

Original: We assume that these scores are normalized such that (cid:80) ck
Revised: We assume that these scores are normalized such that (cid:80) ck

Original: =1 h ( w c ; x ) = 1 .
Revised: =1 h ( w c ; x ) = 1 .

Original: Note that if these functions are implemented by a deep neural network, they can share the lower layers and have individual last layer of connections.
Revised: Note that if these functions are implemented by a deep neural network, they can share the lower layers and have individual last layer of connections.

Original: The
Revised: The

Original: AUC is deﬁned as
Revised: AUC is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Similar to Proposition 1, we can cast the problem into
Revised: Similar to Proposition 1, we can cast the problem into

Original: [Equation]
Revised: [Equation]

Original:  where w = [ w 1 , . . 
Revised: (5) where w = [ w 1 , . . .

Original:  w c ] , a , b , α ∈ R c × c , i, j = 1 , . . .
Revised: , w c ] , a , b , α ∈ R c × c , i, j = 1 , . . .

Original: , c , z
Revised: , c , z

Original: = ( x , y ) ∼ P , p i =
Revised: = ( x , y ) ∼ P , p i =

Original: Pr ( y = i ) , and
Revised: Pr ( y = i ) , and

Original: [Equation] ij 
Revised: F ij ( w i , a ij , b ij , α ij , z ) = p j

Original: [Equation]
Revised: [Equation]

Original: b
Revised: h

Original: [Equation]
Revised: [Equation]

Original: Then we can modify our algorithms to accommodate the multiple class pairs.
Revised: Then we can modify our algorithms to accommodate the multiple class pairs.

Original: We can also add another level of sampling of class pairs into computing the stochastic gradients.
Revised: We can also add another level of sampling of class pairs into computing the stochastic gradients.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section [Figure] we present some empirical results to verify the effectiveness of the proposed algorithms [Figure]
Revised: In this section, we present some empirical results to verify the effectiveness of the proposed algorithms.

Original: We compare our algorithms (PPD-SG and PPD-AdaGrad) with three baseline methods including
Revised: We compare our algorithms (PPD-SG and PPD-AdaGrad) with three baseline methods including

Original: PGA (Algorithm 1), Online AUC method (Ying et al., 2016) (OAUC) that directly employs the standard primal-dual stochastic gradient method with a decreasing step size for solving the min-max formulation [Figure] and the standard stochastic gradient descent (SGD) for minimizing cross-entropy loss.
Revised: PGA (Algorithm 1), Online AUC method (Ying et al., 2016) (OAUC) that directly employs the standard primal-dual stochastic gradient method with a decreasing step size for solving the min-max formulation, and the standard stochastic gradient descent (SGD) for minimizing cross-entropy loss.

Original: Comparing with PGA and OAUC allows us to verify the effectiveness of the proposed algorithms for solving the same formulation [Figure] and comparing with SGD allows us to verify the effectiveness of maximizing AUC for imbalanced data.
Revised: Comparing with PGA and OAUC allows us to verify the effectiveness of the proposed algorithms for solving the same formulation, and comparing with SGD allows us to verify the effectiveness of maximizing AUC for imbalanced data.

Original: We use a residual network with 20 layers (ResNet-20) to implement the deep neural network 
Revised: We use a residual network with 20 layers (ResNet-20) to implement the deep neural network for all algorithms.

Original: We use the stagewise step size strategy as in (He et al., 2016) for SGD, i.e. the step size is decreased by 10 times at 40K, 
Revised: We use the stagewise step size strategy as in (He et al., 2016) for SGD, i.e. the step size is decreased by 10 times at 40K, 60K.

Original: 60K. We use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on C2, C10,

Original: C100, and STL10 respectively.

Original: For PPD-SG and PPD-AdaGrad, we set T s = T 0 3 k , η s = η 0 / 3 k .
Revised: For PPD-SG and PPD-AdaGrad, we set T s = T 0 3 k , η s = η 0 / 3 k .

Original: T 0 , η 0 are tuned on a validation data.
Revised: T 0 , η 0 are tuned on a validation data.

Original: The value of γ is tuned for PGA and the same value is used for PPD-SG and PPD-AdaGrad.
Revised: The value of γ is tuned for PGA and the same value is used for PPD-SG and PPD-AdaGrad.

Original: The initial step size is tuned in [0.1, 0.05, 0.01, 0.008, 0.005] and T 0 is tuned in [200 ∼ 2000] for each algorithm separately.
Revised: The initial step size is tuned in [0.1, 0.05, 0.01, 0.008, 0.005] and T 0 is tuned in [200 ∼ 2000] for each algorithm separately.

Original: The batch size is set to 128.
Revised: The batch size is set to 128.

Original: For STL10, we use a smaller batch size 32 due to the limited training data.
Revised: For STL10, we use a smaller batch size 32 due to the limited training data.

Original: We conduct the comparisons on four benchmark datasets, i.e., Cat&Dog (C2), CIFAR10 (C10),
Revised: We conduct the comparisons on four benchmark datasets, i.e., Cat&Dog (C2), CIFAR10 (C10),

Original: CIFAR100 (C100), STL10.
Revised: CIFAR100 (C100), STL10.

Original: STL10 is an extension of CIFAR10 and the images are acquired from ImageNet.
Revised: STL10 is an extension of CIFAR10 and the images are acquired from ImageNet.

Original: Cat&Dog is from Kaggle containing 25,000 images of dogs and cats and we choose an 80:20 split to construct training and testing set.
Revised: Cat&Dog is from Kaggle containing 25,000 images of dogs and cats and we choose an 80:20 split to construct training and testing set.

Revised: We use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on C2, C10, C100, and STL10 respectively.

Original: For each data sets, we construct multiple binary classiﬁcation tasks with varying imbalanced ratio of number negative examples to number of positive examples.
Revised: For each dataset, we construct multiple binary classiﬁcation tasks with varying imbalanced ratio of number negative examples to number of positive examples.

Original: For details of construction of binary classiﬁcation tasks, please refer to the Appendix A.8.
Revised: For details of construction of binary classiﬁcation tasks, please refer to the Appendix A.8.

Original: We report the convergence of AUC on testing data in Figure 1, where the title shows the ratio of the majority class to the minority class.
Revised: We report the convergence of AUC on testing data in Figure 1, where the title shows the ratio of the majority class to the minority class.

Original: The results about the convergence of AUC versus the time in seconds are also presented in Figure 3.
Revised: The results about the convergence of AUC versus the time in seconds are also presented in Figure 3.

Original: From the results we can see that for the balanced settings with ratio equal to 50%, SGD performs consistently better than other methods on C2 and CIFAR data.
Revised: From the results we can see that for the balanced settings with ratio equal to 50%, SGD performs consistently better than other methods on C2 and CIFAR data.

Original: However, it is worse than AUC optimization based methods on CIFAR100 and STL10.
Revised: However, it is worse than AUC optimization based methods on CIFAR100 and STL10.

Original: For imbalanced settings, AUC maximization based methods are more advantageous than SGD in most cases.
Revised: For imbalanced settings, AUC maximization based methods are more advantageous than SGD in most cases.

Original: In addition, PPD-SG and PPD-AdaGrad are mostly better than other baseline algorithms.
Revised: In addition, PPD-SG and PPD-AdaGrad are mostly better than other baseline algorithms.

Original: In certain cases, PPD-AdaGrad can be faster than PPD-SG.
Revised: In certain cases, PPD-AdaGrad can be faster than PPD-SG.

Original: Finally, we observe even better performance (in Appendix) by a mixed strategy that pre-trains the model with SGD and then switchs to PPD-SG.
Revised: Finally, we observe even better performance (in Appendix) by a mixed strategy that pre-trains the model with SGD and then switchs to PPD-SG.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we consider stochastic AUC maximization problem when the predictive model is a deep neural network.
Revised: In this paper, we consider stochastic AUC maximization problem when the predictive model is a deep neural network.

Original: By abuilding on the saddle point reformulation and exploring Polyak-Łojasiewicz condition in deep learning, we have proposed two algorithms with state-of-the-art complexities for stochastic AUC maximization problem.
Revised: By abuilding on the saddle point reformulation and exploring Polyak-Łojasiewicz condition in deep learning, we have proposed two algorithms with state-of-the-art complexities for stochastic AUC maximization problem.

Original: We have also demonstrated the efﬁciency of our proposed algorithms on several benchmark datasets, and the experimental results indicate that our algorithms converge faster than other baselines.
Revised: We have also demonstrated the efﬁciency of our proposed algorithms on several benchmark datasets, and the experimental results indicate that our algorithms converge faster than other baselines.

Original: One may consider to extend the analysis techniques to other problems with the min-max formulation
Revised: One may consider to extend the analysis techniques to other problems with the min-max formulation.

Revised: A CKNOWLEDGEMENTS

Revised: The authors thank the anonymous reviewers for their helpful comments.

Revised: M. Liu, Z. Yuan and T. Yang are partially supported by National Science Foundation (IIS-1545995)

Original: A A PPENDIX A. P ROOF OF P ROPOSITION
Revised: A A PPENDIX A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: It sufﬁces to prove that
Revised: It sufﬁces to prove that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: | y (cid:48) [Equation] [Equation] ) =
Revised: p

Original: [Equation]
Revised: [Equation]

Original: + E (cid:2) h 2 ( w ; x ) (cid:12)(cid:12)

Original: y

Original: = 1 (cid:3) − ( E [ h ( w ; x ) | y = 1]) 2 + E (cid:2)

Original: h 2 ( w ; x (cid:48) ) (cid:12)(cid:12) y (cid:48) = − 1 (cid:3) − ( E [ h ( w ; x (cid:48) ) | y (cid:48)
Revised: RHS

Original: = − 1])− 2 E

Original: [ h ( w

Original: [Equation]
Revised: [Equation]

Revised: Note that the optimal values of a, b, α are chosen as a ∗ =

Original:  x ) | y = 1] + 2 E [ h ( w [Equation]
Revised: E [ h ( w ; x ) | y = 1] , b =

Original:  x (cid:48) )| y (cid:48)
Revised: E [ h ( w ; x (cid:48) )

Original: = − 1]

Original: + ( E [ h ( w ; x ) | y

Original: = 1] − E [ h ( w ; x (cid:48) )

Original: | y (cid:48)
Revised: | y (cid:48)

Original: = − 1])= 1 + min ( a,b ) ∈ R 2 E (cid:2) ( h ( w ; x ) − a ) 2 (cid:12)(cid:12) y = 1 (cid:3) + E (cid:2) ( h ( w ; x (cid:48) ) − b ) 2 (cid:12)(cid:12) y (cid:48) =− 1 (cid:3) − 2 E
Revised: = − 1] , α ∗ =

Original: [ h ( w ; x ) | y

Original: = 1] + 2 E

Original: [ h ( w ; x (cid:48) ) | y (cid:48)

Original: [-= − 1] + max α ∈ R (cid:2) 
-]
Revised: [-= − 1] + max α ∈ R (cid:2) 
-]

Original: α ( E [ h ( w [Equation] x (cid:48) ) | y (cid:48)
Revised: E [ h ( w ; x (cid:48) ) | y (cid:48)

Original: = − 1] − E [ h ( w ; x ) | y = [-1]) − α 2 (cid:3) = 
-]
Revised: = − 1] − E [ h ( w ; x ) | y = [-1]) − α 2 (cid:3) = 
-]1] .

Original: + min ( a,b ) ∈ R 2 max α ∈ R E z (cid:26)p ( h [Equation] x )
Revised: A. P ROOF OF L EMMA

Original: [Equation]

Original: [Equation]

Original: I [

Original: [Equation]

Original: =1] +

Original: [Equation]

Original: p ( h

Original: [Equation]
Revised: [Equation]

Original: x )

Original: [Equation] b ) [Equation] I [
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: = − 1] +2 (1 + α ) (cid:18)1 − ph ( [Equation] x ) I [ y = − 1] − 1 ph ( w [Equation]
Revised: [Table]

Original: x )

Original: [Equation]
Revised: [Equation]

Original: Note that the optimal values of a, b, α are chosen as a ∗ [Equation] A. P ROOF OF L EMMA Proof
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original:  where (a) comes from the deﬁnition of φ k , (b) holds becausemin
Revised: T k , where (a) comes from the deﬁnition of φ k , (b) holds because

Original: [Equation]
Revised: [Equation]

Original: ) comes from the standard analysis of primal-dual stochastic gradient method.
Revised: comes from the standard analysis of primal-dual stochastic gradient method.

Original: Denote E k − 1 by taking the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.
Revised: Denote E k − 1 by taking the conditional expectation conditioning on all the stochastic events until ¯ v k − 1 is generated.

Original: Taking E k − 1 on both sides and noting that ˆ g kt is an unbiased estimator of g kt for ∀ t, k , we have
Revised: Taking E k − 1 on both sides and noting that ˆ g kt is an unbiased estimator of g kt for ∀ t, k , we have

Original: [Equation]
Revised: [Equation]

Original: By the update of ¯ α k − 1 , 2˜ L -Lipschitz continuity of E [ h ( w ; x ) | y = − 1] − E
Revised: By the update of ¯ α k − 1 , 2˜ L -Lipschitz continuity of E [ h ( w ; x ) | y = − 1] − E

Original: [ h ( w ; x ) | y
Revised: [ h ( w ; x ) | y

Original: = 1] , and noting that α ∗ ,
Revised: = 1] , and noting that α ∗ ,

Original: k = E [ h ( ¯ w k ; x ) | y
Revised: k = E [ h ( ¯ w k ; x ) | y

Original: = − 1] − E
Revised: = − 1] − E

Original: [ h ( ¯ w k ; x ) | y
Revised: [ h ( ¯ w k ; x ) | y

Original: = 1] , we have
Revised: = 1] , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] Taking m k ≥ σ 
Revised: Taking m k ≥ σ 2 η 2 k

Original: [Equation]

Original: k G 2 T k , then we have
Revised: G 2 T k , then we have

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Deﬁne φ k ( v ) =
Revised: Deﬁne φ k ( v ) =

Original: φ ( v )+ 12 γ (cid:107) v − ¯ v k − 1 (cid:107) 2 .
Revised: φ ( v )+ 12 γ (cid:107) v − ¯ v k − 1 (cid:107) 2 .

Original: We can see that φ k ( v ) is convex and smooth function since γ ≤ 1 /L .
Revised: We can see that φ k ( v ) is convex and smooth function since γ ≤ 1 /L .

Original: The smoothness parameter of φ k is ˆ L =
Revised: The smoothness parameter of φ k is ˆ L =

Original: L + γ − 1 .
Revised: L + γ − 1 .

Original: Deﬁne s k = arg min v ∈ R d +2 φ k ( v ) .
Revised: Deﬁne s k = arg min v ∈ R d +2 φ k ( v ) .

Original: According to Theorem 2.1.5 of (Nesterov, 2013), we have
Revised: According to Theorem 2.1.5 of (Nesterov [Equation] [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Combining (8) with Lemma 2 yields
Revised: Combining (8) with Lemma 2 yields

Original: [Equation]
Revised: [Equation]

Original: Note that φ k (¯ v ) is ( γ − 1 − L ) -strongly convex, and γ = 12 L , we have
Revised: Note that φ k (¯ v ) is ( γ − 1 − L ) -strongly convex [Equation] and γ = 12 L , we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in s k into Lemma 2 and combining (10) yield
Revised: Plugging in s k into Lemma 2 and combining (10) yield

Original: [Equation]
Revised: [Equation]

Original: By using η k T k L ≥ max(2 , 8˜ L 2 ) , rearranging the terms [Equation] and noting that φ k (¯ v k − 1 ) =
Revised: By using η k T k L ≥ max(2 , 8˜ L 2 ) , rearranging the terms, and noting that φ k (¯ v k − 1 ) =

Original: φ (¯ v k − 1 ) , we have
Revised: φ (¯ v k − 1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Combining (11) and (9) yields
Revised: Combining (11) and (9) yields

Original: [Equation]
Revised: [Equation]

Original: Taking expectation on both sides over all randomness until ¯ v k − 1 is generated and by the tower property [Equation] we have
Revised: Taking expectation on both sides over all randomness until ¯ v k − 1 is generated and by the tower property, we have

Original: [Equation]
Revised: [Equation]

Original: Note that φ ( v ) is L -smooth and hence is L -weakly convex, so we have
Revised: Note that φ ( v ) is L -smooth and hence is L -weakly convex, so we have

Original: [Equation]
Revised: [Equation]

Original: where (a) and (b) hold by the deﬁnition of φ k.
Revised: where (a) and (b) hold by the deﬁnition of φ k

Revised: [Equation]

Original: Rearranging the terms in (14) yields
Revised: Rearranging the terms in (14) yields

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (a) holds by using (cid:104) a , b (cid:105) ≤ 12 ( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , and (b) holds by the PL property of φ .
Revised: where (a) holds by using (cid:104) a , b (cid:105) ≤ 12 ( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , and (b) holds by the PL property of φ .

Original: Deﬁne ∆ k = φ (¯ v k ) − φ ( v ∗ ) .
Revised: Deﬁne ∆ k = φ (¯ v k ) − φ ( v ∗ ) .

Original: Combining (13) and (15), we can see that
Revised: Combining (13) and (15), we can see that

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:18) 1 − µ/L 5 + µ/L (cid:19) k E [∆ 0 ] + 12 5 G 2 k(cid:88) j = η j (cid:18) 1 −  µ/L 5 + µ/L (cid:19) k
Revised: By setting η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) , we have

Original: +1 − j . By setting η k = η 0 exp (cid:16) − ( k − 1) µ/L 5+ µ/L (cid:17) , we have E [∆ k ] ≤ (cid:18) 1 − µ/L 5 + µ/L (cid:19) k
Revised: [Equation]

Original:  E [∆ 0 ] + 12 5 G 2 ηk (cid:88)
Revised: To achieve E [∆ k ] ≤ (cid:15)

Original: [-j = exp (cid:18) − k µ/L 5 + µ/L (cid:19) ≤ exp (cid:18) − k µ/L 5 + µ/L (cid:19) ∆ 
 + 12 5 G 2 η 0 k exp (cid:18) − k µ/L 5 + µ/L (cid:19) .-]
Revised: [-j = exp (cid:18) − k µ/L 5 + µ/L (cid:19) ≤ exp (cid:18) − k µ/L 5 + µ/L (cid:19) ∆ 
 + 12 5 G 2 η 0 k exp (cid:18) − k µ/L 5 + µ/L (cid:19) .-][Equation]

Original: To achieve E [∆ k ] ≤ (cid:15) , it sufﬁces to let K satisfy exp[-(cid:16) − K µ/L 5+ µ/L (cid:17) ≤ min (cid:16) (cid:15) 2∆ 0 , 5 (cid:15) 24 KG 2 η 0 (cid:17) , i.e. K ≥ (cid:16) 
 Lµ + 1 (cid:17) max (cid:16) log 2∆ 0 (cid:15) , log K + log 24 G 2 η 0 5 (cid:15) (cid:17) .-]
Revised: it sufﬁces to let K satisfy exp[-(cid:16) − K µ/L 5+ µ/L (cid:17) ≤ min (cid:16) (cid:15) 2∆ 0 , 5 (cid:15) 24 KG 2 η 0 (cid:17) , i.e. K ≥ (cid:16) 
 Lµ + 1 (cid:17) max (cid:16) log 2∆ 0 (cid:15) , log K + log 24 G 2 η 0 5 (cid:15) (cid:17) .-]

Original:  Since η k T k L ≥ max(2 , 8˜ L 2 ), by the setting of η k , we set T k = max(2 , 8˜
Revised: [Equation] Since η k T k L ≥ max( [Equation] 8˜ L 2 )

Original: L 2 ) Lηexp (cid:16) ( k − 1) µ/L 5+ µ/L (cid:17) .
Revised: [Equation]

Original: Then the total iteration complexity is K (cid:88) k 
Revised: by the setting of η k [Equation]

Original: = T k ≤ max(2 , 8˜ L 2 )
Revised: we set T k = max(2 , 8˜ L 2 )

Original: Lη
Revised: Lη

Original: [Equation]
Revised: [Equation]

Original: The required number of samples is
Revised: (cid:16) ( k [Equation] [Equation] L

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF L EMMA
Revised: L (cid:17) . Then the total iteration complexity is

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: The required number of samples is

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: A. P ROOF OF L EMMA

Original: [Equation]
Revised: Proof.

Original: [Table] [Equation] [Table] [Equation]
Revised: Deﬁne α ∗ ,k = arg max α f

Original: [Table] [Equation] [Table] [Equation]
Revised: (¯

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original:  (16) where (a) comes from the deﬁnition of φ k , (b) holds because min v max α
Revised: = I + II (16) where (a) comes from the deﬁnition of φ k , (b) holds because min v max α

Original: [Equation]
Revised: [Equation]

Original:  Jensen’s inequality.
Revised: holds by Jensen’s inequality.

Original: Now we bound I and II separately.
Revised: Now we bound I and II separately.

Original: Deﬁne (cid:107) u (cid:107) H = √ u (cid:62) H u , ψ k 0 ( u ) = 0 , ψ k, ∗ T k to be the conjugate of 1 η k ψ kT k , which is ψ k, ∗ T k ( g ) =
Revised: Deﬁne (cid:107) u (cid:107) H = √ u (cid:62) H u , ψ k 0 ( u ) = 0 , ψ k, ∗ T k to be the conjugate of 1 η k ψ kT k , which is ψ k, ∗ T k ( g ) =

Original: sup u (cid:110) (cid:104) g , u (cid:105) − 1 η k ψ kT k ( u ) (cid:111) .
Revised: sup u (cid:110) (cid:104) g , u (cid:105) − 1 η k ψ kT k ( u ) (cid:111) .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where the last equality holds by the deﬁnition of ψ k, ∗ T k .
Revised: where the last equality holds by the deﬁnition of ψ k, ∗ T k .

Original: In addition, note that
Revised: In addition, note that

Original: [Equation]
Revised: [Equation]

Original: (18) where (a) holds due to the update of the Algorithm 3, (b) holds since ψ kt +1 ( u ) ≥ ψ kt ( u ) , (c) holds by the η k -smoothness of ψ k, ∗ t with respect to (cid:107) · (cid:107) ψ k, ∗ t = (cid:107) [Equation] kt [Equation]
Revised: where (a) holds due to the update of the Algorithm 3, (b) holds since ψ kt +1 ( u ) ≥ ψ kt ( u ) , (c) holds by the η k -smoothness of ψ k, ∗ t with respect to (cid:107) · (cid:107) ψ k, ∗ t = (cid:107) · (cid:107) ( H kt ) − 1 .

Original: − 1 . By (18) and noting that ∇ ψ k, ∗ T k − 1 (cid:16) − (cid:80) T k − 1 t  = [Equation]
Revised: By (18) and noting that ∇ ψ k, ∗ T k − 1 (cid:16) − (cid:80) T k − 1 t =1 ˆ g kt (cid:17) = u kT k , we have

Original: g kt (cid:17) = u kT k , we have T k 
Revised: T k (cid:88) t = (cid:10) ˆ

Original: [Equation]
Revised: [Equation]

Original: Using (19) recursively and noting that ψ k 0 ( u ) = 0 , we know that
Revised: Using (19) recursively and noting that ψ k 0 ( u ) = 0 , we know that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combining (17) and (20), we have
Revised: Combining (17) and (20), we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma 4 of (Duchi et al., 2011) and setting δ ≥ max t (cid:107) ˆ g kt (cid:107) ∞ , we know that (cid:80) T k t =1 (cid:107) ˆ g kt (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: k,

Original: [Equation]
Revised: [Equation]

Original: [Equation] 2 (cid:80) d +3 i =1 (cid:107)[Equation]
Revised: By Lemma 4 of (Duchi et al., 2011) and setting δ ≥ max t (cid:107) ˆ g kt (cid:107) ∞ , we know that (cid:80) T k t =1 (cid:107) ˆ g kt (cid:107) 2 ψ k, ∗ t − 1 ≤ 2 (cid:80) d +3 i =1 (cid:107)

Original:  g k 1: T k (cid:107) 2 , and henceI ≤
Revised: ˆ g k 1: T k (cid:107) 2 , and hence

Original: [Equation]
Revised: [Equation]

Original: Denote E k − 1 by taking the conditional expectation conditioning on ﬁltration F k − 1 , where F k − 1 is the σ -algebra generated by all random variables until ¯ v k − 1 is generated.
Revised: Denote E k − 1 by taking the conditional expectation conditioning on ﬁltration F k − 1 , where F k − 1 is the σ -algebra generated by all random variables until ¯ v k − 1 is generated.

Original: Taking E k − 1 on both sidesof (16), and employing (22) yields
Revised: Taking E k − 1 on both sides

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: where the equality holds since ¯ v k − 1 − s k is measurable with respect to F k − 1 . Note that

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: : T k

Original: [Equation]
Revised: [Equation]

Original: c
Revised: i (cid:107) [Equation] η k T k (cid:33)

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: m

Original: [Equation]
Revised: [Equation]

Original: where ( a ) holds because ¯ α k − 1 − α ∗ ,k − 1 and δ +max i (cid:107) ˆ g
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: Tk,i (cid:107)2 2 η k
Revised: max [Equation] (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (cid:33) where ( a ) holds because ¯ α k − 1 − α ∗ ,k − 1 and δ +max i (cid:107) ˆ g k 1:

Original:  k are independent conditioning on F k − 1 , (b) holds because of the update of ¯ α k − 1 and α ∗ ,
Revised: Tk,i (cid:107) 2 2 η k T k are independent conditioning on F k − 1 , (b) holds because of the update of ¯ α k − 1 and α ∗ ,

Original: k = E [ h ( ¯ w k ; x ) | y
Revised: k = E [ h ( ¯ w k ; x ) | y

Original: = − 1] − E
Revised: = − 1] − E

Original: [ h ( ¯ w k ; x ) | y
Revised: [ h ( ¯ w k ; x ) | y

Original: = 1] , (c) holds due to the 2˜ L -Lipschitz continuity of E [ h ( w ; x ) |
Revised: = 1] , (c) holds due to the 2˜ L -Lipschitz continuity of E [ h ( w ; x ) |

Original: y
Revised: y

Original: =
Revised: =

Original: − 1] − E
Revised: − 1] − E

Original: [ h ( w ; x ) | y
Revised: [ h ( w ; x ) | y

Original: = 1] .
Revised: = 1] .

Original: Taking m k ≥ σ 2 ( d +3)
Revised: Taking m k ≥ σ [Equation]

Original:  η 2 k , then we have
Revised: d +3) η 2 k , then we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: i = (cid:107) ˆ g k 1: T k ,i (cid:107)(cid:33) + E k − 1 ( II ) + E k −(cid:32)
Revised: (cid:32) [Equation] (cid:88) [Equation]

Original: δ + max i (cid:107) 
Revised: (cid:107) [Equation] g

Original: ˆ g k 1: T k ,i (cid:107)2 T k (cid:33) · 2 η k ( d + 3) + E k−(cid:32)
Revised: [Equation] 1: T [Equation] ,i (cid:107)(cid:33) + E k

Original:  δ + max i (cid:107)
Revised: [Equation] 1 ( II ) + E k [Equation] (cid:32) δ + max i (cid:107)

Revised: ˆ g k 1: T k ,i (cid:107)2 T k (cid:33) · 2 η k ( d + 3) + E k −(cid:32)

Revised: δ + max i (cid:107)

Original: ˆ g k 1: T k ,i (cid:107)

Revised: ˆ g k 1: T k ,i (cid:107)

Original: η k T k · 8˜ L 2 (cid:107) ¯ v k − 1 − ¯ v k (cid:107)(cid:33) =
Revised: η k T k · 8˜ L 2 (cid:107) ¯ v k − 1 − ¯ v k (cid:107)(cid:33) =

Original: E k −(cid:32)
Revised: E k −(cid:32)

Original: δ + max i (cid:107)
Revised: δ + max i (cid:107)

Original: ˆ g k 1: T k ,i (cid:107)2 η k T k (cid:33) (cid:107) ¯ v k − 1 − s k (cid:107) 2 + E k −(cid:34)
Revised: ˆ g k 1: T k ,i (cid:107)2 η k T k (cid:33) (cid:107) ¯ v k − 1 − s k (cid:107) 2 + E k −(cid:34)

Original: η k
Revised: η k

Original: T k (cid:32)
Revised: T k (cid:32)

Original: d +3 (cid:88) i =
Revised: d +3 (cid:88) i =

Original: (cid:107)
Revised: (cid:107)

Original: ˆ g k 1: T k ,i (cid:107) 2 + ( d + 3) (cid:16) δ
Revised: ˆ g k 1: T k ,i (cid:107) 2 + ( d + 3) (cid:16) δ

Original: + max i (cid:107) ˆ g k 1: T k ,i (cid:107)(cid:17)(cid:33)(cid:35)
Revised: + max i (cid:107) ˆ g k 1: T k ,i (cid:107)(cid:17)(cid:33)(cid:35)

Original: + E k − 1 ( II ) + E k −(cid:32)
Revised: + E k − 1 ( II ) + E k −(cid:32)

Original: δ + max i (cid:107)
Revised: δ + max i (cid:107)

Original: ˆ g k 1: T k ,i (cid:107)

Revised: ˆ g k 1: T k ,i (cid:107)

Original: η k T k · 8˜ L 2 (cid:107) ¯ v k − 1 − ¯ v k (cid:107)(cid:33)
Revised: η k T k · 8˜ L 2 (cid:107) ¯ v k − 1 − ¯ v k (cid:107)(cid:33)

Original: By setting
Revised: By setting

Original: T k = inf (cid:40) τ : τ ≥ M k max (cid:32) ( δ 
Revised: T k = inf (cid:40) τ : τ ≥ M k max (cid:32) ( δ + max

Original: [Equation] max i (cid:107) [Equation] g k 1: τ,i (cid:107) 2 ) max(
Revised: i (cid:107) ˆ g k 1: τ,i (cid:107) 2 ) max(1 , 8˜ L 2 ) 2 c , c (cid:32) d +3 (cid:88)

Original: [Equation] L 2 ) 2 c [Equation]
Revised: i =

Original: c (cid:32)
Revised: (cid:107)

Original: [Equation] 3 (cid:88) i
Revised: ˆ g k 1: τ,

Original: = (cid:107) {+
+}
Revised: i (cid:107) {+
+}

Original: [Equation]
Revised: + ( d + 3) (cid:16) δ

Original: (cid:33)(cid:33)(cid:41) , then T k is a stopping time which is bounded almost surely.
Revised: + max i (cid:107) ˆ g k 1: τ,i (cid:107)(cid:17)(cid:33)(cid:33)(cid:41) , then T k is a stopping time which is bounded almost surely.

Original: By stopping time argument, we have E k − 1 ( II ) = 0 , and hence
Revised: By stopping time argument, we have E k − 1 ( II ) = 0 , and hence

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Deﬁne φ k ( v ) = 
Revised: Deﬁne φ k ( v ) = φ

Revised: [Equation]

Revised: )+ 12 γ

Original: φ ( v )+ 12 γ (cid:107) v − ¯ v k − 1 (cid:107) 2 .
Revised: [Equation] − ¯ [Equation] k − 1 (cid:107) 2 .

Original: We can see that φ k ( v ) is convex and smooth function since γ ≤ 1 /L .
Revised: We can see that φ k ( v ) is convex and smooth function since γ ≤ 1 /L .

Original: The smoothness parameter of φ k is ˆ L =
Revised: The smoothness parameter of φ k is ˆ L =

Original: L + γ − 1 .
Revised: L + γ − 1 .

Original: Deﬁne s k = arg min v ∈ R d +2 φ k ( v ) .
Revised: Deﬁne s k = arg min v ∈ R d +2 φ k ( v ) .

Original: According to Theorem 2.1.5 of (Nesterov, 2013), we have
Revised: According to Theorem 2.1.5 of (Nesterov, 2013), we have

Original: [Equation]
Revised: [Equation]

Revised: Combining (24) with Lemma 3 yields

Original: [Equation]
Revised: [Equation]

Original: Note that φ k (¯ v ) is ( γ − 1 − L ) -strongly convex, and γ = 12 L , we have
Revised: Note that φ k (¯ v ) is ( γ − 1 − L ) -strongly convex, and γ = 12 L , we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in s k intoLemma 3 andcombining (26) yield
Revised: Plugging in s k into 3 and yield

Original: [Equation]
Revised: [Equation]

Original: By taking η k M k L ≥ 4 c , rearranging the terms, and noting that φ k (¯ v k − 1 ) =
Revised: By taking η k M k L ≥ 4 c [Equation] rearranging the terms [Equation]

Revised: and noting that φ k (¯ v k − 1 ) =

Original: φ (¯ v k − 1 ) , we have 
Revised: φ (¯ v k − 1 ) [Equation] we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining (27) and (25) yields
Revised: Combining (27) and (25) yields

Original: [Equation]
Revised: [Equation]

Original: Taking expectation on both sides over all randomness until ¯ v k − 1 is generated and by the tower property
Revised: Taking expectation on both sides over all randomness until ¯ v k − 1 is generated and by the tower property

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that φ ( v ) is L -smooth and hence is L -weakly convex, so we have
Revised: Note that φ ( v ) is L -smooth and hence is L -weakly convex, so we have

Original: [Equation]
Revised: [Equation]

Original: 30) where (a) and (b) hold by the deﬁnition of φ k.
Revised: where (a) and (b) hold by the deﬁnition of φ k

Revised: [Equation]

Original: Rearranging the terms in (30) yields
Revised: Rearranging the terms in (30) yields

Original: [Equation]
Revised: [Equation]

Original: 31) where (a) holds by using (cid:104) a , b (cid:105) ≤ [Equation] a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , and (b) holds by the PL property of φ .
Revised: (31) where (a) holds by using (cid:104) a , b (cid:105) ≤ 12 ( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , and (b) holds by the PL property of φ .

Original: Deﬁne ∆ k = φ (¯ v k ) − φ ( v ∗ ) .
Revised: Deﬁne ∆ k = φ (¯ v k ) − φ ( v ∗ ) .

Original: Combining (29) and (31), we can see that
Revised: Combining (29) and (31), we can see that

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: By setting η k = η
Revised: By setting η k = η

Original: [Equation]
Revised: [Equation]

Original: (cid:16) 
Revised: (cid:16) − (

Original: [Equation]
Revised: [Equation]

Original: stage
Revised: − 1)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: + µ/L

Original: [Equation]
Revised: [Equation]

Original: To achieve E [∆ K ] ≤ (cid:15)
Revised: (cid:17) at k -th stage, we have

Original: [Equation]
Revised: [Equation]

Original:  it sufﬁces to let K satisfy exp (cid:16) − K [Equation] L [Equation]
Revised: To achieve E [∆ K ] ≤ (cid:15) , it sufﬁces to let K satisfy exp (cid:16) − K µ/L 5+ µ/L (cid:17) ≤ min (cid:16) (cid:15) 2∆ 0 , 5 c 2 (cid:15)

Original: +
Revised: Kη 20 L (cid:17) , i.e.

Original: [Equation] ≤ [Equation] K ≥ 
Revised: K ≥ (cid:16) 5 Lµ +

Original: [Equation]
Revised: [Equation]

Original: Take c = 1 √ d +3 .
Revised: Take c = 1 √ d +3 .

Original: If (cid:107) ˆ g k 1: T k ,i (cid:107) 2 ≤ δ · T αk for ∀ k , where 0 ≤ α ≤ 12 , and note that when τ ≥ 1 , max
Revised: If (cid:107) ˆ g k 1: T k ,i (cid:107) 2 ≤ δ · T αk for ∀ k , where 0 ≤ α ≤ 12 , and note that when τ ≥ 1 , max

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:33)(cid:33) ≤(cid:104)
Revised: (cid:17)(cid:33)(cid:33) ≤

Original: [Equation]
Revised: [Equation]

Original: 8˜
Revised: so we have T k ≤ 4 cLη 0 exp (cid:16)

Original: [Equation]
Revised: [Equation]

Original: √
Revised: k − 1) µ

Original: [Equation]
Revised: [Equation]

Original: δ

Original: [Equation]
Revised: [Equation]

Original: α so we have T k ≤ 4 cLη 0 exp (cid:16) ( k − 1) µ [Equation] + 
Revised: (4 + 8˜ L

Original: [Equation]
Revised: [Equation]

Original: (cid:17) · (cid:104) (4 + 8˜ L 2 ) √ d + 3 (cid:105) δT αk , and hence
Revised: ) √ d + 3 (cid:105) δT αk , and hence

Original: T k ≤ (cid:18)4 δc Lη
Revised: T k ≤ (cid:18)

Original: [Equation]
Revised: [Equation]

Original: exp (cid:18)( k
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: 1) µ/L 5 + µ/L (cid:19) ·(cid:104)
Revised: k [Equation] [Equation] L (cid:19) ·

Original: [Equation]
Revised: [Equation]

Original: 4 + 8˜L
Revised: 4 + 8˜

Original: [Equation]
Revised: [Equation]

Original: √ d + 3 (cid:105)(cid:19) 1 1 − α .
Revised: √ d [Equation] − [Equation]

Original: Noting that c = 1 √ d +3 , we can see that the total iteration complexity is
Revised: Noting that c = 1 √ d +3 , we can see that the total iteration complexity is

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: T k ≤(cid:32)
Revised: T k ≤

Original: [Equation]
Revised: [Equation]

Original: δ (4 + 8˜ L 2 )

Original: Lη(cid:33) 1 1 − α · exp (cid:16) K µ/L (5+ µ/L )(
Revised: Lη(cid:33) 1 1 − α · exp [Equation] K [Equation]

Revised: L (

Original: [Equation]
Revised: [Equation]

Revised: µ/L )(2 − 2 α ) (cid:17) −

Revised: [Equation]

Original: The required number of samples is
Revised: The required number of samples is

Revised: [Equation]

Revised: [Equation]

Revised: exp (cid:16)

Original: [Equation]
Revised: [Equation]

Revised: (cid:17) −= (cid:101) O (cid:18)

Revised: L 3 σµ 2 (cid:15) (cid:19) .

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: For any ﬁxed w , deﬁne ( a ∗ w , b ∗ w ) =
Revised: For any ﬁxed w , deﬁne ( a ∗ w , b ∗ [Equation]

Original:  arg min a,b φ ( w , a, b ) ( φ ( w , a, b ) is strongly convex in terms of ( a, b ) , so the argmin is well-deﬁned and unique).
Revised: ) = arg min a,b φ ( w , a, b ) ( φ ( w , a, b ) is strongly convex in terms of ( a, b ) , so the argmin is well-deﬁned and unique).

Original: Note that φ ( v ) − φ ( v ∗ ) =
Revised: Note that [Equation]

Original:  φ ( w , a, b ) −min w ,a,b φ ( w , a, b ) =
Revised: We bound φ ( w , a, b ) − φ [Equation] w , a

Original: φ ( w , a, b ) − φ ( w , a ∗ w , b ∗ w ) + φ ( w , a ∗ w , b ∗ w ) − min w ,a,b φ ( w , a, b ) We bound φ ( w , a, b ) − φ ( w , a ∗ w , b ∗ w ) and φ ( w , a ∗ w , b ∗ w ) − min w ,a,b φ ( w , a, b ) respectively:
Revised: [Equation] , b [Equation]

Original: • Note that φ ( w , a, b ) is strong convex in ( a, b ) with modulus 2 min( p, 1 − p ) , so the PL condition holds, which means that
Revised: ) and φ [Equation] w , a

Original: [Equation]
Revised: [Equation]

Revised: , b

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since φ ( w , a, b ) is strongly convex in ( a, b ) with modulus 2 min( p, 1 − p ) . Combining these two cases
Revised: ) − min w ,a,b φ ( w , a, b ) respectively: • Note that φ ( w , a, b ) is strong convex in ( a, b ) with modulus 2 min( p, 1 − p ) , so the PL condition holds, which means that

Original: [Equation]
Revised: [Equation]

Revised: where the last inequality holds since φ ( w , a, b ) is strongly convex in ( a, b ) with modulus 2 min( p, 1 − p ) .

Original:  we know that φ ( v ) − φ ( v ∗ ) ≤ 12 µ (cid:48) (cid:107)∇ φ ( v ) (cid:107) 2 , where µ (cid:48) 
Revised: Combining these two cases, we know that φ ( v ) − φ ( v ∗ ) ≤ 12 µ (cid:48) (cid:107)∇ φ ( v ) (cid:107) 2 , where µ (cid:48) = max

Original: [Table]
Revised: [Equation]

Original: A. A N EXAMPLE THAT SATISFIES PL CONDITION
Revised: A. A N EXAMPLE THAT SATISFIES PL CONDITION

Original: One Hidden Layer Neural Network One hidden neural network satisﬁes h ( w ; x ) =
Revised: One Hidden Layer Neural Network One hidden neural network satisﬁes h ( w ; x ) =

Original: σ ( w (cid:62) x ) , where σ is the activation function.
Revised: σ ( w (cid:62) x ) , where σ is the activation function.

Original: We have the following theorem:
Revised: We have the following theorem:

Original: Theorem 4.
Revised: Theorem 4.

Original: Let σ be the Leaky ReLU activation function such that σ ( z )
Revised: Let σ be the Leaky ReLU activation function such that σ ( z )

Original: =
Revised: =

Original: c 1 z for z > 0 and σ ( z ) =
Revised: c 1 z for z > 0 and σ ( z ) =

Original: c 2 z if z ≤ 0 .
Revised: c 2 z if z ≤ 0 .

Original: If E [ x | y = 1] =
Revised: If E [ x | y = 1] =

Original: E [ x | y = − 1] = 0 , E (cid:2) xx (cid:48)(cid:62) (cid:12)(cid:12)
Revised: E [ x | y = − 1] = 0 , E (cid:2) xx (cid:48)(cid:62) (cid:12)(cid:12)

Original: y
Revised: y

Original: = 1 , y = − 1 (cid:3) = 

Revised: = 1 , y = − 1 (cid:3) =

Original: d × d , then f ( w ) :
Revised: d × d , then f ( w [Equation]

Original: = E z , z (cid:48) (cid:2)(1 − σ ( w (cid:62) x ) + σ ( w (cid:62) x (cid:48) )) 2 (cid:12)(cid:12) y
Revised: [Equation] (cid:48) (cid:2) ( [Equation] [Equation]

Revised: +

Revised: [Equation]

Revised: (cid:12)(cid:12) y

Original: = 1 , y (cid:48)
Revised: = 1 , y (cid:48)

Original: = − 1 (cid:3) satisﬁes PL condition with µ = 2 min( c 21 , c 22 ) (cid:2) λ min (cid:0) E (cid:2) xx (cid:62) (cid:12)(cid:12) y
Revised: = − 1 (cid:3) satisﬁes PL condition with µ = 2 min( c 21 , c 22 ) (cid:2) λ min (cid:0) E (cid:2) xx (cid:62) (cid:12)(cid:12) y

Original: = 1 (cid:3)(cid:1) + λ min (cid:0) E (cid:2) xx (cid:62) (cid:12)(cid:12)
Revised: = 1 (cid:3)(cid:1) + λ min (cid:0) E (cid:2) xx (cid:62) (cid:12)(cid:12)

Original: y = − 1 (cid:3)(cid:1)(cid:3) , where λ min stands for the minimum eigenvalue.
Revised: y = − 1 (cid:3)(cid:1)(cid:3) , where λ min stands for the minimum eigenvalue.

Original: Remark : Consider the case that x is a zero mean Gaussian distribution with non-degenerate convariance matrix.
Revised: Remark : Consider the case that x is a zero mean Gaussian distribution with non-degenerate convariance matrix.

Original: Then µ > 0 since the minimum eigenvalue appeared in the expression of µ is positive.
Revised: Then µ > 0 since the minimum eigenvalue appeared in the expression of µ is positive.

Original: Proof.
Revised: Proof.

Original: Deﬁne g 1 ( x ) =
Revised: Deﬁne g 1 ( x ) =

Original: (1 − x ) 2 , g 2 ( w )
Revised: (1 − x ) 2 , g 2 ( w )

Original: =
Revised: =

Original: σ ( w (cid:62) x ) − σ ( w (cid:62) x (cid:48) )
Revised: σ ( w (cid:62) x ) − σ ( w (cid:62) x (cid:48) )

Original: , F ( w ) =
Revised: , F ( w ) =

Original: (1 − σ ( w (cid:62) x )+ σ ( w (cid:62) x (cid:48) )) 2 .
Revised: (1 − σ ( w (cid:62) x )+ σ ( w (cid:62) x (cid:48) )) 2 .

Original: We know that f ( w ) =
Revised: We know that f ( w ) =

Original: E z , z (cid:48)
Revised: E z , z (cid:48)

Original: [ F ( w ) | y
Revised: [ F ( w ) | y

Original: = 1 , y (cid:48)
Revised: = 1 , y (cid:48)

Original: = − 1] , F ( w )
Revised: = − 1] , F ( w )

Original: =
Revised: =

Original: g 1 ( g 2 ( w )) .
Revised: g 1 ( g 2 ( w )) .

Original: For ﬁxed x , x (cid:48) , we can write σ ( w (cid:62) x ) and σ ( w (cid:62) x (cid:48) ) as a w (cid:62) x and b w (cid:62) x (cid:48) respectively, and it is obvious that a 2 ≥ min( c 21 , c 22 ) and b 2 ≥ min( c 21 , c 22 ) .
Revised: For ﬁxed x , x (cid:48) , we can write σ ( w (cid:62) x ) and σ ( w (cid:62) x (cid:48) ) as a w (cid:62) x and b w (cid:62) x (cid:48) respectively, and it is obvious that a 2 ≥ min( c 21 , c 22 ) and b 2 ≥ min( c 21 , c 22 ) .

Original: Note that g 1 is 2 -strongly convex.
Revised: Note that g 1 is 2 -strongly convex.

Original: Since the conditional expectation perserves the strong convexity, as a result, for ∀ w , let w p be the closest optimal point of w such that f ∗ =
Revised: Since the conditional expectation perserves the strong convexity, as a result, for ∀ w , let w p be the closest optimal point of w such that f ∗ =

Original: f ( w p ) , we have
Revised: f ( w p ) , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) holds since λ min ( A + B ) ≥ λ min ( A ) + λ min ( B )
Revised: where ( ∗ ) holds since λ min ( A + B ) ≥ λ min ( A ) + λ min ( B )

Original: [Equation]
Revised: [Table]

Original: and the last inequality holds since a 2 ≥ min( c 21 , c 22 ) and b 2 ≥ min( c 21 , c 22 ) .
Revised: and the last inequality holds since a 2 ≥ min( c 21 , c 22 ) and b 2 ≥ min( c 21 , c 22 ) .

Original: A. D ATASET P REPARATION
Revised: A. D ATASET P REPARATION

Original: We construct the datasets in the following ways: For CIFAR10/STL10, we label the ﬁrst 5 classes as negative ("-") class and the last 5 classes as positive ("+") class, which leads to a 50/50 class ratio.
Revised: We construct the datasets in the following ways: For CIFAR10/STL10, we label the ﬁrst 5 classes as negative ("-") class and the last 5 classes as positive ("+") class, which leads to a 50/50 class ratio.

Original: For
Revised: For

Original: CIFAR100, we label the ﬁrst 50 classes as negative ("-") class and the last 50 classes as positve ("+") class.
Revised: CIFAR100, we label the ﬁrst 50 classes as negative ("-") class and the last 50 classes as positve ("+") class.

Original: For the imbalanced cases, we randomly remove 90%, 80%, 60% data from negative samples on all training data, which lead to 91/9, 83/17, 71/29 ratio respectively.
Revised: For the imbalanced cases, we randomly remove 90%, 80%, 60% data from negative samples on all training data, which lead to 91/9, 83/17, 71/29 ratio respectively.

Original: For testing data, we keep them unchanged.
Revised: For testing data, we keep them unchanged.

Original: A. M ORE E XPERIMENTS
Revised: A. M ORE E XPERIMENTS

Original: Model pretraining is effective in many deep learning tasks, and thus we further evaluate the performance of the proposed methods on pretrained models.
Revised: Model pretraining is effective in many deep learning tasks, and thus we further evaluate the performance of the proposed methods on pretrained models.

Original: We ﬁrst train the model using SGD up toiterations with an initial step size of 0.1, and then continue training using PPD-SG.
Revised: We ﬁrst train the model using SGD up toiterations with an initial step size of 0.1, and then continue training using PPD-SG.

Original: We denote this method as PPD-SG+pretrain and the results are shown in Figure 2.
Revised: We denote this method as PPD-SG+pretrain and the results are shown in Figure 2.

Original: The parameters are tuned in the same range as in Section 5.
Revised: The parameters are tuned in the same range as in Section 5.

Original: It is observed that pretraining model helps the convergence of model and it can achieve the better performance in terms of AUC in most cases.
Revised: It is observed that pretraining model helps the convergence of model and it can achieve the better performance in terms of AUC in most cases.

Original: A. A DDITIONAL E XPERIMENTS WITH D IFFERENT L ABELING O RDER
Revised: A. A DDITIONAL E XPERIMENTS WITH D IFFERENT L ABELING O RDER

Original: To investigate the effects of labeling order, we also attempt to randomly partition the classes as positive or negative equally.
Revised: To investigate the effects of labeling order, we also attempt to randomly partition the classes as positive or negative equally.

Original: For CIFAR10 and STL10 dataset, we randomly partition the 10 classes into two labels (i.e., randomly select 5 classes as positive label and other 5 classes as negative label).
Revised: For CIFAR10 and STL10 dataset, we randomly partition the 10 classes into two labels (i.e., randomly select 5 classes as positive label and other 5 classes as negative label).

Original: For CIFAR100 dataset, we randomly partition the 100 classes into two labels (i.e., randomly select 50 classes as positive label and other 50 classes as negative label).
Revised: For CIFAR100 dataset, we randomly partition the 100 classes into two labels (i.e., randomly select 50 classes as positive label and other 50 classes as negative label).

Original: After that we randomly remove 95%, 90%, from negative samples on all training data, which lead to 20:1, 10:1 ratios respectively.
Revised: After that we randomly remove 95%, 90%, from negative samples on all training data, which lead to 20:1, 10:1 ratios respectively.

Original: For testing data, we keep them unchanged.
Revised: For testing data, we keep them unchanged.

Original: We also add AdaGrad for minimizing cross-entropy loss as a new baseline.
Revised: We also add AdaGrad for minimizing cross-entropy loss as a new baseline.

Original: The corresponding experimental results are included in Figure 3.
Revised: The corresponding experimental results are included in Figure 3.

Original: We can see that
Revised: We can see that

Original: PPD-Adagrad and PPD-SG converge faster than other baselines.
Revised: PPD-Adagrad and PPD-SG converge faster than other baselines.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]