Original: T RAINING N EURAL N ETWORKS FOR AND BY I NTERPOLATION
Revised: T RAINING N EURAL N ETWORKS FOR AND BY I NTERPOLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously.
Revised: In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously.

Original: In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning.
Revised: In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning.

Original: Speciﬁcally, we use it to compute an adaptive learning-rate in closed form at each iteration.
Revised: Speciﬁcally, we use it to compute an adaptive learning-rate in closed form at each iteration.

Original: This results in the Adaptive
Revised: This results in the Adaptive

Original: Learning-rates for Interpolation with Gradients (ALI-G) algorithm.
Revised: Learning-rates for Interpolation with Gradients (ALI-G) algorithm.

Original: ALI-G retains the main advantage of SGD which is a low computational cost per iteration.
Revised: ALI-G retains the main advantage of SGD which is a low computational cost per iteration.

Original: But unlike SGD, the learning-rate of ALI-G uses a single constant hyper-parameter and does not require a decay schedule, which makes it considerably easier to tune.
Revised: But unlike SGD, the learning-rate of ALI-G uses a single constant hyper-parameter and does not require a decay schedule, which makes it considerably easier to tune.

Original: We provide convergence guarantees of ALI-G in the stochastic convex setting.
Revised: We provide convergence guarantees of ALI-G in the stochastic convex setting.

Original: Notably, all our convergence results tackle the realistic case where the interpolation property is satisﬁed up to some tolerance.
Revised: Notably, all our convergence results tackle the realistic case where the interpolation property is satisﬁed up to some tolerance.

Original: We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the
Revised: We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the

Original: SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets.
Revised: SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets.

Original: ALI-G produces state-of-the-art results among adaptive methods, and even yields comparable performance with SGD, which requires manually tuned learning-rate schedules.
Revised: ALI-G produces state-of-the-art results among adaptive methods, and even yields comparable performance with SGD, which requires manually tuned learning-rate schedules.

Original: Furthermore, ALI-G is simple to implement in any standard deep learning framework and can be used as a drop-in replacement in existing code.
Revised: Furthermore, ALI-G is simple to implement in any standard deep learning framework and can be used as a drop-in replacement in existing code.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training a deep neural network is a challenging optimization problem: it involves minimizing the average of many high-dimensional non-convex functions.
Revised: Training a deep neural network is a challenging optimization problem: it involves minimizing the average of many high-dimensional non-convex functions.

Original: In practice, the main algorithms of choice are Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951) and adaptive gradient methods such as AdaGrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2015).
Revised: In practice, the main algorithms of choice are Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951) and adaptive gradient methods such as AdaGrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2015).

Original: In recent work, the ability to interpolate – i.e. to achieve near zero loss on all training samples simultaneously – has been used to show convergence of SGD (Ma et al., 2018, Vaswani et al., 2019a, Zhou et al., 2019).
Revised: In recent work, the ability to interpolate – i.e. to achieve near zero loss on all training samples simultaneously – has been used to show convergence of SGD (Ma et al., 2018, Vaswani et al., 2019a, Zhou et al., 2019).

Original: This property is usually satisﬁed in supervised deep learning because of the empirical success of over-parameterized architectures.
Revised: This property is usually satisﬁed in supervised deep learning because of the empirical success of over-parameterized architectures.

Original: However, while the convergence analyses provide a better theoretical understanding of
Revised: However, while the convergence analyses provide a better theoretical understanding of

Original: SGD, they do not help improve its practical behavior.
Revised: SGD, they do not help improve its practical behavior.

Original: In this work, we open a different line of enquiry, namely: can the interpolation property be used to design a robust and efﬁcient optimization algorithm for deep learning?
Revised: In this work, we open a different line of enquiry, namely: can the interpolation property be used to design a robust and efﬁcient optimization algorithm for deep learning?

Original: In order to answer this question, we begin by giving the following two desiderata of an optimization algorithm for deep learning: (i) an inexpensive computational cost per iteration (typically a call to a stochastic ﬁrst-order oracle); and (ii) adaptive learning-rates that do not require a manually designed schedule.
Revised: In order to answer this question, we begin by giving the following two desiderata of an optimization algorithm for deep learning: (i) an inexpensive computational cost per iteration (typically a call to a stochastic ﬁrst-order oracle); and (ii) adaptive learning-rates that do not require a manually designed schedule.

Original: We present ALI-G
Revised: We present ALI-G

Original: (Adaptive Learning-rates for Interpolation with Gradients), an algorithm that takes advantage of interpolation by design and satisﬁes both properties mentioned above.
Revised: (Adaptive Learning-rates for Interpolation with Gradients), an algorithm that takes advantage of interpolation by design and satisﬁes both properties mentioned above.

Original: Key to the
Revised: Key to the

Original: ALI-G algorithm are the following two ideas.
Revised: ALI-G algorithm are the following two ideas.

Original: First, an adaptive learning-rate can be computed for the non-stochastic gradient direction when the minimum value of the objective function is known (Polyak, 1969, Shor, 1985, Brännlund, 1995, Nedi´c & Bertsekas, 2001a;b).
Revised: First, an adaptive learning-rate can be computed for the non-stochastic gradient direction when the minimum value of the objective function is known (Polyak, 1969, Shor, 1985, Brännlund, 1995, Nedi´c & Bertsekas, 2001a;b).

Original: And second, one such minimum value is usually approximately known for interpolating models: for instance, it is close to zero for a model trained with the cross-entropy loss.
Revised: And second, one such minimum value is usually approximately known for interpolating models: for instance, it is close to zero for a model trained with the cross-entropy loss.

Original: By carefully combining these two ideas, we create a stochastic algorithm that provably converges fast in the convex setting and that obtains state-of-the-art results with neural networks
Revised: By carefully combining these two ideas, we create a stochastic algorithm that provably converges fast in the convex setting and that obtains state-of-the-art results with neural networks

Original: Procedurally, ALI-G is close to many existing algorithms, such as Deep Frank-Wolfe (Berrada et al., 2019), A P ROX (Asi & Duchi, 2019) and L 4 (Rolinek & Martius, 2018).
Revised: Procedurally, ALI-G is close to many existing algorithms, such as Deep Frank-Wolfe (Berrada et al., 2019), A P ROX (Asi & Duchi, 2019) and L 4 (Rolinek & Martius, 2018).

Original: And yet uniquely, thanks to its careful design and analysis, ALI-G enables accurate optimization of a wide class of deep neural networks using only a single hyper-parameter that does not need to be decayed.
Revised: And yet uniquely, thanks to its careful design and analysis, ALI-G enables accurate optimization of a wide class of deep neural networks using only a single hyper-parameter that does not need to be decayed.

Original: This makes ALI-G well-suited to the deep learning setting, where hyper-parameter tuning is widely regarded as an onerous and time consuming task.
Revised: This makes ALI-G well-suited to the deep learning setting, where hyper-parameter tuning is widely regarded as an onerous and time consuming task.

Original: Since ALI-G is easy to implement in any deep learning framework, we believe that it can prove to be a practical and reliable optimization tool for deep learning.
Revised: Since ALI-G is easy to implement in any deep learning framework, we believe that it can prove to be a practical and reliable optimization tool for deep learning.

Original: Contributions.
Revised: Contributions.

Original: We summarize the contributions of this work as follows:
Revised: We summarize the contributions of this work as follows:

Original: - We design an optimization algorithm that uses a single hyper-parameter for its learning-rate and does need any decaying schedule.
Revised: - We design an optimization algorithm that uses a single hyper-parameter for its learning-rate and does need any decaying schedule.

Original: In contrast, the closely related A P ROX
Revised: In contrast, the closely related A P ROX

Original: Asi
Revised: Asi

Original: & Duchi,
Revised: & Duchi,

Original: ) and L 4 (Rolinek & Martius, 2018) use respectively two and four hyper-parameters for their learning-rate.
Revised: ) and L 4 (Rolinek & Martius, 2018) use respectively two and four hyper-parameters for their learning-rate.

Original: - We provide convergence rates of ALI-G in various stochastic convex settings.
Revised: - We provide convergence rates of ALI-G in various stochastic convex settings.

Original: Importantly, our theoretical results take into account the error in the estimate of the minimum objective value.
Revised: Importantly, our theoretical results take into account the error in the estimate of the minimum objective value.

Original: To the best of our knowledge, our work is the ﬁrst to establish convergence rates for interpolation with approximate estimates.
Revised: To the best of our knowledge, our work is the ﬁrst to establish convergence rates for interpolation with approximate estimates.

Original: - We demonstrate state-of-the-art results for ALI-G on learning a differentiable neural computer; training variants of residual networks on the SVHN and CIFAR data sets; and training a Bi-LSTM on the Stanford Natural Language Inference data set.
Revised: - We demonstrate state-of-the-art results for ALI-G on learning a differentiable neural computer; training variants of residual networks on the SVHN and CIFAR data sets; and training a Bi-LSTM on the Stanford Natural Language Inference data set.

Original:  T HE A LGORITHM 2. P ROBLEM S ETTING
Revised:  T HE A LGORITHM 2. P ROBLEM S ETTING

Original: Loss Function.
Revised: Loss Function.

Original: We consider a supervised learning task where the model is parameterized by w ∈ R p .
Revised: We consider a supervised learning task where the model is parameterized by w ∈ R p .

Original: Usually, the objective function can be expressed as an expectation over z ∈ Z , a random variable indexing the samples of the training set:
Revised: Usually, the objective function can be expressed as an expectation over z ∈ Z , a random variable indexing the samples of the training set:

Original: [Equation]
Revised: [Equation]

Original: where each (cid:96) z is the loss function associated with the sample z .
Revised: where each (cid:96) z is the loss function associated with the sample z .

Original: We assume that each (cid:96) z is nonnegative, which is the case for the large majority of loss functions used in machine learning.
Revised: We assume that each (cid:96) z is nonnegative, which is the case for the large majority of loss functions used in machine learning.

Original: For instance, suppose that the model is a deep neural network with weights w performing classiﬁcation.
Revised: For instance, suppose that the model is a deep neural network with weights w performing classiﬁcation.

Original: Then for each sample z , (cid:96) z ( w ) can represent the cross-entropy loss, which is always non-negative.
Revised: Then for each sample z , (cid:96) z ( w ) can represent the cross-entropy loss, which is always non-negative.

Original: Other non-negative loss functions include the structured or multi-class hinge loss, and the L 1 or Lloss functions for regression.
Revised: Other non-negative loss functions include the structured or multi-class hinge loss, and the L 1 or Lloss functions for regression.

Original: Regularization.
Revised: Regularization.

Original: It is often desirable to employ a regularization function φ in order to promote generalization.
Revised: It is often desirable to employ a regularization function φ in order to promote generalization.

Original: In this work, we incorporate such regularization as a constraint on the feasible domain: Ω = { w ∈ R p : φ ( w ) ≤ r } for some value of r .
Revised: In this work, we incorporate such regularization as a constraint on the feasible domain: Ω = { w ∈ R p : φ ( w ) ≤ r } for some value of r .

Original: In the deep learning setting, this will allow us to assume that the objective function can be driven close to zero without unrealistic assumptions about the regularization.
Revised: In the deep learning setting, this will allow us to assume that the objective function can be driven close to zero without unrealistic assumptions about the regularization.

Original: Our framework can handle any constraint set Ω on which Euclidean projections are computationally efﬁcient.
Revised: Our framework can handle any constraint set Ω on which Euclidean projections are computationally efﬁcient.

Original: This includes the feasible set induced by L 2 regularization:
Revised: This includes the feasible set induced by L 2 regularization:

Original: Ω = (cid:8) w ∈ R p : (cid:107) w (cid:107) 22 ≤ r (cid:9) , for which the projection is given by a simple rescaling of w .
Revised: Ω = (cid:8) w ∈ R p : (cid:107) w (cid:107) 22 ≤ r (cid:9) , for which the projection is given by a simple rescaling of w .

Original: Finally, note that if we do not wish to use any regularization, we deﬁne Ω = R p and the corresponding projection is the identity.
Revised: Finally, note that if we do not wish to use any regularization, we deﬁne Ω = R p and the corresponding projection is the identity.

Original: Problem Formulation.
Revised: Problem Formulation.

Original: The learning task can be expressed as the problem ( P ) of ﬁnding a feasible vector of parameters w (cid:63) ∈
Revised: The learning task can be expressed as the problem ( P ) of ﬁnding a feasible vector of parameters w (cid:63) ∈

Original: Ω that minimizes f :
Revised: Ω that minimizes f :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Also note that f (cid:63) refers to the minimum value of f over Ω : f (cid:63) (cid:44)
Revised: Also note that f (cid:63) refers to the minimum value of f over Ω : f (cid:63) (cid:44)

Original: min w ∈ Ω f ( w ) .
Revised: min w ∈ Ω f ( w ) .

Original: 2. T HE P OLYAK S TEP -S IZE
Revised: 2. T HE P OLYAK S TEP -S IZE

Revised: Before outlining the ALI-G algorithm, we begin with a brief description of the Polyak step-size, from which ALI-G draws some fundamental ideas.

Original: Setting.
Revised: Setting.

Original: We assume that f (cid:63) is known and we use non-stochastic updates: at each iteration, the full objective f and its derivative are evaluated.
Revised: We assume that f (cid:63) is known and we use non-stochastic updates: at each iteration, the full objective f and its derivative are evaluated.

Original: We denote by ∇ f ( w ) the ﬁrst-order derivative of f at w (e.g. ∇ f ( w ) can be a sub-gradient or the gradient).
Revised: We denote by ∇ f ( w ) the ﬁrst-order derivative of f at w (e.g. ∇ f ( w ) can be a sub-gradient or the gradient).

Original: In addition, (cid:107) · (cid:107) is the standard Euclidean norm in R p , and Π Ω ( w ) is the Euclidean projection of the vector w ∈ R p on the set Ω .
Revised: In addition, (cid:107) · (cid:107) is the standard Euclidean norm in R p , and Π Ω ( w ) is the Euclidean projection of the vector w ∈ R p on the set Ω .

Original: Polyak Step-Size.
Revised: Polyak Step-Size.

Original: At time-step t , using the Polyak step-size (Polyak, 1969, Shor, 1985, Brännlund, 1995, Nedi´c & Bertsekas, 2001a;b) yields the following update:[Equation] where
Revised: At time-step t , using the Polyak step-size (Polyak, 1969, Shor, 1985, Brännlund, 1995, Nedi´c & Bertsekas, 2001a;b) yields the following update:

Original: [Equation]
Revised: [Equation]

Original: where we loosely deﬁne 

Revised: where we loosely deﬁne

Original: = 0 for simplicity purposes.
Revised: = 0 for simplicity purposes.

Original: [Figure]
Revised: [Figure]

Original: Interpretation.
Revised: Interpretation.

Original: It can be shown that w t +1 lies on the intersection between the linearization of f at w t and the horizontal plane z
Revised: It can be shown that w t +1 lies on the intersection between the linearization of f at w t and the horizontal plane z

Original: = f (cid:63) (see Figure 1, more details in Proposition 2 in the supplementary material).
Revised: = f (cid:63) (see Figure 1, more details in Proposition 2 in the supplementary material).

Original: Note that since f (cid:63) is the minimum of f , the Polyak step-size γ t is necessarily non-negative.
Revised: Note that since f (cid:63) is the minimum of f , the Polyak step-size γ t is necessarily non-negative.

Original: Limitations.
Revised: Limitations.

Original: Equation (2) has two major shortcomings that prevent its applicability in a machine learning setting.
Revised: Equation (2) has two major shortcomings that prevent its applicability in a machine learning setting.

Original: First, each update requires a full evaluation of f and its derivative.
Revised: First, each update requires a full evaluation of f and its derivative.

Original: Stochastic extensions have been proposed in Nedi ´ c & Bertsekas (2001a;b), but they still require frequent evaluations of f .
Revised: Stochastic extensions have been proposed in Nedi ´ c & Bertsekas (2001a;b), but they still require frequent evaluations of f .

Original: This is expensive in the large data setting, and even computationally infeasible when using massive data augmentation.
Revised: This is expensive in the large data setting, and even computationally infeasible when using massive data augmentation.

Original: Second, when applying this method to the non-convex setting of deep neural networks, the method sometimes fails to converge.
Revised: Second, when applying this method to the non-convex setting of deep neural networks, the method sometimes fails to converge.

Original: Therefore we would like to design an extension of the Polyak step-size that (i) is inexpensive to compute in a stochastic setting (e.g. with a computational cost that is independent of the total number of training samples), and (ii) converges in practice when used with deep neural networks.
Revised: Therefore we would like to design an extension of the Polyak step-size that (i) is inexpensive to compute in a stochastic setting (e.g. with a computational cost that is independent of the total number of training samples), and (ii) converges in practice when used with deep neural networks.

Original: The next section introduces the ALI-G algorithm, which achieves these two goals in the interpolation setting.
Revised: The next section introduces the ALI-G algorithm, which achieves these two goals in the interpolation setting.

Original: 2. T HE ALI-G A LGORITHM
Revised: 2. T HE ALI-G A LGORITHM

Revised: We now present the ALI-G algorithm.

Original: We assume that we are in an interpolation setting: the model is assumed to be able to drive the loss function to near zero on all samples simultaneously.
Revised: For this, we suppose that we are in an interpolation setting: the model is assumed to be able to drive the loss function to near zero on all samples simultaneously.

Original: Algorithm.
Revised: Algorithm.

Original: The main steps of the ALI-G algorithm are provided in Algorithm 1.
Revised: The main steps of the ALI-G algorithm are provided in Algorithm 1.

Original: ALI-G iterates over three operations until convergence.
Revised: ALI-G iterates over three operations until convergence.

Original: First, it computes a stochastic approximation of the learning objective and its derivative (line 3).
Revised: First, it computes a stochastic approximation of the learning objective and its derivative (line 3).

Original: Second, it computes a step-size decay parameter γ t based on the stochastic information (line 4).
Revised: Second, it computes a step-size decay parameter γ t based on the stochastic information (line 4).

Original: Third, it updates the parameters by moving in the negative derivative direction by an amount speciﬁed by the step-size and projecting the resulting vector on to the feasible region (line 5).
Revised: Third, it updates the parameters by moving in the negative derivative direction by an amount speciﬁed by the step-size and projecting the resulting vector on to the feasible region (line 5).

Original: Algorithm 

Revised: Algorithm

Original: TheALI-G algorithm
Revised: The

Original: Require: maximal learning-rate η , initial feasible w 0 ∈ Ω , small constant δ >1: t =2: while not converged do 3:
Revised: Require: maximal learning-rate η , initial feasible w 0 ∈ Ω , small constant δ >1: t =2: while not converged do 3:

Original: Get (cid:96) z t ( w t ) , ∇ (cid:96) z t ( w t ) with z t drawn i.i.d.
Revised: Get (cid:96) z t ( w t ) , ∇ (cid:96) z t ( w t ) with z t drawn i.i.d.

Original: 4: γ t = min (cid:110) (cid:96) zt ( w t ) (cid:107)∇ (cid:96) zt ( w t ) (cid:107) 2 + δ , η (cid:111) 5: w t +1 = Π Ω ( w t − γ t ∇ (cid:96) z t ( w t )) 6: t = t +7: end while
Revised: 4: γ t = min (cid:110) (cid:96) zt ( w t ) (cid:107)∇ (cid:96) zt ( w t ) (cid:107) 2 + δ , η (cid:111) 5: w t +1 = Π Ω ( w t − γ t ∇ (cid:96) z t ( w t )) 6: t = t +7: end while

Original: Comparison with the Polyak Step-Size.
Revised: Comparison with the Polyak Step-Size.

Original: There are three main differences to the update in equation (2).
Revised: There are three main differences to the update in equation (2).

Original: First, each update only uses the loss (cid:96) z t and its derivative rather than the full objective f and its derivative.
Revised: First, each update only uses the loss (cid:96) z t and its derivative rather than the full objective f and its derivative.

Original: Second, the learning-rate γ t is clipped to η , the maximal learning-rate hyper-parameter.
Revised: Second, the learning-rate γ t is clipped to η , the maximal learning-rate hyper-parameter.

Original: We emphasize that η remains constant throughout the iterations, therefore it is a single hyper-parameter and does not need a schedule like SGD learning-rate.
Revised: We emphasize that η remains constant throughout the iterations, therefore it is a single hyper-parameter and does not need a schedule like SGD learning-rate.

Original: Third, the minimum f (cid:63) has been replaced by the lower-bound of 0 .
Revised: Third, the minimum f (cid:63) has been replaced by the lower-bound of 0 .

Original: All these modiﬁcations will be justiﬁed in the next section.
Revised: All these modiﬁcations will be justiﬁed in the next section.

Original: The ALI-G ∞ Variant.
Revised: The ALI-G ∞ Variant.

Original: When ALI-G uses no maximal learning-rate, we refer to the algorithm as
Revised: When ALI-G uses no maximal learning-rate, we refer to the algorithm as

Original: ALI-G ∞ , since it is equivalent to use an inﬁnite maximal learning-rate.
Revised: ALI-G ∞ , since it is equivalent to use an inﬁnite maximal learning-rate.

Original: Note that ALI-G ∞ requires no hyper-parameter for its step-size.
Revised: Note that ALI-G ∞ requires no hyper-parameter for its step-size.

Original:  J USTIFICATION AND A NALYSIS 3. I NTERPOLATION E NABLES I NEXPENSIVE S TOCHASTIC U PDATES
Revised:  J USTIFICATION AND A NALYSIS 3. I NTERPOLATION E NABLES I NEXPENSIVE S TOCHASTIC U PDATES

Original: By deﬁnition, the interpolation setting gives f (cid:63) = 0 , which we used in ALI-G to simplify the formula of the learning-rate γ t .
Revised: By deﬁnition, the interpolation setting gives f (cid:63) = 0 , which we used in ALI-G to simplify the formula of the learning-rate γ t .

Original: More subtly, the interpolation property also allows the updates to rely on the stochastic estimate (cid:96) z t ( w t ) rather than the exact but expensive f ( w t ) .
Revised: More subtly, the interpolation property also allows the updates to rely on the stochastic estimate (cid:96) z t ( w t ) rather than the exact but expensive f ( w t ) .

Original: Intuitively, this is possible because in the interpolation setting, each training sample can use its own learning-rate without harming progress on the other ones.
Revised: Intuitively, this is possible because in the interpolation setting, each training sample can use its own learning-rate without harming progress on the other ones.

Original: Recall that ALI-G ∞ is the variant of ALI-G that uses no maximal learning-rate.
Revised: Recall that ALI-G ∞ is the variant of ALI-G that uses no maximal learning-rate.

Original: The following result formalizes the convergence guarantee of ALI-G ∞ in the stochastic convex setting:
Revised: The following result formalizes the convergence guarantee of ALI-G ∞ in the stochastic convex setting:

Original: Theorem 1 (Convex and Lipschitz) .
Revised: Theorem 1 (Convex and Lipschitz) .

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: We assume that Ω is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let w (cid:63) be a solution of ( P ), and assume that the interpolation property is approximately satisﬁed: ∀ z ∈ Z , (cid:96) z ( w (cid:63) ) ≤ ε , for some interpolation tolerance ε ≥ 0 .
Revised: Let w (cid:63) be a solution of ( P ), and assume that the interpolation property is approximately satisﬁed: ∀ z ∈ Z , (cid:96) z ( w (cid:63) ) ≤ ε , for some interpolation tolerance ε ≥ 0 .

Original: Then
Revised: Then

Original: ALI-G ∞ applied to f satisﬁes:
Revised: ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Revised: In other words, by assuming interpolation, ALI-G provably converges while requiring only (cid:96) z t ( w t ) and ∇ (cid:96) z t ( w t ) (stochastic estimation per sample) to compute its learning-rate.

Revised: In contrast, the

Revised: Polyak step-size, which does not exploit interpolation, would require f ( w t ) and ∇ f ( w t ) to compute the learning-rate (deterministic computation over all training samples).

Revised: This constitutes a major computational advantage of ALI-G over the usual Polyak step-size.

Original: We emphasize that  our careful analysis explicitly shows the dependency of the convergence result on the interpolation tolerance ε .
Revised: We emphasize that in Theorem 1, our careful analysis explicitly shows the dependency of the convergence result on the interpolation tolerance ε .

Original: It is reassuring to note that convergence is exact when the interpolation property is exactly satisﬁed ( ε = 0 ).
Revised: It is reassuring to note that convergence is exact when the interpolation property is exactly satisﬁed ( ε = 0 ).

Original: In the supplementary material, we also establish convergence rates of O (1 /T ) for smooth convex functions, and O (exp( − αT/ 8 β )) for α -strongly convex and β -smooth functions.
Revised: In the supplementary material, we also establish convergence rates of O (1 /T ) for smooth convex functions, and O (exp( − αT/ 8 β )) for α -strongly convex and β -smooth functions.

Original: Similar results can be proved when using a maximal learning-rate η : the convergence speed then remains unchanged provided that η is large enough, and it is lowered when η is small.
Revised: Similar results can be proved when using a maximal learning-rate η : the convergence speed then remains unchanged provided that η is large enough, and it is lowered when η is small.

Original: We refer the interested reader to the supplementary for the formal results and their proofs.
Revised: We refer the interested reader to the supplementary for the formal results and their proofs.

Original: 3. A M AXIMAL L EARNING -R ATE H ELPS WITH N ON -C ONVEXITY
Revised: 3. A M AXIMAL L EARNING -R ATE H ELPS WITH N ON -C ONVEXITY

Original: [Equation] [Figure] The Polyak step-size may fail to converge when the objective is non-convex, as ﬁgure 2 illustrates: in this (non-convex) setting, gradient descent with Polyak step-size oscillates between two symmetrical points because its step-size is too large (details in the supplementary).
Revised: The Polyak step-size may fail to converge when the objective is non-convex, as ﬁgure 2 illustrates: in this (non-convex) setting, gradient descent with Polyak step-size oscillates between two symmetrical points because its step-size is too large (details in the supplementary).

Original: Indeed, we empirically ﬁnd that non-convexity usually leads to overestimation of the Polyak step-size.
Revised: Indeed, we empirically ﬁnd that non-convexity usually leads to overestimation of the Polyak step-size.

Original: Intuitively, using a maximal learning-rate allows ALI-G to behave like constant step-size SGD in non-convex regions where the Polyak stepsize would be over-estimated, and to automatically use the
Revised: Intuitively, using a maximal learning-rate allows ALI-G to behave like constant step-size SGD in non-convex regions where the Polyak step-size would be over-estimated, and to automatically use the

Original: Polyak step-size once reaching a convex basin of convergence.
Revised: Polyak step-size once reaching a convex basin of convergence.

Original: Importantly, using a maximal learning-rate can be seen as a very natural extension of SGD when using a non-negative loss function:
Revised: Importantly, using a maximal learning-rate can be seen as a very natural extension of SGD when using a non-negative loss function:

Original: Proposition 1.
Revised: Proposition 1.

Original: [Proximal Interpretation] Suppose that Ω = R p and let δ = 0 .
Revised: [Proximal Interpretation] Suppose that Ω = R p and let δ = 0 .

Original: We consider the update performed by SGD: w SGD t +1 =
Revised: We consider the update performed by SGD: w SGD t +1 =

Original: w t − η t ∇ (cid:96) z t ( w t ) ; and the update performed by ALI-G: [Equation]
Revised: w t − η t ∇ (cid:96) z t ( w t ) ; and the update performed by ALI-G: w ALI-G t +1 =

Original: = [Equation] t ( w t ) , where γ t = min (cid:110)
Revised: w t − γ t ∇ (cid:96) z t ( w t ) , where γ t = min (cid:110)

Original: [Equation]
Revised: [Equation]

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original:  In other words, at each iteration, ALI-G solves a proximal problem in closed form in a similar way to SGD.
Revised: [Figure] In other words, at each iteration, ALI-G solves a proximal problem in closed form in a similar way to SGD.

Original: In both cases, the loss function (cid:96) z t is locally approximated by a ﬁrst-order Taylor expansion at w t .
Revised: In both cases, the loss function (cid:96) z t is locally approximated by a ﬁrst-order Taylor expansion at w t .

Original: The difference is that ALI-G also exploits the fact that (cid:96) z t is non-negative.
Revised: The difference is that ALI-G also exploits the fact that (cid:96) z t is nonnegative.

Original: This allows ALI-G to use a constant value for η in the interpolation setting, while the learning-rate η t of SGD needs to be manually decayed.
Revised: This allows ALI-G to use a constant value for η in the interpolation setting, while the learning-rate η t of SGD needs to be manually decayed.

Original: It is currently an open question whether ALI-G can be proved to converge in the  non-convex setting given a sufﬁciently small yet constant maximal learning-rate η .
Revised: It is currently an open question whether ALI-G can be proved to converge in the stochastic non-convex setting given a sufﬁciently small yet constant maximal learning-rate η .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Interpolation in Deep Learning.
Revised: Interpolation in Deep Learning.

Original: As mentioned in the introduction, recent works have successfully exploited the interpolation assumption to prove convergence of SGD in the context of deep learning (Ma et al., 2018, Vaswani et al., 2019a, Zhou et al., 2019).
Revised: As mentioned in the introduction, recent works have successfully exploited the interpolation assumption to prove convergence of SGD in the context of deep learning (Ma et al., 2018, Vaswani et al., 2019a, Zhou et al., 2019).

Original: Such works are complementary to ours in the sense that they provide a convergence analysis of an existing algorithm for deep learning.
Revised: Such works are complementary to ours in the sense that they provide a convergence analysis of an existing algorithm for deep learning.

Original: Adaptive Gradient Methods.
Revised: Adaptive Gradient Methods.

Original: Similarly to ALI-G, adaptive gradient methods also rely on tuning a single hyper-parameter, thereby providing a more pragmatic alternative to SGD that needs a speciﬁcation of the full learning-rate schedule.
Revised: Similarly to ALI-G, adaptive gradient methods also rely on tuning a single hyper-parameter, thereby providing a more pragmatic alternative to SGD that needs a speciﬁcation of the full learning-rate schedule.

Original: While the most popular ones are Adagrad (Duchi et al., 2011), RMSPROP (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2015) and AMSGrad (Reddi et al., 2018), there have been many other variants (Zeiler, 2012, Orabona & Pál, 2015, Défossez & Bach, 2017, Levy, 2017, Mukkamala & Hein, 2017, Zheng & Kwok, 2017, Bernstein et al., 2018,
Revised: While the most popular ones are Adagrad (Duchi et al., 2011), RMSPROP (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2015) and AMSGrad (Reddi et al., 2018), there have been many other variants (Zeiler, 2012, Orabona & Pál, 2015, Défossez & Bach, 2017, Levy, 2017, Mukkamala & Hein, 2017, Zheng & Kwok, 2017, Bernstein et al., 2018,

Original: Chen & Gu, 2018, Shazeer & Stern, 2018, Zaheer et al., 2018, Chen et al., 2019, Loshchilov & Hutter, 2019, Luo et al., 2019).
Revised: Chen & Gu, 2018, Shazeer & Stern, 2018, Zaheer et al., 2018, Chen et al., 2019, Loshchilov & Hutter, 2019, Luo et al., 2019).

Original: However, as pointed out in Wilson et al.
Revised: However, as pointed out in Wilson et al.

Original: (2017), adaptive gradient methods tend to give poor generalization in supervised learning.
Revised: (2017), adaptive gradient methods tend to give poor generalization in supervised learning.

Original: In our experiments, the results provided by
Revised: In our experiments, the results provided by

Original: ALI-G are signiﬁcantly better than those obtained by the most popular adaptive gradient methods.
Revised: ALI-G are signiﬁcantly better than those obtained by the most popular adaptive gradient methods.

Original: Adaptive Learning-Rate Algorithms.
Revised: Adaptive Learning-Rate Algorithms.

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2019b) show that one can use line search in a stochastic setting for interpolating models while guaranteeing convergence.
Revised: (2019b) show that one can use line search in a stochastic setting for interpolating models while guaranteeing convergence.

Original: However, in contrast to our work, the resulting algorithm requires more than one hyper-parameter (up to four), and the line-search is not computed in closed form.
Revised: However, in contrast to our work, the resulting algorithm requires more than one hyper-parameter (up to four), and the line-search is not computed in closed form.

Original: Less closely related methods have proposed adaptive learning-rates without using the minimum for the computation of the learning rate (Schaul et al., 2013,
Revised: Less closely related methods have proposed adaptive learning-rates without using the minimum for the computation of the learning rate (Schaul et al., 2013,

Original: Tan et al., 2016, Zhang et al., 2017, Baydin et al., 2018, Wu et al., 2018, Li & Orabona, 2019), but they have not demonstrated competitiveness against SGD with a well-tuned hand-designed schedule.
Revised: Tan et al., 2016, Zhang et al., 2017, Baydin et al., 2018, Wu et al., 2018, Li & Orabona, 2019), but they have not demonstrated competitiveness against SGD with a well-tuned hand-designed schedule.

Original: L 4 Algorithm.
Revised: L 4 Algorithm.

Original: The L 4 algorithm (Rolinek & Martius, 2018) also uses a modiﬁed version of the
Revised: The L 4 algorithm (Rolinek & Martius, 2018) also uses a modiﬁed version of the

Original: Polyak step-size.
Revised: Polyak step-size.

Original: However, the L 4 algorithm computes an online estimate of f (cid:63) rather than relying on a ﬁxed value.
Revised: However, the L 4 algorithm computes an online estimate of f (cid:63) rather than relying on a ﬁxed value.

Original: This requires three hyper-parameters, which are in practice sensitive to noise and crucial for empirical convergence of the method.
Revised: This requires three hyper-parameters, which are in practice sensitive to noise and crucial for empirical convergence of the method.

Original: In addition, L 4 does not come with convergence guarantees.
Revised: In addition, L 4 does not come with convergence guarantees.

Original: In contrast, by utilizing the interpolation property and a maximal learning-rate, our method is able to (i) provide reliable and accurate minimization with only a single hyper-parameter, and (ii) offer guarantees of convergence in the stochastic convex setting.
Revised: In contrast, by utilizing the interpolation property and a maximal learning-rate, our method is able to (i) provide reliable and accurate minimization with only a single hyper-parameter, and (ii) offer guarantees of convergence in the stochastic convex setting.

Original: Frank-Wolfe Methods.
Revised: Frank-Wolfe Methods.

Original: The proximal interpretation in Proposition 1 allows us to draw additional parallels to existing methods.
Revised: The proximal interpretation in Proposition 1 allows us to draw additional parallels to existing methods.

Original: In particular, the formula of the learning-rate γ t may remind the reader of the Frank-Wolfe algorithm (Frank & Wolfe, 1956) in some of its variants (Locatello et al., 2017), or other dual methods (Lacoste-Julien et al., 2013, Shalev-Shwartz & Zhang, 2016).
Revised: In particular, the formula of the learning-rate γ t may remind the reader of the Frank-Wolfe algorithm (Frank & Wolfe, 1956) in some of its variants (Locatello et al., 2017), or other dual methods (Lacoste-Julien et al., 2013, Shalev-Shwartz & Zhang, 2016).

Original: This is because such methods solve in closed form the dual of problem (5), and problems in the form of (5) naturally appear in dual coordinate ascent methods (Shalev-Shwartz & Zhang, 2016).
Revised: This is because such methods solve in closed form the dual of problem (5), and problems in the form of (5) naturally appear in dual coordinate ascent methods (Shalev-Shwartz & Zhang, 2016).

Original: When no regularization is used, ALI-G and Deep Frank-Wolfe (DFW) (Berrada et al., 2019) are procedurally identical algorithms.
Revised: When no regularization is used, ALI-G and Deep Frank-Wolfe (DFW) (Berrada et al., 2019) are procedurally identical algorithms.

Original: This is because in such a setting, one iteration of DFW also amounts to solving (5) in closed-form – more generally, DFW is designed to train deep neural networks by solving proximal linear support vector machine problems approximately.
Revised: This is because in such a setting, one iteration of DFW also amounts to solving (5) in closed-form – more generally, DFW is designed to train deep neural networks by solving proximal linear support vector machine problems approximately.

Original: However, we point out the two fundamental advantages of ALI-G over DFW: (i) ALI-G can handle arbitrary (lower-bounded) loss functions, while DFW can only use convex piece-wise linear loss functions; and (ii) as seen previously, ALI-G provides convergence guarantees in the convex setting. 
Revised: However, we point out the two fundamental advantages of ALI-G over DFW: (i) ALI-G can handle arbitrary (lower-bounded) loss functions, while DFW can only use convex piece-wise linear loss functions; and (ii) as seen previously, ALI-G provides convergence guarantees in the convex setting. SGD with Polyak’s Learning-Rate.

Revised: Oberman & Prazeres (2019) extend the Polyak step-size to rely on a stochastic estimation of the gradient ∇ (cid:96) z t ( w t ) only, instead of the expensive deterministic gradient ∇ f ( w t ) .

Revised: However, they still require to evaluate f ( w t ) , the objective function over the entire training data set, in order to compute its learning-rate, which makes the method impractical.

Revised: In addition, since they do not do exploit the interpolation setting nor the fact that regularization can be expressed as a constraint, they also require the optimal objective function value f (cid:63) in advance.

Original: A P ROX Algorithm.
Revised: A P ROX Algorithm.

Original: Asi & Duchi (2019) have recently introduced the A P ROX algorithm, a family of proximal stochastic optimization algorithms for convex problems.
Revised: Asi & Duchi (2019) have recently introduced the A P ROX algorithm, a family of proximal stochastic optimization algorithms for convex problems.

Original: Notably, the A P ROX “truncated model” version is similar to ALI-G.
Revised: Notably, the A P ROX “truncated model” version is similar to ALI-G.

Original: However, there are four clear advantages of our work over (Asi & Duchi, 2019) in the interpolation setting, in particular for training neural networks.
Revised: However, there are four clear advantages of our work over (Asi & Duchi, 2019) in the interpolation setting, in particular for training neural networks.

Original: First, our work is the ﬁrst to empirically demonstrate the applicability and usefulness of the algorithm on varied modern deep learning tasks – most of our experiments use several orders of magnitude more data and model parameters than the small-scale convex problems of (Asi & Duchi, 2019).
Revised: First, our work is the ﬁrst to empirically demonstrate the applicability and usefulness of the algorithm on varied modern deep learning tasks – most of our experiments use several orders of magnitude more data and model parameters than the small-scale convex problems of (Asi & Duchi, 2019).

Original: Second, our analysis and insights allow us to make more aggressive choices of learning rate than (Asi & Duchi, 2019).
Revised: Second, our analysis and insights allow us to make more aggressive choices of learning rate than (Asi & Duchi, 2019).

Original: Indeed, Asi & Duchi (2019) assume that the maximal learning-rate is exponentially decaying, even in the interpolating convex setting.
Revised: Indeed, Asi & Duchi (2019) assume that the maximal learning-rate is exponentially decaying, even in the interpolating convex setting.

Original: In contrast, by avoiding the need for an exponential decay, the learning-rate of ALI-G requires only one hyper-parameters instead of two for A P ROX .
Revised: In contrast, by avoiding the need for an exponential decay, the learning-rate of ALI-G requires only one hyper-parameters instead of two for A P ROX .

Original: Third, our analysis takes into account the interpolation tolerance ε ≥ 0 rather than unrealistically assuming the perfect case ε = 

Revised: Third, our analysis takes into account the interpolation tolerance ε ≥ 0 rather than unrealistically assuming the perfect case ε =

Original: (that would require inﬁnite weights when using the cross-entropy loss for instance).
Revised: (that would require inﬁnite weights when using the cross-entropy loss for instance).

Original: Fourth, our analysis proves fast convergence in function space rather than iterate space.
Revised: Fourth, our analysis proves fast convergence in function space rather than iterate space.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We empirically compare ALI-G to the optimization algorithms most commonly used in deep learning.
Revised: We empirically compare ALI-G to the optimization algorithms most commonly used in deep learning.

Original: Our experiments span a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training wide residual networks on SVHN; (iii) training a Bi-LSTM on the Stanford Natural
Revised: Our experiments span a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training wide residual networks on SVHN; (iii) training a Bi-LSTM on the Stanford Natural

Original: Language Inference data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets.
Revised: Language Inference data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets.

Original: Note that the tasks of training wide residual networks on SVHN and CIFAR100 are part of the DeepOBS benchmark Schneider et al.
Revised: Note that the tasks of training wide residual networks on SVHN and CIFAR100 are part of the DeepOBS benchmark Schneider et al.

Original: (2019), which aims at standardizing baselines for deep learning optimizers.
Revised: (2019), which aims at standardizing baselines for deep learning optimizers.

Original: In particular, these tasks are among the most difﬁcult ones of the benchmark because the SGD baseline beneﬁts from a manual schedule for the learning rate.
Revised: In particular, these tasks are among the most difﬁcult ones of the benchmark because the SGD baseline beneﬁts from a manual schedule for the learning rate.

Original: Despite this, ALI-G obtains competitive performance with SGD.
Revised: Despite this, ALI-G obtains competitive performance with SGD.

Original: In addition, ALI-G is the best performing method with a single hyper-parameter on the difﬁcult tasks of Bi-LSTM on SNLI and ResNet variants on CIFAR.
Revised: In addition, ALI-G is the best performing method with a single hyper-parameter on the difﬁcult tasks of Bi-LSTM on SNLI and ResNet variants on CIFAR.

Original: he code to reproduce our results will be made publicly available.
Revised: The code to reproduce our results will be made publicly available.

Original: In the Tensorﬂow (Abadi et al., 2015) experiment, we use the ofﬁcial and publicly available implementation of L 4 1 .
Revised: In the Tensorﬂow (Abadi et al., 2015) experiment, we use the ofﬁcial and publicly available implementation of L 4 1 .

Original: In the PyTorch (Paszke et al., 2017) experiments, we use our implementation of L 4 , which we unit-test against the ofﬁcial Tensorﬂow implementation.
Revised: In the PyTorch (Paszke et al., 2017) experiments, we use our implementation of L 4 , which we unit-test against the ofﬁcial Tensorﬂow implementation.

Original: In addition, we employ the ofﬁcial implementation of DFW 2 and we re-use their code for the experiments on SNLI and CIFAR.
Revised: In addition, we employ the ofﬁcial implementation of DFW 2 and we re-use their code for the experiments on SNLI and CIFAR.

Original: All experiments are performed either on a 12-core CPU (differentiable neural computer) or on a single GPU (SVHN, SNLI, CIFAR).
Revised: All experiments are performed either on a 12-core CPU (differentiable neural computer) or on a single GPU (SVHN, SNLI, CIFAR).

Original: 5. D IFFERENTIABLE N EURAL C OMPUTERS
Revised: 5. D IFFERENTIABLE N EURAL C OMPUTERS

Original: Setting.
Revised: Setting.

Original: The Differentiable Neural Computer (DNC) (Graves et al., 2016) is a recurrent neural network that aims at performing computing tasks by learning from examples rather than by executing an explicit program.
Revised: The Differentiable Neural Computer (DNC) (Graves et al., 2016) is a recurrent neural network that aims at performing computing tasks by learning from examples rather than by executing an explicit program.

Original: In this case, the DNC learns to repeatedly copy a ﬁxed size string given as input.
Revised: In this case, the DNC learns to repeatedly copy a ﬁxed size string given as input.

Original: Although this learning task is relatively simple, the complex architecture of the DNC makes it an interesting benchmark problem for optimization algorithms.
Revised: Although this learning task is relatively simple, the complex architecture of the DNC makes it an interesting benchmark problem for optimization algorithms.

Original: Methods.
Revised: Methods.

Original: We use the ofﬁcial and publicly available implementation of DNC 3 .
Revised: We use the ofﬁcial and publicly available implementation of DNC 3 .

Original: We vary the initial learning rate as powers of ten between 10 − 4 and 10 4 for each method except for L 4 Adam and
Revised: We vary the initial learning rate as powers of ten between 10 − 4 and 10 4 for each method except for L 4 Adam and

Original: L 4 Mom.
Revised: L 4 Mom.

Original: For L 4 Adam and L 4 Mom, since the main hyper-parameter α is designed to lie in (0 , 1) , we vary it between 0 . 05 and 0 .
Revised: For L 4 Adam and L 4 Mom, since the main hyper-parameter α is designed to lie in (0 , 1) , we vary it between 0 . 05 and 0 .

Original: 095 with a step of 0 . 1 .
Revised: 095 with a step of 0 . 1 .

Original: The gradient norm is clipped for all methods except for ALI-G, L 4 Adam and L 4 Mom (as recommended by Rolinek & Martius (2018)).
Revised: The gradient norm is clipped for all methods except for ALI-G, L 4 Adam and L 4 Mom (as recommended by Rolinek & Martius (2018)).

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: 3e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: 1e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: 3e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: 1e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: 3e+0 Learning-Rate ( η ): AdaGrad Adam RMSProp

Original: [Figure]
Revised: SGD

Original: [Table]{+SGD w/ Mom 1e+ 9e+ 6e+ 2e+ 3e1e- 4e6e- 1e+ 6e3e- 2e5e- 3e2e- 5e+ 1e+ 1e+ 2e7e- 4e5e- 2e2e- 3e7e+ 1e+ 1e+ 9e+ 6e+ 3e+ 7e7e- 2e5e- 1e+ 6e+ 2e+ 6e6e- 2e4e- 9e5e+ 1e+ 1e3e-1 1e+0 3e+0 1e+1 3e+1 1e+2 1e+3 1e+4 1e+5 1e+6 1e+7 Maximal Learning-Rate ( η ): ALI-G 8e-3 2e-3 4e-4 2e-
 6e-5 3e-5 4e-
+}
Revised: {+SGD w/ Mom 1e+ 9e+ 6e+ 2e+ 3e1e- 4e6e- 1e+ 6e3e- 2e5e- 3e2e- 5e+ 1e+ 1e+ 2e7e- 4e5e- 2e2e- 3e7e+ 1e+ 1e+ 9e+ 6e+ 3e+ 7e7e- 2e5e- 1e+ 6e+ 2e+ 6e6e- 2e4e- 9e5e+ 1e+ 1e3e-1 1e+0 3e+0 1e+1 3e+1 1e+2 1e+3 1e+4 1e+5 1e+6 1e+7 Maximal Learning-Rate ( η ): ALI-G 8e-3 2e-3 4e-4 2e-
 6e-5 3e-5 4e-
+}

Original: - 5e7e- 8e2e-{+5e-7 1e-7 1e-
 4e-8+} 2e-
Revised: {+5e-7 1e-7 1e-
 4e-8+} 2e-

Original: Results.
Revised: Results.

Original: We present the results in Figure 3.
Revised: We present the results in Figure 3.

Original: ALI-G provides accurate optimization for any η ≥ 0 . 1 ,  and is among the best performing methods by reaching an objective function of 2 .
Revised: ALI-G provides accurate optimization for any η within [10 − 1 , 10 6 ] , and is among the best performing methods by reaching an objective function of 4 .

Original: 10 − 7 .
Revised: 10 − 8 .

Original: On this task,  L 4 Adam and L 4 Mom also provide accurate and robust optimization.
Revised: On this task, RMSProp, L 4 Adam and L 4 Mom also provide accurate and robust optimization.

Original: In contrast to ALI-G and the L 4 methods, the most commonly used algorithms such as SGD, SGD with momentum and Adam are very sensitive to their main learning-rate hyper-parameter.
Revised: In contrast to ALI-G and the L 4 methods, the most commonly used algorithms such as SGD, SGD with momentum and Adam are very sensitive to their main learning-rate hyper-parameter.

Revised: Note that the difference between well-performing methods is not signiﬁcant here because these reach the numerical precision limit of single-precision ﬂoat numbers.

Original: 5. W IDE R ESIDUAL N ETWORKS ON SVHN
Revised: 5. W IDE R ESIDUAL N ETWORKS ON SVHN

Original: Setting.
Revised: Setting.

Original: The SVHN data set contains 73k training samples, 26k testing samples and 531k additional easier samples.
Revised: The SVHN data set contains 73k training samples, 26k testing samples and 531k additional easier samples.

Original: From the 73k difﬁcult training examples, we select 6k samples for validation; we use all remaining (both difﬁcult and easy) examples for training, for a total of 598k samples.
Revised: From the 73k difﬁcult training examples, we select 6k samples for validation; we use all remaining (both difﬁcult and easy) examples for training, for a total of 598k samples.

Original: We train a wide residual network 16-4 following Zagoruyko & Komodakis (2016).
Revised: We train a wide residual network 16-4 following Zagoruyko & Komodakis (2016).

Original: Method.
Revised: Method.

Original: For SGD, we use the manual schedule for the learning rate of Zagoruyko & Komodakis (2016).
Revised: For SGD, we use the manual schedule for the learning rate of Zagoruyko & Komodakis (2016).

Original: For L 4 Adam and L 4 Mom, we cross-validate the main learning-rate hyper-parameter α to be in { 0 . 0015 , 0 . 015 , 0 . 15 } ( 0 . 15 is the value recommended by Rolinek & Martius (2018)).
Revised: For L 4 Adam and L 4 Mom, we cross-validate the main learning-rate hyper-parameter α to be in { 0 . 0015 , 0 . 015 , 0 . 15 } ( 0 . 15 is the value recommended by Rolinek & Martius (2018)).

Original: For other methods, the learning rate hyper-parameter is tuned as a power of 10.
Revised: For other methods, the learning rate hyper-parameter is tuned as a power of 10.

Original: The L 2 regularization is cross-validated in { 0 . 0001 , 0 . 0005 } for all methods but ALI-G. For ALI-G, the regularization is expressed as a constraint on the L 2 -norm of the parameters, and its maximal value is set to 50 .
Revised: The L 2 regularization is cross-validated in { 0 . 0001 , 0 . 0005 } for all methods but ALI-G. For ALI-G, the regularization is expressed as a constraint on the L 2 -norm of the parameters, and its maximal value is set to 50 .

Original: SGD,
Revised: SGD,

Original: ALI-G and BPGrad usea Nesterov momentum of0.9. All methods use  a dropout rate of 0.4.
Revised: ALI-G and BPGrad use of use and a budget of 160 epochs, following (Zagoruyko & Komodakis, 2016).

Original: Results.
Revised: Results.

Original: The results are presented in Table 1.
Revised: The results are presented in Table 1.

Original: On this relatively easy task, most methods achieve about 98% test accuracy.
Revised: On this relatively easy task, most methods achieve about 98% test accuracy.

Original: Despite the cross-validation, L 4 Mom does not converge on this task.
Revised: Despite the cross-validation, L 4 Mom does not converge on this task.

Original: Even though SGD beneﬁts from a hand-designed schedule, ALI-G and other adaptive methods obtain close performance to it.
Revised: Even though SGD beneﬁts from a hand-designed schedule, ALI-G and other adaptive methods obtain close performance to it.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 5. B I -LSTM ON SNLI
Revised: 5. B I -LSTM ON SNLI

Original: Setting.
Revised: Setting.

Original: We train a Bi-LSTM of 47M parameters on the Stanford Natural Language Inference (SNLI) data set (Bowman et al., 2015).
Revised: We train a Bi-LSTM of 47M parameters on the Stanford Natural Language Inference (SNLI) data set (Bowman et al., 2015).

Original: The SNLI data set consists in 570k pairs of sentences, with each pair labeled as entailment, neutral or contradiction.
Revised: The SNLI data set consists in 570k pairs of sentences, with each pair labeled as entailment, neutral or contradiction.

Original: This large scale data set is commonly used as a pre-training corpus for transfer learning to many other natural language tasks where labeled data is scarcer (Conneau et al., 2017) – much like ImageNet is used for pre-training in computer vision.
Revised: This large scale data set is commonly used as a pre-training corpus for transfer learning to many other natural language tasks where labeled data is scarcer (Conneau et al., 2017) – much like ImageNet is used for pre-training in computer vision.

Original: We follow the protocol of Berrada et al.
Revised: We follow the protocol of Berrada et al.

Original: (2019); we also re-use their code and results for the baselines. 
Revised: (2019); we also re-use their code and results for the baselines. Method.

Original:  – compared to the recommended value of 0.15, this helped convergence and considerably improved performance.
Revised: For L 4 Adam and L 4 Mom, the main hyper-parameter α is cross-validated in { 0 . 015 , 0 . 15 } – compared to the recommended value of 0.15, this helped convergence and considerably improved performance.

Original: The SGD algorithm beneﬁts from a hand-designed schedule, where the learning-rate is decreased by 5 when the validation accuracy does not improve.
Revised: The SGD algorithm beneﬁts from a hand-designed schedule, where the learning-rate is decreased by 5 when the validation accuracy does not improve.

Original: Other methods use adaptive learningrates and do not require such schedule.
Revised: Other methods use adaptive learningrates and do not require such schedule.

Original: The value of the main hyper-parameter η is cross-validated as a power of ten for the ALI-G algorithm and for previously reported adaptive methods.
Revised: The value of the main hyper-parameter η is cross-validated as a power of ten for the ALI-G algorithm and for previously reported adaptive methods.

Original: Following the implementation by Conneau et al.
Revised: Following the implementation by Conneau et al.

Original: (2017), no L 2 regularization is used.
Revised: (2017), no L 2 regularization is used.

Original: The algorithms are evaluated with the Cross-Entropy (CE) loss and the multi-class hinge loss (SVM), except for DFW which is designed for SVMs only.
Revised: The algorithms are evaluated with the Cross-Entropy (CE) loss and the multi-class hinge loss (SVM), except for DFW which is designed for SVMs only.

Original: [Table]
Revised: For all optimization algorithms, the model is trained for 20 epochs, following (Conneau et al., 2017). Loss

Original: A d a g r a d ∗
Revised: A d a g r a d ∗

Original: A d a m ∗
Revised: A d a m ∗

Original: A M S G r a d ∗
Revised: A M S G r a d ∗

Original: B P G r a d ∗
Revised: B P G r a d ∗

Original: D F W ∗
Revised: D F W ∗

Original: L 4 A d a m L 4 M o m A L I- G ∞
Revised: L 4 A d a m L 4 M o m A L I- G ∞

Original: A L I- G
Revised: A L I- G

Original: S G D ∗ S G D †
Revised: S G D ∗ S G D †

Original: CE83.
Revised: CE

Original: [Table]
Revised: [Table]

Original: † is the result from Conneau et al.

Original: Results.
Revised: Results.

Original: We present the results in Table 2.
Revised: We present the results in Table 2.

Original: ALI-G ∞ is the only method that requires no hyperparameter for its learning-rate.
Revised: ALI-G ∞ is the only method that requires no hyperparameter for its learning-rate.

Original: Despite this, and the fact that SGD employs a learning-rate schedule that has been hand designed for good validation performance, ALI-G ∞ is still able to obtain results that are competitive with SGD.
Revised: Despite this, and the fact that SGD employs a learning-rate schedule that has been hand designed for good validation performance, ALI-G ∞ is still able to obtain results that are competitive with SGD.

Original: Moreover, ALI-G, which requires a single hyper-parameter for the learning-rate, outperforms all other methods for both the SVM and the CE loss functions.
Revised: Moreover, ALI-G, which requires a single hyper-parameter for the learning-rate, outperforms all other methods for both the SVM and the CE loss functions.

Original: 5. W IDE R ESIDUAL N ETWORKS AND D ENSELY C ONNECTED N ETWORKS ON CIFAR
Revised: 5. W IDE R ESIDUAL N ETWORKS AND D ENSELY C ONNECTED N ETWORKS ON CIFAR

Original: Setting.
Revised: Setting.

Original: We follow the methodology of Berrada et al.
Revised: We follow the methodology of Berrada et al.

Original: (2019); we also re-use their code and we reproduce their results.
Revised: (2019); we also re-use their code and we reproduce their results.

Original: We test two architectures: a Wide Residual Network (WRN) 40-4 (Zagoruyko & Komodakis, 2016) and a bottleneck DenseNet (DN) 40-40 (Huang et al., 2017).
Revised: We test two architectures: a Wide Residual Network (WRN) 40-4 (Zagoruyko & Komodakis, 2016) and a bottleneck DenseNet (DN) 40-40 (Huang et al., 2017).

Original: We use 45k samples for training and 5k for validation.
Revised: We use 45k samples for training and 5k for validation.

Original: The images are centered and normalized per channel.
Revised: The images are centered and normalized per channel.

Original: We apply standard data augmentation with random horizontal ﬂipping and random crops.
Revised: We apply standard data augmentation with random horizontal ﬂipping and random crops.

Original: AMSGrad was selected in Berrada et al.
Revised: AMSGrad was selected in Berrada et al.

Original: (2019) because it was the best adaptive method on similar tasks, outperforming in particular Adam and Adagrad.
Revised: (2019) because it was the best adaptive method on similar tasks, outperforming in particular Adam and Adagrad.

Original: In addition to the baselines from Berrada et al.
Revised: In addition to the baselines from Berrada et al.

Original: (2019), we also provide the performance of L 4 Adam, L 4 Mom, AdamW (Loshchilov & Hutter, 2019) and Yogi (Zaheer et al., 2018).
Revised: (2019), we also provide the performance of L 4 Adam, L 4 Mom, AdamW (Loshchilov & Hutter, 2019) and Yogi (Zaheer et al., 2018).

Original: Method.
Revised: Method.

Original: All optimization methods employ the cross-entropy loss, except for the DFW algorithm, which is designed to use an SVM loss.
Revised: All optimization methods employ the cross-entropy loss, except for the DFW algorithm, which is designed to use an SVM loss.

Original: For DN and WRN respectively, SGD uses the manual learning rate schedules from Huang et al.
Revised: For DN and WRN respectively, SGD uses the manual learning rate schedules from Huang et al.

Original: (2017) and Zagoruyko & Komodakis (2016).
Revised: (2017) and Zagoruyko & Komodakis (2016).

Original: Following
Revised: Following

Original: Berrada et al.
Revised: Berrada et al.

Original: (2019), the batch-size is cross-validated in { 64 , 128 , 256 } for the DN architecture, and { 128 , 256 , 512 } for the WRN architecture.
Revised: (2019), the batch-size is cross-validated in { 64 , 128 , 256 } for the DN architecture, and { 128 , 256 , 512 } for the WRN architecture.

Original: For L 4 Adam and L 4 Mom, the learning-rate hyperparameter α is cross-validated in { 0 . 015 , 0 . 15 } .
Revised: For L 4 Adam and L 4 Mom, the learning-rate hyperparameter α is cross-validated in { 0 . 015 , 0 . 15 } .

Original: For AMSGrad, AdamW, Yogi, DFW and ALI-G, the learning-rate hyper-parameter η is cross-validated as a power of 10 (in practice η ∈ { 0 . 1 , 1 } for
Revised: For AMSGrad, AdamW, Yogi, DFW and ALI-G, the learning-rate hyper-parameter η is cross-validated as a power of 10 (in practice η ∈ { 0 . 1 , 1 } for

Original: ALI-G).
Revised: ALI-G).

Original: SGD, DFW and ALI-G use a Nesterov momentum of 0.9.
Revised: SGD, DFW and ALI-G use a Nesterov momentum of 0.9.

Original: Following Berrada et al.
Revised: Following Berrada et al.

Original: for all methods but ALI-G and AdamW, the L 2 regularization is cross-validated in { 0 . 0001 , 0 . 0005 } on the WRN architecture, and is set to 0 .
Revised: for all methods but ALI-G and AdamW, the L 2 regularization is cross-validated in { 0 . 0001 , 0 . 0005 } on the WRN architecture, and is set to 0 .

Original: 0001 for the DN architecture.
Revised: 0001 for the DN architecture.

Original: For AdamW, the weight-decay is cross-validated as a power of 10.
Revised: For AdamW, the weight-decay is cross-validated as a power of 10.

Original: For ALI-G, L 2 regularization is expressed as a constraint on the norm on the vector of parameters; its maximal value is set to 100 for the WRN models,  andto 75 forthe DN ones.
Revised: For ALI-G, L 2 regularization is expressed as a constraint on the norm on the vector of parameters; its maximal value is set to 100 for the WRN models, 80 for DN on CIFAR-10 and 75 for DN on CIFAR-100.

Revised: For all optimization algorithms, the WRN model is trained for 200 epochs and the DN model for 300 epochs, following respectively (Zagoruyko & Komodakis, 2016) and (Huang et al., 2017).

Original: Results.
Revised: Results.

Original: We present the results in Table 3.
Revised: We present the results in Table 3.

Original: In this setting again, ALI-G obtains competitive performance with manually decayed SGD.
Revised: In this setting again, ALI-G obtains competitive performance with manually decayed SGD.

Original: ALI-G largely outperforms AMSGrad, AdamW and Yogi.
Revised: ALI-G largely outperforms AMSGrad, AdamW and Yogi.

Original: In addition, it signiﬁcantly bridges the gap between DFW and SGD on CIFAR-10 with the WRN model, and on CIFAR-100 with the DN one.
Revised: In addition, it signiﬁcantly bridges the gap between DFW and SGD on CIFAR-10 with the WRN model, and on CIFAR-100 with the DN one.

Original: [Table]
Revised: [Table]

Original: A M S G r a d A d a m W Y o g
Revised: L

Original: i DF W
Revised: [Table] D

Original: L 4 A d a m L 4 M o m A L I- G S G D
Revised: [Table]

Original: S G D †
Revised: D †

Original: [Table]
Revised: [Table]

Revised: , SGD beneﬁts from a hand-designed schedule for its learning-rate.

Revised: In black, adaptive methods, including ALI-G, have a single hyperparameter for their learning-rate.

Revised: SGD † refers to the result from Zagoruyko & Komodakis (2016).

Revised: Each reported result is an average over three independent runs; the standard deviations are reported in Appendix (they are at most 0 .

Revised: 3 for ALI-G and SGD).

Original: 5. T RAINING P ERFORMANCE
Revised: 5. T RAINING P ERFORMANCE

Original: The experiments have so far focused on testing accuracy (except for the DNC task), because that is the main metric driving practitioners’ choice of optimization algorithm.
Revised: The experiments have so far focused on testing accuracy (except for the DNC task), because that is the main metric driving practitioners’ choice of optimization algorithm.

Original: In this section, we empirically assess the performance of ALI-G and its competitors in terms of training objective.
Revised: In this section, we empirically assess the performance of ALI-G and its competitors in terms of training objective.

Original: In order to have comparable objective functions, the L 2 regularization is deactivated.
Revised: In order to have comparable objective functions, the L 2 regularization is deactivated.

Original: We do not use dropout.
Revised: We do not use dropout.

Original: The learning-rate is selected as a power of ten for best ﬁnal objective value, and the batch-size is set to its default value.
Revised: The learning-rate is selected as a power of ten for best ﬁnal objective value, and the batch-size is set to its default value.

Revised: All methods use a ﬁx budget of 160 epochs for WRN-SVHN, 200 epochs for WRNCIFAR and 300 epochs for DN-CIFAR, following (Zagoruyko & Komodakis, 2016) and (Huang et al., 2017).

Original: The L 4 methods diverge on CIFAR-100 in this setting.
Revised: The L 4 methods diverge on CIFAR-100 in this setting.

Original: For clarity, we only display the performance of SGD, Adam, Adagrad and ALI-G. Here SGD uses a constant learning-rate to emphasize the need for adaptivity.
Revised: For clarity, we only display the performance of SGD, Adam, Adagrad and ALI-G (DFW does not support the cross-entropy loss). Here SGD uses a constant learning-rate to emphasize the need for adaptivity.

Original: Therefore all methods use one hyper-parameter for their learningrate.
Revised: Therefore all methods use one hyper-parameter for their learning-rate.

Original: As can be seen, ALI-G provides better training performance than the baseline algorithms on all tasks.
Revised: As can be seen, ALI-G provides better training performance than the baseline algorithms on all tasks.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We hope that the ALI-G algorithm is a helpful step towards efﬁcient and reliable training of deep neural networks.
Revised: We hope that the ALI-G algorithm is a helpful step towards efﬁcient and reliable training of deep neural networks.

Original: ALI-G is readily applicable to a broad range of applications in deep learning where the model can interpolate the data.
Revised: ALI-G is readily applicable to a broad range of applications in deep learning where the model can interpolate the data.

Original: When that is not the case however, it would be interesting to design new algorithms that adapt the minimum f (cid:63) online while requiring few hyper-parameters.
Revised: When that is not the case however, it would be interesting to design new algorithms that adapt the minimum f (cid:63) online while requiring few hyper-parameters.

Original: This could be achieved for instance by building upon the works of Nedi´c & Bertsekas (2001b) and Rolinek & Martius (2018)
Revised: This could be achieved for instance by building upon the works of Nedi´c & Bertsekas (2001b) and Rolinek & Martius (2018).

Original: A L OCAL I NTERPRETATION OF THE P OLYAK S TEP -S IZE
Revised: A L OCAL I NTERPRETATION OF THE P OLYAK S TEP -S IZE

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose that the problem is unconstrained: Ω = R p .
Revised: Suppose that the problem is unconstrained: Ω = R p .

Original: Let w t
Revised: Let w t

Original: +

Revised: +

Original: =
Revised: =

Original: w t − f ( w t ) − f (cid:63) (cid:107)∇ f ( w t ) (cid:107) 2 ∇ f
Revised: w t − f ( w t ) − f (cid:63) (cid:107)∇ f ( w t ) (cid:107) 2 ∇ f

Original: ( w t ) .
Revised: ( w t ) .

Original: Then w t +1 veriﬁes: [Equation] t +1 = arg
Revised: Then w t +1 veriﬁes: w t +1 = argmin w ∈ R p (cid:107) w − w t (cid:107) subject to: f ( w t ) +

Original: [Equation]
Revised: [Equation]

Original: R p (cid:107) w
Revised: f (

Original: [Equation]
Revised: [Equation]

Original: (cid:107) subject to
Revised: (cid:62) ( w

Original: [Equation]
Revised: [Equation]

Original:  f ( [Equation] (6) where we remind that f (cid:63) is the minimum of f , and w (cid:55)→ f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) is the linearization of f at w t .
Revised: w t ) = f (cid:63) , (6) where we remind that f (cid:63) is the minimum of f , and w (cid:55)→ f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) is the linearization of f at w t .

Original: In other words, w t +1 is the closest point to w t that lies on the hyper-plane f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) =
Revised: In other words, w t +1 is the closest point to w t that lies on the hyper-plane f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) =

Original: f (cid:63) .
Revised: f (cid:63) .

Original: Proof: See Appendix D.
Revised: Proof: See Appendix D.

Original: B C ONVERGENCE R ESULTS
Revised: B C ONVERGENCE R ESULTS

Original: Before we detail our convergence results, we introduce the notions of uniform lower bound and ε -interpolation.
Revised: Before we detail our convergence results, we introduce the notions of uniform lower bound and ε -interpolation.

Original: B. N OTATION
Revised: B. N OTATION

Original: Intuitively, a uniform lower bound on the problem ( P ) is a lower bound on all loss functions (cid:96) z on their unconstrained domain R p .
Revised: Intuitively, a uniform lower bound on the problem ( P ) is a lower bound on all loss functions (cid:96) z on their unconstrained domain R p .

Original: We formalize this below:
Revised: We formalize this below:

Original: Deﬁnition 1 (Uniform Lower Bound) .
Revised: Deﬁnition 1 (Uniform Lower Bound) .

Original: We say that B is a uniform lower bound on ( P ) if:
Revised: We say that B is a uniform lower bound on ( P ) if:

Original: [Equation]
Revised: [Equation]

Original: Note that in the main paper, we have used the special case B = 0 as the uniform lower bound.
Revised: Note that in the main paper, we have used the special case B = 0 as the uniform lower bound.

Original: The deﬁnition above makes B a useful statistic to analyze the behavior of each loss function (cid:96) z around w (cid:63) , in a uniform way (that is, independently of z ).
Revised: The deﬁnition above makes B a useful statistic to analyze the behavior of each loss function (cid:96) z around w (cid:63) , in a uniform way (that is, independently of z ).

Original: The quality of a uniform lower bound B can be quantiﬁed by the notion of ε -interpolation:
Revised: The quality of a uniform lower bound B can be quantiﬁed by the notion of ε -interpolation:

Original: Deﬁnition 2 ( ε -Interpolation) .
Revised: Deﬁnition 2 ( ε -Interpolation) .

Original: Let B be a uniform lower bound on ( P ), w (cid:63) be a solution of ( P ) and ε ≥ 0 be a non-negative number.
Revised: Let B be a uniform lower bound on ( P ), w (cid:63) be a solution of ( P ) and ε ≥ 0 be a non-negative number.

Original: Then we say that w (cid:63) is an ε -interpolation for (( P ), B ) if:
Revised: Then we say that w (cid:63) is an ε -interpolation for (( P ), B ) if:

Original: [Equation]
Revised: [Equation]

Original: By taking the expectation over equation (8), we can see that if w (cid:63) is an ε -interpolation for (( P ), B ), then we immediately have: f (cid:63) − ε ≤ B ≤ f (cid:63) .
Revised: By taking the expectation over equation (8), we can see that if w (cid:63) is an ε -interpolation for (( P ), B ), then we immediately have: f (cid:63) − ε ≤ B ≤ f (cid:63) .

Original: In other words, B is also an approximation of f (cid:63) by below, and its quality is quantiﬁed by ε .
Revised: In other words, B is also an approximation of f (cid:63) by below, and its quality is quantiﬁed by ε .

Original: We further note that f (cid:63) does not satisfy the deﬁnition of a uniform lower bound in the general case.
Revised: We further note that f (cid:63) does not satisfy the deﬁnition of a uniform lower bound in the general case.

Original: However when f (cid:63) actually is a uniform lower bound, for any solution w (cid:63) of ( P ), w (cid:63) is a ε = 0 -interpolation for (( P ), f (cid:63) ).
Revised: However when f (cid:63) actually is a uniform lower bound, for any solution w (cid:63) of ( P ), w (cid:63) is a ε = 0 -interpolation for (( P ), f (cid:63) ).

Original: In the general case where B may be different from 0 , the ALI-G step-size can be deﬁned as:
Revised: In the general case where B may be different from 0 , the ALI-G step-size can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We now turn to our convergence results, which give convergence rates in three settings: convex and
Revised: We now turn to our convergence results, which give convergence rates in three settings: convex and

Original: Lipschitz functions, convex and smooth function, and strongly convex and smooth functions.
Revised: Lipschitz functions, convex and smooth function, and strongly convex and smooth functions.

Original: In each setting, we analyze three regimes which complement each other: no (inﬁnite) maximal learning-rate, large maximal learning-rate and small maximal learning-rate.
Revised: In each setting, we analyze three regimes which complement each other: no (inﬁnite) maximal learning-rate, large maximal learning-rate and small maximal learning-rate.

Original: B. L IPSCHITZ C ONVEX F UNCTIONS
Revised: B. L IPSCHITZ C ONVEX F UNCTIONS

Original: Theorem 2.
Revised: Theorem 2.

Original: [Convex and Lipschitz] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: [Convex and Lipschitz] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ).
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ).

Original: Then ALI-G ∞ applied to f satisﬁes:
Revised: Then ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.3.
Revised: Proof: See Appendix D.3.

Original: Theorem 3.
Revised: Theorem 3.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ).
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ).

Original: We further assume that η > εδ .
Revised: We further assume that η > εδ .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.4.
Revised: Proof: See Appendix D.4.

Original: We note that for very large values of η ( η → ∞ ), Theorem 3 gives the exact same result as Theorem 2.
Revised: We note that for very large values of η ( η → ∞ ), Theorem 3 gives the exact same result as Theorem 2.

Original: However when η is small, the convergence error of Theorem 3 is large.
Revised: However when η is small, the convergence error of Theorem 3 is large.

Original: This is corrected in the following result which is informative in the regime where η is small:
Revised: This is corrected in the following result which is informative in the regime where η is small:

Original: Theorem 4.
Revised: Theorem 4.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ).
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ).

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.5.
Revised: Proof: See Appendix D.5.

Original: B. S MOOTH C ONVEX F UNCTIONS
Revised: B. S MOOTH C ONVEX F UNCTIONS

Original: We now tackle the convex and β -smooth case.
Revised: We now tackle the convex and β -smooth case.

Original: Our proof techniques naturally produce the separation η ≥ 12 β and η ≤ 12 β .
Revised: Our proof techniques naturally produce the separation η ≥ 12 β and η ≤ 12 β .

Original: Whenever η ≥ 12 β , the convergence result is exactly the same as when η → ∞ .
Revised: Whenever η ≥ 12 β , the convergence result is exactly the same as when η → ∞ .

Original: When η ≤ 12 β , the speed of convergence is limited by the value of η .
Revised: When η ≤ 12 β , the speed of convergence is limited by the value of η .

Original: Theorem 5.
Revised: Theorem 5.

Original: [Convex and Smooth]
Revised: [Convex and Smooth]

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .

Original: Then ALI-G ∞ applied to f satisﬁes:
Revised: Then ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.6.
Revised: Proof: See Appendix D.6.

Original: Theorem 6.
Revised: Theorem 6.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≥ 12 β .
Revised: Further assume that η ≥ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.7.
Revised: Proof: See Appendix D.7.

Original: Theorem 7.
Revised: Theorem 7.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≤ 12 β .
Revised: Further assume that η ≤ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.8.
Revised: Proof: See Appendix D.8.

Original: B. S MOOTH AND S TRONGLY C ONVEX F UNCTIONS
Revised: B. S MOOTH AND S TRONGLY C ONVEX F UNCTIONS

Original: Finally, we consider the α -strongly convex and β -smooth case.
Revised: Finally, we consider the α -strongly convex and β -smooth case.

Original: Again, our proof yields a natural separation between η ≥ 12 β and η ≤ 12 β .
Revised: Again, our proof yields a natural separation between η ≥ 12 β and η ≤ 12 β .

Original: In a similar way to the β -smooth case, when η ≥ 12 β ,
Revised: In a similar way to the β -smooth case, when η ≥ 12 β ,

Original: Theorem 9 gives the exact same result as η → ∞ .
Revised: Theorem 9 gives the exact same result as η → ∞ .

Original: And when η ≤ 12 β , the rate of convergence given by Theorem 10 is limited by the value of η .
Revised: And when η ≤ 12 β , the rate of convergence given by Theorem 10 is limited by the value of η .

Original: Theorem 8.
Revised: Theorem 8.

Original: [Strongly Convex and Smooth] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: [Strongly Convex and Smooth] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .

Original: Then
Revised: Then

Original: ALI-G ∞ applied to f satisﬁes:
Revised: ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: In other words, f approximately converges to f (cid:63) at a rate of O (exp( − αT/ 8 β )) .
Revised: In other words, f approximately converges to f (cid:63) at a rate of O (exp( − αT/ 8 β )) .

Original: Proof: See Appendix D.9.
Revised: Proof: See Appendix D.9.

Original: Theorem 9.
Revised: Theorem 9.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≥ 12 β .
Revised: Further assume that η ≥ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.10.
Revised: Proof: See Appendix D.10.

Original: Theorem 10.
Revised: Theorem 10.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≤ 12 β .
Revised: Further assume that η ≤ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: See Appendix D.11.
Revised: Proof: See Appendix D.11.

Original: C O N THE N EED FOR A M AXIMAL L EARNING -R ATE FOR N ON -C ONVEXITY
Revised: C O N THE N EED FOR A M AXIMAL L EARNING -R ATE FOR N ON -C ONVEXITY

Original: The Restricted Secant Inequality (RSI) is a milder assumption than convexity.
Revised: The Restricted Secant Inequality (RSI) is a milder assumption than convexity.

Original: It can be deﬁned as follows:
Revised: It can be deﬁned as follows:

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Let f : R p
Revised: Let f : R p

Original: → R be a lower-bounded differentiable function achieving its minimum at w (cid:63) .
Revised: → R be a lower-bounded differentiable function achieving its minimum at w (cid:63) .

Original: We say that f satisﬁes the RSI if there exists α > 0 such that:
Revised: We say that f satisﬁes the RSI if there exists α > 0 such that:

Original: [Equation]
Revised: [Equation]

Original: The RSI is sometimes used to prove convergence of optimization algorithms without assuming convexity (Vaswani et al., 2019b).
Revised: The RSI is sometimes used to prove convergence of optimization algorithms without assuming convexity (Vaswani et al., 2019b).

Original: As we prove below, the Polyak step-size may not convergence under the RSI assumption, even in a non-stochastic setting with the exact minimum known.
Revised: As we prove below, the Polyak step-size may not convergence under the RSI assumption, even in a non-stochastic setting with the exact minimum known.

Original: We introduce the function f : w ∈ [ − 35 ; 35 ] (cid:55)→ w 2 − | w | 3 .
Revised: We introduce the function f : w ∈ [ − 3 5 ; 3 5 ] (cid:55)→ w 2 − | w | 3 .

Original: We restrict our domain of study to [ − 35 ; 35 ] for simplicity purposes – an extension to R can easily be constructed by extending f .
Revised: We restrict our domain of study to [ − 3 5 ; 3 5 ] for simplicity purposes – an extension to R can easily be constructed by extending f .

Original: We will ﬁrst show that f fulﬁlls the RSI assumption, and then that it oscillates between two points for a well chosen initialization.
Revised: We will ﬁrst show that f fulﬁlls the RSI assumption, and then that it oscillates between two points for a well chosen initialization.

Original: Let us show that f satisﬁes the RSI with α = 15 .
Revised: Let us show that f satisﬁes the RSI with α = 15 .

Original: First we note that f achieves its minimum at w (cid:63) = 0 , and
Revised: First we note that f achieves its minimum at w (cid:63) = 0 , and

Original: that f ( w (cid:63) ) = 0 .
Revised: that f ( w (cid:63) ) = 0 .

Original: In addition, we introduce the sign function σ ( w ) , which is equal to 1 if w ≥ 0 ,
Revised: In addition, we introduce the sign function σ ( w ) , which is equal to 1 if w ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: and − 1 otherwise.
Revised: and − 1 otherwise.

Original: Now let w ∈ [ − 35 ; 35 ] .
Revised: Now let w ∈ [ − 35 ; 35 ] .

Original: Then we have that:
Revised: Then we have that:

Original: [Equation]
Revised: [Equation]

Original: Now let us show that if we apply gradient descent with a Polyak step-size to f , with starting point w 0 = − 35 , we obtain w 1 = 35 .
Revised: Now let us show that if we apply gradient descent with a Polyak step-size to f , with starting point w 0 = − 35 , we obtain w 1 = 35 .

Original: This will prove oscillation of the iterates by symmetry of the problem.
Revised: This will prove oscillation of the iterates by symmetry of the problem.

Original: Let w 0 = − 35 .
Revised: Let w 0 = − 35 .

Original: Then we have f ( w 0 )
Revised: Then we have f ( w 0 )

Original: =
Revised: =

Original: 925 − 27125 = 18125 .
Revised: 925 − 27125 = 18125 .

Original: Furthermore, ∇ f ( w 0 ) =
Revised: Furthermore, ∇ f ( w 0 ) =

Original: 2( − 35 ) + 3( 925 ) =
Revised: 2( − 35 ) + 3( 925 ) =

Original: − 325 .
Revised: − 325 .

Original: Therefore:
Revised: Therefore:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D P ROOFS D. P ROPOSITION
Revised: D P ROOFS D. P ROPOSITION

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose that the problem is unconstrained: Ω = R p .
Revised: Suppose that the problem is unconstrained: Ω = R p .

Original: Let w t
Revised: Let w t

Original: +

Revised: +

Original: =
Revised: =

Original: w t − f ( w t ) − f (cid:63) (cid:107)∇ f ( w t ) 
Revised: w t − f ( w t ) − f (cid:63) (cid:107)∇ f ( w t ) (cid:107) 2 ∇ f

Original: [Equation] 2 ∇ f ( w t ) .
Revised: ( w t ) .

Original: Then w t +1 veriﬁes:
Revised: Then w t +1 veriﬁes:

Original: [Equation]
Revised: [Equation]

Original: w ∈ R p (cid:107) w − w t (cid:107) subject to: f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) =
Revised: subject to

Original: f (cid:63) , (6) where we remind that f (cid:63) is the minimum of f , and w (cid:55)→ f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) is the linearization of f at w t .
Revised: [Equation] [Equation] where we remind that f (cid:63) is the minimum of f , and w (cid:55)→ f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) is the linearization of f at w t .

Original: In other words, w t +1 is the closest point to w t that lies on the hyper-plane f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) =
Revised: In other words, w t +1 is the closest point to w t that lies on the hyper-plane f ( w t ) + ∇ f ( w t ) (cid:62) ( w − w t ) =

Original: f (cid:63) .
Revised: f (cid:63) .

Original: Proof: First we show that w t +1 satisﬁes the linear equality constraint:
Revised: Proof: First we show that w t +1 satisﬁes the linear equality constraint:

Original: [Equation]
Revised: [Equation]

Original: Now let us show that it has a minimal distance to w t .
Revised: Now let us show that it has a minimal distance to w t .

Original: We take ˆ w ∈ R p a solution of the linear equality constraint, and we will show that (cid:107) w t +1 − w t (cid:107) ≤ (cid:107) ˆ w − w t (cid:107) .
Revised: We take ˆ w ∈ R p a solution of the linear equality constraint, and we will show that (cid:107) w t +1 − w t (cid:107) ≤ (cid:107) ˆ w − w t (cid:107) .

Original: By deﬁnition, we have that ˆ w satisﬁes:
Revised: By deﬁnition, we have that ˆ w satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Now we can write:
Revised: Now we can write:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Proposition 1.
Revised: Proposition 1.

Original: [Proximal Interpretation] Suppose that Ω = R p and let δ = 0 .
Revised: [Proximal Interpretation] Suppose that Ω = R p and let δ = 0 .

Original: We consider the update performed by SGD: w SGD t +1 =
Revised: We consider the update performed by SGD: w SGD t +1 =

Original: w t − η t ∇ (cid:96) z t ( w t ) ; and the update performed by ALI-G: w ALI-G t +1 =
Revised: w t − η t ∇ (cid:96) z t ( w t ) ; and the update performed by ALI-G: w ALI-G t +1 =

Original: w t − γ t ∇ (cid:96) z t ( w t ) , where γ t = min (cid:110)
Revised: w t − γ t ∇ (cid:96) z t ( w t ) , where γ t = min (cid:110)

Original: [Equation]
Revised: [Equation]

Original: zt
Revised: zt

Original: [Equation]
Revised: [Equation]

Original: w t ) (cid:107)∇
Revised: w t ) (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: zt ( w t ) (cid:107) [Equation] + δ , η (cid:111) .
Revised: zt ( w t ) (cid:107) 2 + δ , η (cid:111) .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: We tackle the slightly more general case where B may be different from zero.
Revised: Proof: We tackle the slightly more general case where B may be different from zero.

Original: In order to make the notation simpler, we use d t (cid:44) ∇ (cid:96) z t ( w t ) and l t (cid:44) (cid:96) z t ( w t ) − B .
Revised: In order to make the notation simpler, we use d t (cid:44) ∇ (cid:96) z t ( w t ) and l t (cid:44) (cid:96) z t ( w t ) − B .

Original: First, let us consider d t = 0 .
Revised: First, let us consider d t = 0 .

Original: Then we choose γ t = 

Revised: Then we choose γ t =

Original: and it is clear that w t +1 =
Revised: and it is clear that w t +1 =

Original: w t − ηγ t d t = w t is the optimal solution of problem (5).
Revised: w t − ηγ t d t = w t is the optimal solution of problem (5).

Original: We now assume d t (cid:54) = 0 .
Revised: We now assume d t (cid:54) = 0 .

Original: We can successively re-write the proximal problem (5) as :
Revised: We can successively re-write the proximal problem (5) as :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The inner problem is now smooth in w and υ .
Revised: The inner problem is now smooth in w and υ .

Original: We write its KKT conditions:
Revised: We write its KKT conditions:

Original: [Equation]
Revised: [Equation]

Original: We plug in these results and obtain:
Revised: We plug in these results and obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is a one-dimensional quadratic problem in ν .
Revised: This is a one-dimensional quadratic problem in ν .

Original: It can be solved in closed-form by ﬁnding the global maximum of the quadratic objective
Revised: It can be solved in closed-form by ﬁnding the global maximum of the quadratic objective

Original: [Equation]
Revised: [Equation]

Original: and projecting the solution on [0 , 1] .
Revised: and projecting the solution on [0 , 1] .

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Since d t (cid:54) = 0 and η (cid:54) = 0 , this gives the optimal solution:
Revised: Since d t (cid:54) = 0 and η (cid:54) = 0 , this gives the optimal solution:

Original: [Equation]
Revised: [Equation]

Original: since l t , η, (cid:107) d t (cid:107) 2 ≥ 0 .
Revised: since l t , η, (cid:107) d t (cid:107) 2 ≥ 0 .

Original: Plugging this back in the KKT conditions, we obtain that the solution w t +1 of the primal problem can be written as:
Revised: Plugging this back in the KKT conditions, we obtain that the solution w t +1 of the primal problem can be written as:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: [Convex and Lipschitz] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: [Convex and Lipschitz] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ).
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ).

Original: Then ALI-G ∞ applied to f satisﬁes:
Revised: Then ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: We consider the update at time t , which we condition on the draw of z t ∈ Z :
Revised: We consider the update at time t , which we condition on the draw of z t ∈ Z :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We re-write this inequality as:
Revised: We re-write this inequality as:

Original: [Equation]
Revised: [Equation]

Original: We can now use the Cauchy-Schwarz inequality to bound the sum over the iterations:
Revised: We can now use the Cauchy-Schwarz inequality to bound the sum over the iterations:

Original: [Equation]
Revised: [Equation]

Original: Therefore we can write:
Revised: Therefore we can write:

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 , (35) which yields:
Revised: (cid:107) 2 , (35) which yields:

Original: [Equation]
Revised: [Equation]

Original: We can now take the expectation over the z t :
Revised: We can now take the expectation over the z t :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Dividing by T + 1 and exploiting convexity of f
Revised: Dividing by T + 1 and exploiting convexity of f

Original: [Equation]
Revised: [Equation]

Original: we ﬁnally get:
Revised: we ﬁnally get:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 3.
Revised: Theorem 3.

Original: We assume that X is a convex set
Revised: We assume that X is a convex set

Original: [Equation]
Revised: [Equation]

Original: and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ).
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ).

Original: We further assume that η > εδ .
Revised: We further assume that η > εδ .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: We consider the update at time t
Revised: We consider the update at time t

Original: [Equation]
Revised: [Equation]

Original: which we condition on the draw of z t ∈ Z :
Revised: which we condition on the draw of z t ∈ Z :

Original: [Equation]
Revised: [Equation]

Original: (because γ
Revised: (because γ

Original: [Equation]
Revised: [Equation]

Original: because (cid:96) z
Revised: because (cid:96) z

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: convexity of
Revised: convexity of

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now consider different cases, according to the value that γ t takes: γ t =
Revised: We now consider different cases, according to the value that γ t takes: γ t =

Original: [Equation]
Revised: [Equation]

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: suppose that γ t =
Revised: suppose that γ t =

Original: [Equation]
Revised: [Equation]

Original: Then we can follow the proof of Theorem 2 to obtain:
Revised: Then we can follow the proof of Theorem 2 to obtain:

Original: [Equation]
Revised: [Equation]

Original: Now suppose γ t = η
Revised: Now suppose γ t = η

Original: and (cid:96)
Revised: and (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We can use γ
Revised: We can use γ

Original: [Equation]
Revised: [Equation]

Original: write:
Revised: write:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality has used
Revised: where the last inequality has used

Original: [Equation]
Revised: [Equation]

Original: B ≥ 0 .
Revised: B ≥ 0 .

Original: Therefore we are exactly in the same situation as the ﬁrst case (where we used γ t = (cid:96) zt ( w t )
Revised: Therefore we are exactly in the same situation as the ﬁrst case (where we used γ t = (cid:96) zt ( w t )

Original: [Equation]
Revised: [Equation]

Original: z t ( w t )
Revised: z t ( w t )

Original: [Equation]
Revised: [Equation]

Original: 2 + δ ), and thus we have again:
Revised: 2 + δ ), and thus we have again:

Original: [Equation]
Revised: [Equation]

Original: Now suppose that γ t = η
Revised: Now suppose that γ t = η

Original: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .
Revised: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .

Original: The inequality (39) gives:
Revised: The inequality (39) gives:

Original: [Equation]
Revised: [Equation]

Original: z t ( w t )
Revised: z t [Equation] w t )

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: t ( w (cid:63) ))
Revised: t ( w (cid:63) ))

Original: + γ t ε.
Revised: + γ t ε.

Original: (deﬁnition of ε , γ t ≥ 0 )
Revised: (deﬁnition of ε , γ t ≥ 0 )

Original: [Equation]
Revised: [Equation]

Original: (cid:107)∇

Original: [Equation]

Original: t ( w t
Revised: t ( w t

Original: [Equation]
Revised: [Equation]

Original: δ [Equation] (because γ t ≤(cid:96)
Revised: (because γ t ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: , (because (cid:96) z
Revised: , (because (cid:96) z

Original: [Equation]
Revised: [Equation]

Original: w(cid:63)
Revised: w

Original: [Equation]
Revised: [Equation]

Original: We now introduce I T and J T as follows:
Revised: We now introduce I T and J T as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, by combining inequalities (40)
Revised: Then, by combining inequalities (40)

Original: [Equation]
Revised: [Equation]

Original: (42) and (43)
Revised: (42) and (43)

Original: [Equation]
Revised: [Equation]

Original: and using a telescopic sum
Revised: and using a telescopic sum

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: Using (cid:107) w T +1 − w (cid:63) (cid:107) 2 ≥ 0 , we obtain:
Revised: Using (cid:107) w T +1 − w (cid:63) (cid:107) 2 ≥ 0 , we obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In particular, the inequality (46) gives that:
Revised: In particular, the inequality (46) gives that:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for every t ∈ I T , we have ( (cid:96) z t ( w t ) − (cid:96)
Revised: Furthermore, for every t ∈ I T , we have ( (cid:96) z t ( w t ) − (cid:96)

Original: [Equation]
Revised: [Equation]

Original: ( w (cid:63) )) ≥ 0 , which yields (cid:0) η − εδ (cid:1) (cid:80) t ∈I T (
Revised: ( w (cid:63) )) ≥ 0 , which yields (cid:0) η − εδ (cid:1) (cid:80) t ∈I T (

Original: [Equation]
Revised: [Equation]

Original: ( w
Revised: ( w

Original: [Equation]
Revised: [Equation]

Original: − (cid:96) z t ( w (cid:63) )) ≥ 0 since η > (cid:15)δ .
Revised: − (cid:96) z t ( w (cid:63) )) ≥ 0 since η > (cid:15)δ .

Original: Thus the inequality (46) also gives:
Revised: Thus the inequality (46) also gives:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the Cauchy-Schwarz inequality, we can further write:
Revised: Using the Cauchy-Schwarz inequality, we can further write:

Original: [Equation]
Revised: [Equation]

Original: Therefore we have:
Revised: Therefore we have:

Original: [Equation]
Revised: [Equation]

Original: We can now put together inequalities (47) and (49) by writing:
Revised: We can now put together inequalities (47) and (49) by writing:

Original: [Equation]
Revised: [Equation]

Original: Dividing by T + 1 and taking the expectation
Revised: Dividing by T + 1 and taking the expectation

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 4.
Revised: Theorem 4.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and C -Lipschitz.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ).
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ).

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: We consider the update at time t , which we condition on the draw of z t ∈ Z .
Revised: We consider the update at time t , which we condition on the draw of z t ∈ Z .

Original: We re-use the inequality (39) from the proof of Theorem 3:
Revised: We re-use the inequality (39) from the proof of Theorem 3:

Original: [Equation]
Revised: [Equation]

Original: We consider again different cases, according to the value of γ t and the sign of (cid:96) z t ( w
Revised: We consider again different cases, according to the value of γ t and the sign of (cid:96) z t ( w

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: w (cid:63) ) .
Revised: w (cid:63) ) .

Original: Suppose that (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≤ 0 .
Revised: Suppose that (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≤ 0 .

Original: Then the inequality (53) gives:
Revised: Then the inequality (53) gives:

Original: [Equation]
Revised: [Equation]

Original: Now suppose (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 and γ t = η .
Revised: Now suppose (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 and γ t = η .

Original: Then the inequality (53) gives:
Revised: Then the inequality (53) gives:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: suppose that (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 and γ t = (cid:96) zt ( w t
Revised: suppose that (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 and γ t = (cid:96) zt ( w t

Original: [Equation]
Revised: [Equation]

Original: − B (cid:107)∇ (cid:96) zt ( w t ) (cid:107) 2 + δ .
Revised: − B (cid:107)∇ (cid:96) zt ( w t ) (cid:107) 2 + δ .

Original: Then the inequality (53) gives:
Revised: Then the inequality (53) gives:

Original: [Equation]
Revised: [Equation]

Original: We now introduce I T and J T as follows:
Revised: We now introduce I T and J T as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, by combining inequalities (54), (55) and (56), and using a telescopic sum, we obtain:
Revised: Then, by combining inequalities (54), (55) and (56), and using a telescopic sum, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Using (cid:107) w T +1 − w (cid:63) (cid:107) 2 ≥ 0 , we obtain:
Revised: Using (cid:107) w T +1 − w (cid:63) (cid:107) 2 ≥ 0 , we obtain:

Original: [Equation]
Revised: [Equation]

Original: We now take the expectation and obtain:
Revised: We now take the expectation and obtain:

Original: [Equation]
Revised: [Equation]

Original: Since E [ U ] 2 ≤ E [ U 2 ] for any real-valued random variable, we can write:
Revised: Since E [ U ] 2 ≤ E [ U 2 ] for any real-valued random variable, we can write:

Original: [Equation]
Revised: [Equation]

Original: Since each f ( w t ) − f (cid:63) ≥
Revised: Since each f ( w t ) − f (cid:63) ≥

Original: [Equation]
Revised: [Equation]

Original: the inequality (61) gives that:
Revised: the inequality (61) gives that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and:
Revised: and:

Original: [Equation]
Revised: [Equation]

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we can further write:
Revised: we can further write:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore we have: (cid:88)
Revised: Therefore we have: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can now put together inequalities (62) and (65) by writing:
Revised: We can now put together inequalities (62) and (65) by writing:

Original: T

Original: [Equation]
Revised: [Equation]

Revised: t ∈I T

Original: [Equation]
Revised: [Equation]

Original: t ∈I T f
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Dividing by T + 1 , we obtain: f
Revised: Dividing by T + 1 , we obtain: f

Original: [Equation]
Revised: [Equation]

Original: ( f is
Revised: f is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Lemma 1.
Revised: Lemma 1.

Original: Let z ∈ Z .
Revised: Let z ∈ Z .

Original: Assume that (cid:96) z is convex, β -smooth and is lower-bounded on R p by B ∈ R .
Revised: Assume that (cid:96) z is convex, β -smooth and is lower-bounded on R p by B ∈ R .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: Let w ∈ R p and suppose that (cid:96) z reaches its inﬁmum at w ∈ ( R ∪ {−∞ , + ∞} ) p .
Revised: Proof: Let w ∈ R p and suppose that (cid:96) z reaches its inﬁmum at w ∈ ( R ∪ {−∞ , + ∞} ) p .

Original: First, let us consider the case w ∈ R p .
Revised: First, let us consider the case w ∈ R p .

Original: Then by Lemma 3.5 of Bubeck (2015), we have:
Revised: Then by Lemma 3.5 of Bubeck (2015), we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore we can write:
Revised: Therefore we can write:

Original: [Equation]
Revised: [Equation]

Original: Now let us assume that w / ∈ R p .
Revised: Now let us assume that w / ∈ R p .

Original: Then we can construct a sequence ( w k ) k ∈ N ∈ ( R p ) N that converges to w .
Revised: Then we can construct a sequence ( w k ) k ∈ N ∈ ( R p ) N that converges to w .

Original: Since (cid:96) z and ∇ (cid:96) z are continuous functions (they are respectively convex and smooth), we have:
Revised: Since (cid:96) z and ∇ (cid:96) z are continuous functions (they are respectively convex and smooth), we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore the previous case gives the wanted result by using w k in place of w and then taking the limit k → ∞ .
Revised: Therefore the previous case gives the wanted result by using w k in place of w and then taking the limit k → ∞ .

Original: Lemma 2.
Revised: Lemma 2.

Original: Let z ∈ Z .
Revised: Let z ∈ Z .

Original: Assume that (cid:96) z is convex, β -smooth and is lower-bounded on R p by B ∈ R .
Revised: Assume that (cid:96) z is convex, β -smooth and is lower-bounded on R p by B ∈ R .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: Let w ∈ R p .
Revised: Let w ∈ R p .

Original: We apply Lemma 1 and we write successively:
Revised: We apply Lemma 1 and we write successively:

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.
Revised: Theorem 5.

Original: [Convex and Smooth]
Revised: [Convex and Smooth]

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B )
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B )

Original: [Equation]
Revised: [Equation]

Original: and that δ > 2 βε .
Revised: and that δ > 2 βε .

Original: Then ALI-G ∞ applied to f satisﬁes:
Revised: Then ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: We consider the update at time t , which we condition on the draw of z t ∈ Z :
Revised: We consider the update at time t , which we condition on the draw of z t ∈ Z :

Original: [Equation]
Revised: [Equation]

Original: We now lower
Revised: We now lower

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: z t ( w (cid:63) ) − B ) individually.
Revised: z t ( w (cid:63) ) − B ) individually.

Original: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) , for which we distinguish two cases according to its sign:
Revised: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) , for which we distinguish two cases according to its sign:

Original: Suppose ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) ≥ 0 .
Revised: Suppose ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) ≥ 0 .

Original: Then we can write:
Revised: Then we can write:

Original: [Equation]
Revised: [Equation]

Original: deﬁnition of
Revised: deﬁnition of

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now suppose (
Revised: Now suppose (

Original: [Equation]
Revised: [Equation]

Original: z t
Revised: z t

Original: [Equation]
Revised: [Equation]

Original: w t ) −
Revised: w t ) −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: )) ≤ 0 , and let us show that the same result holds.
Revised: )) ≤ 0 , and let us show that the same result holds.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Therefore we have:
Revised: Therefore we have:

Original: [Equation]
Revised: [Equation]

Original: In conclusion
Revised: In conclusion

Original: [Equation]
Revised: [Equation]

Original: regardless of the sign
Revised: regardless of the sign

Original: [Equation]
Revised: [Equation]

Original: it always holds true that:
Revised: it always holds true that:

Original: [Equation]
Revised: [Equation]

Original: We now upper
Revised: We now upper

Original: [Equation]
Revised: [Equation]

Original: Putting inequalities (74), (78) and (79) together
Revised: Putting inequalities (74), (78) and (79) together

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore we have:
Revised: Therefore we have:

Original: [Equation]
Revised: [Equation]

Original: By summing over t and taking the expectation over the z t
Revised: By summing over t and taking the expectation over the z t

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: By assumption, we have that δ − 2 βε > 0 .
Revised: By assumption, we have that δ − 2 βε > 0 .

Original: Dividing by T + 1 and using the convexity of f , we ﬁnally obtain:
Revised: Dividing by T + 1 and using the convexity of f , we ﬁnally obtain:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 6.
Revised: Theorem 6.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≥ 12 β .
Revised: Further assume that η ≥ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: By using the fact that γ t ≤
Revised: By using the fact that γ t ≤

Original: [Equation]
Revised: [Equation]

Original: rather than γ
Revised: rather than γ

Original: [Equation]
Revised: [Equation]

Original: (cid:96) zt (
Revised: (cid:96) zt (

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 + δ , we can see that the inequality (74) is still valid:
Revised: (cid:107) 2 + δ , we can see that the inequality (74) is still valid:

Original: [Equation]
Revised: [Equation]

Original: As previously, we lower bound γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) and upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) individually.
Revised: As previously, we lower bound γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) and upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) individually.

Original: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) .
Revised: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) .

Original: We remark that either γ t =
Revised: We remark that either γ t =

Original: [Equation]
Revised: [Equation]

Original: + δ or γ t = η .
Revised: + δ or γ t = η .

Original: Suppose γ t = (cid:96) zt ( wt
Revised: Suppose γ t = (cid:96) [Equation] ( w

Original: [Equation]
Revised: [Equation]

Original: − B (cid:107)∇ (cid:96) zt ( w t ) (cid:107) 

Revised: − B (cid:107)∇ (cid:96) zt ( w t ) (cid:107)

Original: +
Revised: +

Original: δ
Revised: δ

Original: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .
Revised: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .

Original: Then we are in the same condition as in
Revised: Then we are in the same condition as in

Original: Theorem 5, and thus the inequality (78) holds true:
Revised: Theorem 5, and thus the inequality (78) holds true:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now suppose γ t = η
Revised: Now suppose γ t = η

Original: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .
Revised: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: because η
Revised: because η

Original: [Equation]
Revised: [Equation]

Original: Now suppose (cid:96)
Revised: Now suppose (cid:96)

Original: [Equation]
Revised: [Equation]

Original: By using
Revised: By using

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: δ , we can see that the inequality (76) is still valid, which gives:
Revised: δ , we can see that the inequality (76) is still valid, which gives:

Original: [Equation]
Revised: [Equation]

Original: We now use (cid:96) z t ( w
Revised: We now use (cid:96) z t ( w

Original: [Equation]
Revised: [Equation]

Original: − (cid:96)
Revised: − (cid:96)

Original: [Equation]
Revised: [Equation]

Original: 0 to write:
Revised: 0 to write:

Original: [Equation]
Revised: [Equation]

Original: In conclusion
Revised: In conclusion

Original: [Equation]
Revised: [Equation]

Original: in all cases
Revised: in all cases

Original: [Equation]
Revised: [Equation]

Original: it holds true that:
Revised: it holds true that:

Original: [Equation]
Revised: [Equation]

Original: By using γ
Revised: By using γ

Original: [Equation]
Revised: [Equation]

Original: , we can remark that the inequality (79) holds true and gives:
Revised: , we can remark that the inequality (79) holds true and gives:

Original: [Equation]
Revised: [Equation]

Original: We now put together inequalities (84), (89) and (90):
Revised: We now put together inequalities (84), (89) and (90):

Original: [Equation]
Revised: [Equation]

Original: This is exactly the same result as in the inequality (81) from the proof of Theorem 5.
Revised: This is exactly the same result as in the inequality (81) from the proof of Theorem 5.

Original: Therefore the rest of the proof of Theorem 5 follows and we obtain the desired result.
Revised: Therefore the rest of the proof of Theorem 5 follows and we obtain the desired result.

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 7.
Revised: Theorem 7.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≤ 12 β .
Revised: Further assume that η ≤ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: By using the fact that γ t ≤
Revised: By using the fact that γ t ≤

Original: [Equation]
Revised: [Equation]

Original:  rather than
Revised: δ rather than

Original: [Equation]
Revised: [Equation]

Original: 2 + δ , we can see that the inequality (74) is still valid:
Revised: 2 + δ , we can see that the inequality (74) is still valid:

Original: [Equation]
Revised: [Equation]

Original: As previously, we lower bound γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) and upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) individually.
Revised: As previously, we lower bound γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) and upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) individually.

Original: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) .
Revised: We begin with γ t ( (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) )) .

Original: We remark that either γ t =
Revised: We remark that either γ t =

Original: [Equation]
Revised: [Equation]

Original:  B
Revised: − B

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2 + δ or γ t = η
Revised: 2 + δ or γ t = η

Original: [Equation]
Revised: [Equation]

Original: Suppose γ t =
Revised: Suppose γ t =

Original: [Equation]
Revised: [Equation]

Original: zt

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 2 + δ
Revised: 2 + δ

Original: and (cid:96) z t
Revised: and (cid:96) z t

Original: [Equation]
Revised: [Equation]

Original: ) −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: First we write:
Revised: First we write:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:96) z t
Revised: Since (cid:96) [Equation] t

Original: [Equation]
Revised: [Equation]

Original: t ) −
Revised: t ) −

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: , this yields:
Revised: , this yields:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We now notice that since γ t =
Revised: We now notice that since γ t =

Original: [Equation]
Revised: [Equation]

Original: and γ
Revised: and γ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: , then necessarily
Revised: , then necessarily

Original: [Equation]
Revised: [Equation]

Original: This gives:
Revised: This gives:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now suppose γ t = η
Revised: Now suppose γ t = η

Original: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .
Revised: and (cid:96) z t ( w t ) − (cid:96) z t ( w (cid:63) ) ≥ 0 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: ) −
Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: η
Revised: η

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now suppose (cid:96) z t ( w t
Revised: Now suppose (cid:96) z t ( w t

Original: [Equation]
Revised: [Equation]

Original: − (cid:96) z t ( w (cid:63) ) ≤ 0 .
Revised: − (cid:96) z t ( w (cid:63) ) ≤ 0 .

Original: Since γ t ≤ η by deﬁnition, we have that:
Revised: Since γ t ≤ η by deﬁnition, we have that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In conclusion, in all cases, it holds true that:
Revised: In conclusion, in all cases, it holds true that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) as follows:
Revised: We upper bound γ t ( (cid:96) z t ( w (cid:63) ) − B ) as follows:

Original: [Equation]
Revised: [Equation]

Original: We combine inequalities (92), (98) and (99) and obtain:
Revised: We combine inequalities (92), (98) and (99) and obtain:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By taking the expectation and using a telescopic sum, we obtain
Revised: By taking the expectation and using a telescopic sum, we obtain

Original: [Equation]
Revised: [Equation]

Original: Re-arranging and using the convexity of f , we ﬁnally obtain:
Revised: Re-arranging and using the convexity of f , we ﬁnally obtain:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Lemma 3.
Revised: Lemma 3.

Original: For any a, b ∈ R p , we have that:
Revised: For any a, b ∈ R p , we have that:

Original: [Equation]
Revised: [Equation]

Original: Proof: This is a simple application of the parallelogram law, but we give the proof here for completeness.
Revised: Proof: This is a simple application of the parallelogram law, but we give the proof here for completeness.

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Let z ∈ Z .
Revised: Let z ∈ Z .

Original: Assume that (cid:96) z is α -strongly convex and is lower-bounded on R p by B ∈ R such that inf (cid:96) z − B ≤ ε .
Revised: Assume that (cid:96) z is α -strongly convex and is lower-bounded on R p by B ∈ R such that inf (cid:96) z − B ≤ ε .

Original: In addition, suppose that δ ≥ 2 αε .
Revised: In addition, suppose that δ ≥ 2 αε .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: Let w ∈ R p and suppose that (cid:96) z reaches its minimum at w ∈ R p (this minimum exists because of strong convexity).
Revised: Let w ∈ R p and suppose that (cid:96) z reaches its minimum at w ∈ R p (this minimum exists because of strong convexity).

Original: By deﬁnition of strong convexity, we have that:
Revised: By deﬁnition of strong convexity, we have that:

Original: [Equation]
Revised: [Equation]

Original: We minimize the right hand-side over ˆ w , which gives:
Revised: We minimize the right hand-side over ˆ w , which gives:

Original: [Equation]
Revised: [Equation]

Original: Thus by choosing ˆ w = w and re-ordering
Revised: Thus by choosing ˆ w = w and re-ordering

Original: [Equation]
Revised: [Equation]

Original: we obtain the following result (a
Revised: we obtain the following result (a

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: a. the Polyak-Lojasiewicz inequality):
Revised: a. the Polyak-Lojasiewicz inequality):

Original: [Equation]
Revised: [Equation]

Original: Therefore we can write:
Revised: Therefore we can write:

Original: [Equation]
Revised: [Equation]

Original: We introduce the function ψ : x ∈ R
Revised: We introduce the function ψ : x ∈ R

Original: + (cid:55)→ 12 α x + ε x + δ , and we compute its derivative:
Revised: + (cid:55)→ 12 α x + ε x + δ , and we compute its derivative:

Original: [Equation]
Revised: [Equation]

Original: Therefore ψ is monotonically increasing.
Revised: Therefore ψ is monotonically increasing.

Original: As a result, we have:
Revised: As a result, we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore we have that:
Revised: Therefore we have that:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  which concludes the proof.
Revised: (111) which concludes the proof.

Original: Theorem 8.
Revised: Theorem 8.

Original: [Strongly Convex and Smooth] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: [Strongly Convex and Smooth] We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).
Revised: Let B be a uniform lower bound on ( P ) and w (cid:63) be a solution of ( P ).

Original: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .
Revised: Further suppose that w (cid:63) is an ε -interpolation for (( P ), B ), and that δ > 2 βε .

Original: Then
Revised: Then

Original: ALI-G ∞ applied to f satisﬁes:
Revised: ALI-G ∞ applied to f satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: In other words, f approximately converges to f (cid:63) at a rate of O (exp( − αT/ 8 β )) .
Revised: In other words, f approximately converges to f (cid:63) at a rate of O (exp( − αT/ 8 β )) .

Original: Proof:
Revised: Proof:

Original: We condition the update on z t drawn at random.
Revised: We condition the update on z t drawn at random.

Original: The beginning of the proof is identical to that of Theorem 5 (and in particular requires δ > 2 βε ).
Revised: The beginning of the proof is identical to that of Theorem 5 (and in particular requires δ > 2 βε ).

Original: In addition, we remark that δ > 2 βε ≥ 2 αε , because it always holds true that β ≥ α .
Revised: In addition, we remark that δ > 2 βε ≥ 2 αε , because it always holds true that β ≥ α .

Original: Combining inequalities (74) and (78), we obtain:
Revised: Combining inequalities (74) and (78), we obtain:

Original: [Equation]
Revised: [Equation]

Original: (deﬁnition of
Revised: (deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: Let w ( z t ) be the minimizer of (cid:96) z t on its unconstrained domain R p (its existence is guaranteed by the strong convexity property).
Revised: Let w ( z t ) be the minimizer of (cid:96) z t on its unconstrained domain R p (its existence is guaranteed by the strong convexity property).

Original: Then we exploit strong convexity to lower bound the progress made:
Revised: Then we exploit strong convexity to lower bound the progress made:

Original: [Equation]
Revised: [Equation]

Original: We now combine inequalities (112) and (113):
Revised: We now combine inequalities (112) and (113):

Original: [Equation]
Revised: [Equation]

Original: We use a trivial induction over t and write:
Revised: We use a trivial induction over t and write:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In particular, we remark that the right hand-side of the equation is independent on z t .
Revised: In particular, we remark that the right hand-side of the equation is independent on z t .

Original: Given an arbitrary w ∈ R p , we now wish to relate the distance (cid:107) w − w (cid:63) (cid:107) 2 to the function values f ( w ) − f ( w (cid:63) ) .
Revised: Given an arbitrary w ∈ R p , we now wish to relate the distance (cid:107) w − w (cid:63) (cid:107) 2 to the function values f ( w ) − f ( w (cid:63) ) .

Original: Since each (cid:96) z is α -strongly convex and β -smooth, so is f =
Revised: Since each (cid:96) z is α -strongly convex and β -smooth, so is f =

Original: E z
Revised: E z

Original: [ (cid:96) z ] .
Revised: [ (cid:96) z ] .

Original: We introduce w the minimizer of f on its unconstrained domain R p .
Revised: We introduce w the minimizer of f on its unconstrained domain R p .

Original: Then we can write that for any w ∈ R p :
Revised: Then we can write that for any w ∈ R p :

Original: [Equation]
Revised: [Equation]

Original: We combine the results to obtain the ﬁnal result:
Revised: We combine the results to obtain the ﬁnal result:

Original: [Equation]
Revised: [Equation]

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 9.
Revised: Theorem 9.

Original: We assume that X is a convex set
Revised: We assume that X is a convex set

Original: [Equation]
Revised: [Equation]

Original: and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B )
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B )

Original: [Equation]
Revised: [Equation]

Original: and suppose that δ > 2 βε .
Revised: and suppose that δ > 2 βε .

Original: Further assume that η ≥ 12 β .
Revised: Further assume that η ≥ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f , we have:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof: Re-using inequalities (84) and (89) from the proof of Theorem 6, we obtain:
Revised: Proof: Re-using inequalities (84) and (89) from the proof of Theorem 6, we obtain:

Original: [Equation]
Revised: [Equation]

Original: This is exactly the same result as the ﬁrst line of the inequality (112) in the proof of Theorem 8.
Revised: This is exactly the same result as the ﬁrst line of the inequality (112) in the proof of Theorem 8.

Original: Then the rest of the proof is identical to the one of Theorem 8.
Revised: Then the rest of the proof is identical to the one of Theorem 8.

Original: D. T HEOREM
Revised: D. T HEOREM

Original: Theorem 10.
Revised: Theorem 10.

Original: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.
Revised: We assume that X is a convex set, and that for every z ∈ Z , (cid:96) z is α -strongly convex and β -smooth.

Original: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .
Revised: Let w (cid:63) be an ε -interpolation for (( P ), B ), and suppose that δ > 2 βε .

Original: Further assume that η ≤ 12 β .
Revised: Further assume that η ≤ 12 β .

Original: Then if we apply ALI-G with a maximal learning-rate of η to f
Revised: Then if we apply ALI-G with a maximal learning-rate of η to f

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Proof: Re-using inequalities (92) and (98) from the proof of Theorem 7, we can write:
Revised: Proof: Re-using inequalities (92) and (98) from the proof of Theorem 7, we can write:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the inequality (113) gives:
Revised: Furthermore, the inequality (113) gives:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can write:
Revised: we can write:

Original: [Equation]
Revised: [Equation]

Original: Then a trivial induction gives that:
Revised: Then a trivial induction gives that:

Original: [Equation]
Revised: [Equation]

Original: We now re-use the inequality (116) to write:
Revised: We now re-use the inequality (116) to write:

Original: [Equation]
Revised: [Equation]

Revised: E A DDITIONAL E XPERIMENTAL D ETAILS E. S TANDARD D EVIATION OF CIFAR R ESULTS

Revised: [Table]