Original: Coordinate Linear Variance Reduction for Generalized Linear Programming
Revised: Coordinate Linear Variance Reduction for Generalized Linear Programming

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We study a class of generalized linear programs (GLP) in a large-scale setting,which includes a simple, possibly nonsmooth convex regularizer and simple convexset constraints.
Revised: We study a class of generalized linear programs (GLP) in a large-scale setting,which includes a simple, possibly nonsmooth convex regularizer and simple convexset constraints.

Original: By reformulating (GLP) as an equivalent convex-concave min-maxproblem, we show that the linear structure in the problem can be used to designan efﬁcient, scalable ﬁrst-order algorithm, to which we give the name Coordinate
Revised: By reformulating (GLP) as an equivalent convex-concave min-maxproblem, we show that the linear structure in the problem can be used to designan efﬁcient, scalable ﬁrst-order algorithm, to which we give the name Coordinate

Original: Linear Variance Reduction ( CLVR ; pronounced “clever”).
Revised: Linear Variance Reduction ( CLVR ; pronounced “clever”).

Original: CLVR yields improvedcomplexity results for (GLP) that depend on the max row norm of the linearconstraint matrix in (GLP) rather than the spectral norm.
Revised: CLVR yields improvedcomplexity results for (GLP) that depend on the max row norm of the linearconstraint matrix in (GLP) rather than the spectral norm.

Original: When the regularizationterms and constraints are separable, CLVR admits an efﬁcient lazy update strategythat makes its complexity bounds scale with the number of nonzero elements ofthe linear constraint matrix in (GLP) rather than the matrix dimensions.
Revised: When the regularizationterms and constraints are separable, CLVR admits an efﬁcient lazy update strategythat makes its complexity bounds scale with the number of nonzero elements ofthe linear constraint matrix in (GLP) rather than the matrix dimensions.

Original: Further,for the special case of linear programs and by exploiting sharpness, we proposea restart scheme for CLVR to obtain empirical linear convergence.
Revised: Further,for the special case of linear programs and by exploiting sharpness, we proposea restart scheme for CLVR to obtain empirical linear convergence.

Original: Finally, weshow that Distributionally Robust Optimization (DRO) problems with ambiguitysets based on both f -divergence and Wasserstein metrics can be reformulated as(GLPs) by introducing sparsely connected auxiliary variables.
Revised: Finally, weshow that Distributionally Robust Optimization (DRO) problems with ambiguitysets based on both f -divergence and Wasserstein metrics can be reformulated as(GLPs) by introducing sparsely connected auxiliary variables.

Original: We complementour theoretical guarantees with numerical experiments that verify our algorithm’spractical effectiveness in terms of wall-clock time and number of data passes.
Revised: We complementour theoretical guarantees with numerical experiments that verify our algorithm’spractical effectiveness in terms of wall-clock time and number of data passes.

Original:  Introduction
Revised:  Introduction

Original: We study the following generalized linear program (GLP):
Revised: We study the following generalized linear program (GLP):

Original: [Equation]
Revised: [Equation]

Original: where x , c ∈ R d , A ∈ R n × d , b ∈ R n , r : R d → R is a convex regularizer and X ⊆
Revised: where x , c ∈ R d , A ∈ R n × d , b ∈ R n , r : R d → R is a convex regularizer and X ⊆

Original: R
Revised: R

Original: d is a closedconvex set, such that a proximal / projection operator involving r and X can be computed efﬁciently.
Revised: d is a closedconvex set, such that a proximal / projection operator involving r and X can be computed efﬁciently.

Original: When X is the nonnegative orthant { x : x i ≥ 0 , i ∈ [ d ] } and r ≡ 0 , (GLP) reduces to the standardform of a linear program (LP).
Revised: When X is the nonnegative orthant { x : x i ≥ 0 , i ∈ [ d ] } and r ≡ 0 , (GLP) reduces to the standardform of a linear program (LP).

Original: When X is a convex cone and r ≡ 0 , (GLP) reduces to a coniclinear program.
Revised: When X is a convex cone and r ≡ 0 , (GLP) reduces to a coniclinear program.

Original: (GLP) has had a signiﬁcant impact in traditional engineering disciplines such astransportation, energy, telecommunications, and manufacturing.
Revised: (GLP) has had a signiﬁcant impact in traditional engineering disciplines such astransportation, energy, telecommunications, and manufacturing.

Original: In modern data science, we observethe renaissance of (GLP) due to its modeling power in such areas as reinforcement learning [18],optimal transport [57], and neural network veriﬁcation [39].
Revised: In modern data science, we observethe renaissance of (GLP) due to its modeling power in such areas as reinforcement learning [18],optimal transport [57], and neural network veriﬁcation [39].

Original: For traditional engineering disciplineswith moderate scale or exploitable sparsity, off-the-shelf interior point methods that form and factorizematrices in each iteration are often good choices as practical solvers [25].
Revised: For traditional engineering disciplineswith moderate scale or exploitable sparsity, off-the-shelf interior point methods that form and factorizematrices in each iteration are often good choices as practical solvers [25].

Original: In data science applications,however, where the data are often dense or of extreme scale, the amount of computation and/ormemory required by matrix factorization is prohibitive.
Revised: In data science applications,however, where the data are often dense or of extreme scale, the amount of computation and/ormemory required by matrix factorization is prohibitive.

Original: Thus, ﬁrst-order methods that avoid matrixfactorizations are potentially appealing options.
Revised: Thus, ﬁrst-order methods that avoid matrixfactorizations are potentially appealing options.

Original: In this context, because the presence of the linearequality constraint in (GLP) may complicate projection operations onto the feasible set, we consider an equivalent reformulation of (GLP) as a min-max problem involving the Lagrangian:
Revised: In this context, because the presence of the linearequality constraint in (GLP) may complicate projection operations onto the feasible set, we consider an equivalent reformulation of (GLP) as a min-max problem involving the Lagrangian:

Original: [Equation]
Revised: [Equation]

Original: In data science applications, both n and d can be very large.
Revised: In data science applications, both n and d can be very large.

Original: (PD-GLP) can be viewed as a structuredbilinearly coupled min-max problem, where the linearity of L ( x , y ) in the dual variable vector y isvital to our algorithmic development.
Revised: (PD-GLP) can be viewed as a structuredbilinearly coupled min-max problem, where the linearity of L ( x , y ) in the dual variable vector y isvital to our algorithmic development.

Original: 1.1 Background
Revised: 1.1 Background

Original: While the literature addressing (PD-GLP) is sparse — some special cases have been consideredin [13, 24, 41–43, 60, 62, 63] — there has been signiﬁcant recent work on ﬁrst-order methods forgeneral bilinearly coupled convex-concave min-max problems.
Revised: While the literature addressing (PD-GLP) is sparse — some special cases have been consideredin [13, 24, 41–43, 60, 62, 63] — there has been signiﬁcant recent work on ﬁrst-order methods forgeneral bilinearly coupled convex-concave min-max problems.

Original: Deterministic ﬁrst-order methodsinclude the proximal point method (PPM) [51], the extragradient/mirror-prox method (EGM) [34, 45],the primal-dual hybrid gradient (PDHG) method [14], and the alternating direction method ofmultipliers (ADMM) [19].
Revised: Deterministic ﬁrst-order methodsinclude the proximal point method (PPM) [51], the extragradient/mirror-prox method (EGM) [34, 45],the primal-dual hybrid gradient (PDHG) method [14], and the alternating direction method ofmultipliers (ADMM) [19].

Original: All these methods have per-iteration cost Θ(nnz( A )) and convergencerate 1 /k , where nnz( A ) denotes the number of nonzero elements of A and k is the number ofiterations.
Revised: All these methods have per-iteration cost Θ(nnz( A )) and convergencerate 1 /k , where nnz( A ) denotes the number of nonzero elements of A and k is the number ofiterations.

Original: For better scalability, stochastic counterparts of these methods have been proposed.
Revised: For better scalability, stochastic counterparts of these methods have been proposed.

Original: [10, 33, 47, 49]have used “vanilla” stochastic gradients to replace the full gradients of their deterministic counterparts.[2, 12, 27] have exploited the ﬁnite-sum structure of the interaction term (cid:104)
Revised: [10, 33, 47, 49]have used “vanilla” stochastic gradients to replace the full gradients of their deterministic counterparts.[2, 12, 27] have exploited the ﬁnite-sum structure of the interaction term (cid:104)

Original: y , Ax (cid:105) involving bothprimal and dual variables to perform variance reduction.
Revised: y , Ax (cid:105) involving bothprimal and dual variables to perform variance reduction.

Original: With a separability assumption for thedual variables, [3] and [15] have combined incremental coordinate approaches on the dual variableswith an implicit variance reduction strategy on the primal variables.
Revised: With a separability assumption for thedual variables, [3] and [15] have combined incremental coordinate approaches on the dual variableswith an implicit variance reduction strategy on the primal variables.

Original: Recently, under a separabilityassumption for dual variables, [55] have proposed a new incremental coordinate method with aninitialization step that requires a single access to the full data.
Revised: Recently, under a separabilityassumption for dual variables, [55] have proposed a new incremental coordinate method with aninitialization step that requires a single access to the full data.

Original: This approach, known as variancereduction via primal-dual accelerated dual averaging ( VRPDA 2 ), obtains the ﬁrst theoretical boundsthat are better than their deterministic counterparts in the class of incremental coordinate approaches.
Revised: This approach, known as variancereduction via primal-dual accelerated dual averaging ( VRPDA 2 ), obtains the ﬁrst theoretical boundsthat are better than their deterministic counterparts in the class of incremental coordinate approaches.

Original: The VRPDA 2 algorithm serves as the main motivation for our approach.
Revised: The VRPDA 2 algorithm serves as the main motivation for our approach.

Original: It is of particular interest to design algorithms that scale with the number of nonzero elements in
Revised: It is of particular interest to design algorithms that scale with the number of nonzero elements in

Original: A for at least two reasons: (i) the data matrix can be sparse; and (ii) when we consider simpliﬁedreformulations of certain complicated models, we often need to introduce sparsely connected auxiliaryvariables.
Revised: A for at least two reasons: (i) the data matrix can be sparse; and (ii) when we consider simpliﬁedreformulations of certain complicated models, we often need to introduce sparsely connected auxiliaryvariables.

Original: Nevertheless, the randomized coordinate algorithms of [3, 15, 55] have O ( d ) per-iterationcost regardless of the sparsity of A .
Revised: Nevertheless, the randomized coordinate algorithms of [3, 15, 55] have O ( d ) per-iterationcost regardless of the sparsity of A .

Original: To address this issue, [23, 35] have proposed incremental primaldual coordinate methods with per-iteration cost that scales with the number of nonzero elements inthe row from A that is used in each iteration, at the price of smaller step size for dense A .
Revised: To address this issue, [23, 35] have proposed incremental primaldual coordinate methods with per-iteration cost that scales with the number of nonzero elements inthe row from A that is used in each iteration, at the price of smaller step size for dense A .

Original: Moreover,[5] has proposed a random extrapolation approach that admits both low per-iteration cost and largestep size.
Revised: Moreover,[5] has proposed a random extrapolation approach that admits both low per-iteration cost and largestep size.

Original: Despite these developments, all these algorithms produce less accurate iterates than themethods with O ( d ) per-iteration cost, thus degrading their worst-case complexity.
Revised: Despite these developments, all these algorithms produce less accurate iterates than themethods with O ( d ) per-iteration cost, thus degrading their worst-case complexity.

Original: Finally, for the special case of LP, based on the positive Hoffman constant [30], [9] proved that theprimal-dual formulation of LP exhibits a sharpness property, which bounds below the growth of anormalized primal-dual gap from the same work.
Revised: Finally, for the special case of LP, based on the positive Hoffman constant [30], [9] proved that theprimal-dual formulation of LP exhibits a sharpness property, which bounds below the growth of anormalized primal-dual gap from the same work.

Original: Based on this sharpness property, [9] proposed arestart scheme for the deterministic ﬁrst-order methods discussed above to obtain linear convergence.[8] further extended this restart strategy using various heuristics to improve practical performance.
Revised: Based on this sharpness property, [9] proposed arestart scheme for the deterministic ﬁrst-order methods discussed above to obtain linear convergence.[8] further extended this restart strategy using various heuristics to improve practical performance.

Original: 1.2 Motivation
Revised: 1.2 Motivation

Original: We sharpen the focus from general bilinearly coupled convex-concave min-max problems to (GLP)and its primal-dual formulation (PD-GLP), because many complicated models can be reformulatedas (GLP) and (GLP) possesses additional structure that can be exploited in algorithm design.
Revised: We sharpen the focus from general bilinearly coupled convex-concave min-max problems to (GLP)and its primal-dual formulation (PD-GLP), because many complicated models can be reformulatedas (GLP) and (GLP) possesses additional structure that can be exploited in algorithm design.

Original: Ourmotivation for focusing on (GLP) is to bridge the large gap between the well-studied stochasticvariance reduced ﬁrst-order methods [6, 32, 54, 55] and the increasingly popular and complicated,yet highly structured large-scale problems arising in distributionally robust optimization (DRO) [20–22, 31, 38, 44, 52, 58, 61]; see also a recent survey by [50] and references therein.
Revised: Ourmotivation for focusing on (GLP) is to bridge the large gap between the well-studied stochasticvariance reduced ﬁrst-order methods [6, 32, 54, 55] and the increasingly popular and complicated,yet highly structured large-scale problems arising in distributionally robust optimization (DRO) [20–22, 31, 38, 44, 52, 58, 61]; see also a recent survey by [50] and references therein.

Original: For DRO problems with ambiguity sets deﬁned by f -divergence [31, 37, 44], the original formulationis a nonbilinearly coupled convex-concave min-max problem.
Revised: For DRO problems with ambiguity sets deﬁned by f -divergence [31, 37, 44], the original formulationis a nonbilinearly coupled convex-concave min-max problem.

Original: Even the well constructed reformulationin [37] does not admit unbiased stochastic gradients, leading to complicated algorithms and analysis.
Revised: Even the well constructed reformulationin [37] does not admit unbiased stochastic gradients, leading to complicated algorithms and analysis.

Original: For DRO problems with ambiguity sets deﬁned by Wasserstein metric [22, 29, 38, 52, 61], the originalformulation is in general inﬁnite-dimensional.
Revised: For DRO problems with ambiguity sets deﬁned by Wasserstein metric [22, 29, 38, 52, 61], the originalformulation is in general inﬁnite-dimensional.

Original: (Finite-dimensional reformulations [22, 52] exist forspecial cases of logistic regression and smooth convex losses.)
Revised: (Finite-dimensional reformulations [22, 52] exist forspecial cases of logistic regression and smooth convex losses.)

Original: Solvers that have been proposed for
Revised: Solvers that have been proposed for

Original: DRO with Wasserstein metric are either multiple-loop deterministic ADMM [38] or are designed forgeneral convex-concave problems [61].
Revised: DRO with Wasserstein metric are either multiple-loop deterministic ADMM [38] or are designed forgeneral convex-concave problems [61].

Original: By introducing auxiliary variables with sparse connections, 1 we show that DRO with ambiguitysets based on both f -divergence and the Wasserstein metric can be reformulated as (GLP).
Revised: By introducing auxiliary variables with sparse connections, 1 we show that DRO with ambiguitysets based on both f -divergence and the Wasserstein metric can be reformulated as (GLP).

Original: Thus,complicated DRO problems can be addressed by a simple, efﬁcient, and scalable algorithm for (GLP).
Revised: Thus,complicated DRO problems can be addressed by a simple, efﬁcient, and scalable algorithm for (GLP).

Original: Our algorithm for solving (GLP) and the proposed reformulations of DRO are our main contributions.
Revised: Our algorithm for solving (GLP) and the proposed reformulations of DRO are our main contributions.

Original: 1.3 Contributions
Revised: 1.3 Contributions

Original: Algorithm.
Revised: Algorithm.

Original: Motivated by VRPDA 2 [55], we propose a simple, efﬁcient, and scalable algorithmfor (PD-GLP).
Revised: Motivated by VRPDA 2 [55], we propose a simple, efﬁcient, and scalable algorithmfor (PD-GLP).

Original: Our algorithm combines an incremental coordinate method with exploitation of thelinear structure for the dual variables in (PD-GLP) and the implicit variance reduction effect inthe algorithm, so we name it coordinate linear variance reduction ( CLVR , pronounced “clever”).
Revised: Our algorithm combines an incremental coordinate method with exploitation of thelinear structure for the dual variables in (PD-GLP) and the implicit variance reduction effect inthe algorithm, so we name it coordinate linear variance reduction ( CLVR , pronounced “clever”).

Original: CLVR is inspired by VRPDA 2 but customized to the particular structure of (PD-GLP).
Revised: CLVR is inspired by VRPDA 2 but customized to the particular structure of (PD-GLP).

Original: In particular,by exploiting the fact that the max problem is linear and unconstrained in the dual variable vectory ∈ R n , we ﬁnd that the expensive initialization step used in VRPDA 2 is not needed and we cantake simpler and larger steps.
Revised: In particular,by exploiting the fact that the max problem is linear and unconstrained in the dual variable vectory ∈ R n , we ﬁnd that the expensive initialization step used in VRPDA 2 is not needed and we cantake simpler and larger steps.

Original: Further, in the structured case in which A is sparse and the convexconstraint set X and the regularizer r ( x ) are fully separable 2 , we show that the dual averaging updatein CLVR enables us to design an efﬁcient lazy update strategy for which the per-iteration cost of
Revised: Further, in the structured case in which A is sparse and the convexconstraint set X and the regularizer r ( x ) are fully separable 2 , we show that the dual averaging updatein CLVR enables us to design an efﬁcient lazy update strategy for which the per-iteration cost of

Original: CLVR scales with the number of nonzero elements of the selected row from A in each iteration, whichis potentially much lower than the order- d cost in VRPDA 2 .
Revised: CLVR scales with the number of nonzero elements of the selected row from A in each iteration, whichis potentially much lower than the order- d cost in VRPDA 2 .

Original: Finally, CLVR uses extrapolation ondual variables rather than on primal variables considered in VRPDA 2 , which signiﬁcantly reducesimplementation complexity of our lazy update strategy for structured variants of (PD-GLP).
Revised: Finally, CLVR uses extrapolation ondual variables rather than on primal variables considered in VRPDA 2 , which signiﬁcantly reducesimplementation complexity of our lazy update strategy for structured variants of (PD-GLP).

Original: Onthe technical side, although both CLVR and VRPDA 2 are randomized algorithms that bound theprimal-dual gap in expectation, the guarantee provided by CLVR is stronger as it allows bounding theexpectation of the supremum gap as opposed to the supremum of expected gap in VRPDA 2 .
Revised: Onthe technical side, although both CLVR and VRPDA 2 are randomized algorithms that bound theprimal-dual gap in expectation, the guarantee provided by CLVR is stronger as it allows bounding theexpectation of the supremum gap as opposed to the supremum of expected gap in VRPDA 2 .

Original: To state our complexity results, we make the following scaling assumption.
Revised: To state our complexity results, we make the following scaling assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: L := (cid:107)
Revised: L := (cid:107)

Original: A (cid:107) and each row of A in (GLP)
Revised: A (cid:107) and each row of A in (GLP)

Original: is normalized with Euclidean norm R .
Revised: is normalized with Euclidean norm R .

Original: As a comment, it is known that preprocessing in modern LP solvers [25] often ensures normalizedrows/columns for the data matrix.
Revised: As a comment, it is known that preprocessing in modern LP solvers [25] often ensures normalizedrows/columns for the data matrix.

Original: Observe that R ≤ L ≤ √ nR. The upper bound is achieved whenall elements of A have identical value.
Revised: Observe that R ≤ L ≤ √ nR. The upper bound is achieved whenall elements of A have identical value.

Original: Although this is an extreme case, there exist ill-conditionedpractical datasets where we can expect signiﬁcant performance gains if the complexity can bereduced from O ( L ) to O ( R ) .
Revised: Although this is an extreme case, there exist ill-conditionedpractical datasets where we can expect signiﬁcant performance gains if the complexity can bereduced from O ( L ) to O ( R ) .

Original: In Table 1, we give the overall complexity bounds (total number ofarithmetic operations) and the per-iteration cost of a representative set of existing algorithms and
Revised: In Table 1, we give the overall complexity bounds (total number ofarithmetic operations) and the per-iteration cost of a representative set of existing algorithms and

Original: our
Revised: our

Original: CLVR algorithm for solving structured (PD-GLP), i.e., X = X 1 × · · · × X d with X i ∈ R ( i ∈
Revised: CLVR algorithm for solving structured (PD-GLP), i.e., X = X 1 × · · · × X d with X i ∈ R ( i ∈

Original: [ d ])and r ( x ) := (cid:80) di =1 r ( x i ) .
Revised: [ d ])and r ( x ) := (cid:80) di =1 r ( x i ) .

Original: We further assume that for the stochastic algorithms [2, 15, 55] and our
Revised: We further assume that for the stochastic algorithms [2, 15, 55] and our

Original: CLVR algorithm, we draw one row of A per iteration uniformly at random to make the complexityresults comparable.
Revised: CLVR algorithm, we draw one row of A per iteration uniformly at random to make the complexityresults comparable.

Original: The general convex setting corresponds to r ( x ) being general convex ( σ = 0 ),while the strongly convex setting corresponds to r ( x ) being σ -strongly convex ( σ > 0 ).
Revised: The general convex setting corresponds to r ( x ) being general convex ( σ = 0 ),while the strongly convex setting corresponds to r ( x ) being σ -strongly convex ( σ > 0 ).

Original: As shown in Table 1, all the algorithms have the optimal dependence on (cid:15) [48], while the dependenceon the ambient dimensions n, d , the number of nonzero elements of A , nnz( A ) , and the constants
Revised: As shown in Table 1, all the algorithms have the optimal dependence on (cid:15) [48], while the dependenceon the ambient dimensions n, d , the number of nonzero elements of A , nnz( A ) , and the constants

Original: L and R are quite different.
Revised: L and R are quite different.

Original: For both the general convex and strongly convex settings and amongcoordinate-type methods, CLVR is the ﬁrst algorithm that reduces the runtime dependence on theinput matrix size from nd to nnz( A ) .
Revised: For both the general convex and strongly convex settings and amongcoordinate-type methods, CLVR is the ﬁrst algorithm that reduces the runtime dependence on theinput matrix size from nd to nnz( A ) .

Original: Moreover, the complexity of CLVR depends on the max rownorm R rather than the spectral norm L , and the per-iteration cost of CLVR only depends on thenonzero elements of the selected row from A in each iteration, which can be far less than d .
Revised: Moreover, the complexity of CLVR depends on the max rownorm R rather than the spectral norm L , and the per-iteration cost of CLVR only depends on thenonzero elements of the selected row from A in each iteration, which can be far less than d .

Original: By exploiting the linear structure again, we provide explicit guarantees for both the objective valueand the constraint satisfaction of (GLP).
Revised: By exploiting the linear structure again, we provide explicit guarantees for both the objective valueand the constraint satisfaction of (GLP).

Original: Further, the analysis of CLVR applies to the more general
Revised: Further, the analysis of CLVR applies to the more general

Original: [Table]
Revised: [Table]

Original: block-coordinate update setting, which is better suited to modern parallel computing platforms.
Revised: block-coordinate update setting, which is better suited to modern parallel computing platforms.

Original: Finally, following the restart strategy based on the normalized duality gap for LP introduced in[9]
Revised: Finally, following the restart strategy based on the normalized duality gap for LP introduced in[9]

Original: [Table]
Revised: [Table]

Original: we propose a more straightforward strategy to restart our CLVR algorithm (as well as otheriterative algorithms for (PD-GLP)):
Revised: we propose a more straightforward strategy to restart our CLVR algorithm (as well as otheriterative algorithms for (PD-GLP)):

Original: Restart the algorithm every time a widely known metric for LPoptimality [7] halves.
Revised: Restart the algorithm every time a widely known metric for LPoptimality [7] halves.

Original: Compared with the normalized duality gap, the LP metric can be computedmore efﬁciently and in a more straightforward fashion.
Revised: Compared with the normalized duality gap, the LP metric can be computedmore efﬁciently and in a more straightforward fashion.

Original: DRO Reformulations.
Revised: DRO Reformulations.

Original: When the loss function is convex, DRO problems with ambiguity sets basedon f -divergence [44] or Wasserstein metric [22] are convex.
Revised: When the loss function is convex, DRO problems with ambiguity sets basedon f -divergence [44] or Wasserstein metric [22] are convex.

Original: However, as both problems either havecomplicated constraints or are inﬁnite-dimensional, vanilla ﬁrst-order methods are inapplicable.
Revised: However, as both problems either havecomplicated constraints or are inﬁnite-dimensional, vanilla ﬁrst-order methods are inapplicable.

Original: For DRO with f -divergence, we show that by using convex conjugates and introducing auxiliaryvariables, the problem can be reformulated as a (GLP).
Revised: For DRO with f -divergence, we show that by using convex conjugates and introducing auxiliaryvariables, the problem can be reformulated as a (GLP).

Original: As a result, we do not have the issue ofbiased stochastic gradients encountered in [37] and CLVR can be applied.
Revised: As a result, we do not have the issue ofbiased stochastic gradients encountered in [37] and CLVR can be applied.

Original: Even though the resultingproblem has larger dimensions, due to the sparseness of the introduced auxiliary variables and thelazy update strategy of CLVR , it can be solved with complexity scaling only with the number ofnonzero elements of the data matrix.
Revised: Even though the resultingproblem has larger dimensions, due to the sparseness of the introduced auxiliary variables and thelazy update strategy of CLVR , it can be solved with complexity scaling only with the number ofnonzero elements of the data matrix.

Original: Due to being cast as a (GLP), the DRO problem can be solvedwith
Revised: Due to being cast as a (GLP), the DRO problem can be solvedwith

Original: O (1 /(cid:15) ) iteration complexity with CLVR , while existing methods such as [37] have O (1 /(cid:15) 2 )iteration complexity, with higher iteration cost because of the batch of samples needed to reduce bias.
Revised: O (1 /(cid:15) ) iteration complexity with CLVR , while existing methods such as [37] have O (1 /(cid:15) 2 )iteration complexity, with higher iteration cost because of the batch of samples needed to reduce bias.

Original: This improvement is enabled in part by considering the primal-dual gap (rather than the primal gapconsidered in [37]) and by allowing the constraints to be approximately satisﬁed (see Corollary 1).
Revised: This improvement is enabled in part by considering the primal-dual gap (rather than the primal gapconsidered in [37]) and by allowing the constraints to be approximately satisﬁed (see Corollary 1).

Original: For DRO with Wasserstein metric, following the reformulation of [52, Theorem 1], we show furtherthat the problem can be cast in the form of (GLP).
Revised: For DRO with Wasserstein metric, following the reformulation of [52, Theorem 1], we show furtherthat the problem can be cast in the form of (GLP).

Original: Compared with the existing reformulations [22, 38,52, 61], our reformulation can handle both smooth and nonsmooth convex loss functions.
Revised: Compared with the existing reformulations [22, 38,52, 61], our reformulation can handle both smooth and nonsmooth convex loss functions.

Original: In fact, ourreformulation can provide a more compact form for nonsmooth piecewise-linear convex loss functions(such as hinge loss).
Revised: In fact, ourreformulation can provide a more compact form for nonsmooth piecewise-linear convex loss functions(such as hinge loss).

Original: Moreover, compared with algorithms customized to this problem [38] andextragradient methods [34, 45, 61] for general convex-concave min-max problems, our CLVR methodattains the best-known iteration complexity and per-iteration cost, as shown in Table 1.
Revised: Moreover, compared with algorithms customized to this problem [38] andextragradient methods [34, 45, 61] for general convex-concave min-max problems, our CLVR methodattains the best-known iteration complexity and per-iteration cost, as shown in Table 1.

Original:  Notation and Preliminaries
Revised:  Notation and Preliminaries

Original: For any positive integer p , we use [ p ] to denote { 1 , 2 , . .
Revised: For any positive integer p , we use [ p ] to denote { 1 , 2 , . .

Original: . , p } .
Revised: . , p } .

Original: We assume that there is a given partitionof the set [ n ] into sets S j , j ∈ [ m ] , where | S j | = n j > 0 and (cid:80) mj
Revised: We assume that there is a given partitionof the set [ n ] into sets S j , j ∈ [ m ] , where | S j | = n j > 0 and (cid:80) mj

Original: =1 n j = n .
Revised: =1 n j = n .

Original: For j ∈ [ m ] , we use A S jto denote the submatrix of A with rows indexed by S j and y S j to denote the subvector of y indexedby S j .
Revised: For j ∈ [ m ] , we use A S jto denote the submatrix of A with rows indexed by S j and y S j to denote the subvector of y indexedby S j .

Original: We use 0 d and 1 d to denote the vectors with all ones and all zeros in d dimensions, respectively.
Revised: We use 0 d and 1 d to denote the vectors with all ones and all zeros in d dimensions, respectively.

Original: Unless otherwise speciﬁed, we use (cid:107)·(cid:107) to denote the Euclidean norm for vectors and the spectral normfor matrices.
Revised: Unless otherwise speciﬁed, we use (cid:107)·(cid:107) to denote the Euclidean norm for vectors and the spectral normfor matrices.

Original: For a given proper convex lower semi-continuous function f : R → R ∪ { + ∞} , wedeﬁne the convex conjugate in the standard way as f ∗ ( y )
Revised: For a given proper convex lower semi-continuous function f : R → R ∪ { + ∞} , wedeﬁne the convex conjugate in the standard way as f ∗ ( y )

Original: =
Revised: =

Original: sup x ∈ R { yx − f ( x ) } (so that f ∗∗ = f ) .
Revised: sup x ∈ R { yx − f ( x ) } (so that f ∗∗ = f ) .

Original: For a vector u , the inequality u ≥ 0 is applied entry-wise.
Revised: For a vector u , the inequality u ≥ 0 is applied entry-wise.

Original: For a convex function r ( x ) , we use r (cid:48)
Revised: For a convex function r ( x ) , we use r (cid:48)

Original: ( x )to denote an element of the subdifferential set ∂r ( x ) .
Revised: ( x )to denote an element of the subdifferential set ∂r ( x ) .

Original: The proximal operator of r ( x ) over X is
Revised: The proximal operator of r ( x ) over X is

Original: [Equation]
Revised: [Equation]

Original: Further, we make the following assumptions, which apply throughout the convergence analysis.
Revised: Further, we make the following assumptions, which apply throughout the convergence analysis.

Original: Assumption 2.
Revised: Assumption 2.

Original: (PD-GLP) attains at least one primal-dual solution ( x ∗ , y ∗ ) .
Revised: (PD-GLP) attains at least one primal-dual solution ( x ∗ , y ∗ ) .

Original: W ∗ denotes the set ofall primal-dual solutions.
Revised: W ∗ denotes the set ofall primal-dual solutions.

Original: Due to the convex-concave property of (PD-GLP), W ∗ is a convex set in X × R n .
Revised: Due to the convex-concave property of (PD-GLP), W ∗ is a convex set in X × R n .

Original: Assumption 3.
Revised: Assumption 3.

Original: ˆ L = max j ∈ [ m ] (cid:107) A S j (cid:107) is given at the input, where (cid:107) A S j (cid:107) =
Revised: ˆ L = max j ∈ [ m ] (cid:107) A S j (cid:107) is given at the input, where (cid:107) A S j (cid:107) =

Original: max (cid:107) x
Revised: max (cid:107) x

Original: (cid:107)≤ 1 (cid:107) A S j x (cid:107) .
Revised: (cid:107)≤ 1 (cid:107) A S j x (cid:107) .

Original: Note that ˆ L can be obtained either via preprocessing of the data or by parameter tuning.
Revised: Note that ˆ L can be obtained either via preprocessing of the data or by parameter tuning.

Original: By combining
Revised: By combining

Original: Assumptions 1 and 3, it follows that R ≤ ˆ L ≤ (cid:112) max j ∈ [ m ] | S j | R.
Revised: Assumptions 1 and 3, it follows that R ≤ ˆ L ≤ (cid:112) max j ∈ [ m ] | S j | R.

Original: Assumption 4.
Revised: Assumption 4.

Original: r ( x ) is σ -strongly convex ( σ ≥ 0) ; that is, for all x 1 and x 2 in X and all r (cid:48) ( x 2 ) ∈∂r ( x 2 ) , we have r ( x 1 ) ≥ r ( x 2 ) + (cid:104) r (cid:48) ( x 2 ) , x 1 − x 2 (cid:105) + σ 2 (cid:107) x 1 − x 2 (cid:107) 2 .
Revised: r ( x ) is σ -strongly convex ( σ ≥ 0) ; that is, for all x 1 and x 2 in X and all r (cid:48) ( x 2 ) ∈∂r ( x 2 ) , we have r ( x 1 ) ≥ r ( x 2 ) + (cid:104) r (cid:48) ( x 2 ) , x 1 − x 2 (cid:105) + σ 2 (cid:107) x 1 − x 2 (cid:107) 2 .

Original: For convex-concave min-max problems, a common metric for measuring solution quality is theprimal-dual gap, which, for a feasible solution ( x , y ) of (PD-GLP), is deﬁned by
Revised: For convex-concave min-max problems, a common metric for measuring solution quality is theprimal-dual gap, which, for a feasible solution ( x , y ) of (PD-GLP), is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: However, as the domain of v is unbounded, the primal-dual gap can be inﬁnite, which makes it apoor metric for measuring the progress of algorithms.
Revised: However, as the domain of v is unbounded, the primal-dual gap can be inﬁnite, which makes it apoor metric for measuring the progress of algorithms.

Original: As a result, for measuring the progress of ouralgorithm, we consider the following restricted primal-dual gap instead:
Revised: As a result, for measuring the progress of ouralgorithm, we consider the following restricted primal-dual gap instead:

Original: [Equation]
Revised: [Equation]

Original: (3) where W ⊂ X × R n is a compact (i.e., closed and bounded) convex set.
Revised: (3) where W ⊂ X × R n is a compact (i.e., closed and bounded) convex set.

Original: Such a restricted version ofprimal-dual gap is standard in the existing literature; see, e.g., [14, 46].
Revised: Such a restricted version ofprimal-dual gap is standard in the existing literature; see, e.g., [14, 46].

Original:  The CLVR Algorithm
Revised:  The CLVR Algorithm

Original: 3.1 Algorithm and Analysis for General Formulation
Revised: 3.1 Algorithm and Analysis for General Formulation

Original: Algorithm 1 speciﬁes CLVR for (PD-GLP) in the general setting.
Revised: Algorithm 1 speciﬁes CLVR for (PD-GLP) in the general setting.

Original: The algorithm alternates the fullupdate for x k in Step 4 ( O ( d ) cost) with an incremental block coordinate update for y k in Steps 5 and6 (with O ( | S j k | d ) cost for dense A ).
Revised: The algorithm alternates the fullupdate for x k in Step 4 ( O ( d ) cost) with an incremental block coordinate update for y k in Steps 5 and6 (with O ( | S j k | d ) cost for dense A ).

Original: The auxiliary variables z k and q k accumulate the cancellationterms in the estimation sequence and give a pathway to a straightforward development of the laziﬁed
Revised: The auxiliary variables z k and q k accumulate the cancellationterms in the estimation sequence and give a pathway to a straightforward development of the laziﬁed

Original: CLVR, which appears as Algorithm 2 in the appendix.
Revised: CLVR, which appears as Algorithm 2 in the appendix.

Original: The cost of updating auxiliary vectors z k andq k is O ( | S j k | d ) and O ( d ) , respectively.
Revised: The cost of updating auxiliary vectors z k andq k is O ( | S j k | d ) and O ( d ) , respectively.

Original: In essence, CLVR is a primal-dual coordinate method thatuses a dual averaging update for x k , then updates the state variables { q k } by a linear recursion , andcomputes x k from q k − 1 via a proximal step without direct dependence on x k − 1 .
Revised: In essence, CLVR is a primal-dual coordinate method thatuses a dual averaging update for x k , then updates the state variables { q k } by a linear recursion , andcomputes x k from q k − 1 via a proximal step without direct dependence on x k − 1 .

Original: The output ˜ x
Revised: The output ˜ x

Original: Kis a convex combination of the iterates { x k } Kk =1 , as is standard for primal-dual methods.
Revised: Kis a convex combination of the iterates { x k } Kk =1 , as is standard for primal-dual methods.

Original: However,˜ y K is only an afﬁne (not convex) combination of { y k } Kk =0 , as it involves the term − ( m − 1)
Revised: However,˜ y K is only an afﬁne (not convex) combination of { y k } Kk =0 , as it involves the term − ( m − 1)

Original: y
Revised: y

Original: 0(whose coefﬁcient is negative) and some of the coefﬁcients ma k − ( m − 1) a k +1 multiplying y kfor k ∈ { 1 , . . .
Revised: 0(whose coefﬁcient is negative) and some of the coefﬁcients ma k − ( m − 1) a k +1 multiplying y kfor k ∈ { 1 , . . .

Original: K − 1 } may also be negative.
Revised: K − 1 } may also be negative.

Original: An afﬁne combination still provides valid boundsbecause the dual variable vector y appears linearly in (PD-GLP).
Revised: An afﬁne combination still provides valid boundsbecause the dual variable vector y appears linearly in (PD-GLP).

Original: Moreover, in Step 9, the termma k ( z k − z k − 1 ) serves to cancel certain errors from the randomization of the update w.r.t.
Revised: Moreover, in Step 9, the termma k ( z k − z k − 1 ) serves to cancel certain errors from the randomization of the update w.r.t.

Original: y k , thusplaying a key role in implicit variance reduction.
Revised: y k , thusplaying a key role in implicit variance reduction.

Original: Theorem 1 provides the convergence results for Algorithm 1.
Revised: Theorem 1 provides the convergence results for Algorithm 1.

Original: The proof is provided in Appendix B.
Revised: The proof is provided in Appendix B.

Original: In the theorem (as in the algorithm), γ is a positive parameter that can be tuned.
Revised: In the theorem (as in the algorithm), γ is a positive parameter that can be tuned.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let x k , y k , k ∈ [ K ] , be the iterates of Algorithm 1 and let ˜ x k , ˜
Revised: Let x k , y k , k ∈ [ K ] , be the iterates of Algorithm 1 and let ˜ x k , ˜

Original: y k be deﬁned by
Revised: y k be deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for k ∈ [ K ] .
Revised: for k ∈ [ K ] .

Original: Let W k ⊂ X × R n , k ∈ [ K ] , be a sequence of compact convex sets such that
Revised: Let W k ⊂ X × R n , k ∈ [ K ] , be a sequence of compact convex sets such that

Original: [Equation]
Revised: [Equation]

Original: ˜ x k , ˜ y k ) ∈ W k ⊂ W ⊂ X × R n , where W is also convex and compact.
Revised: ˜ x k , ˜ y k ) ∈ W k ⊂ W ⊂ X × R n , where W is also convex and compact.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: 1: Input: x 0 ∈ X , y 0 ∈ R n , z 

Revised: 1: Input: x 0 ∈ X , y 0 ∈ R n , z

Original: =
Revised: =

Original: A T y 0 , γ >
Revised: A T y 0 , γ >

Original: [Equation]
Revised: [Equation]

Original: ˆ L
Revised: ˆ L

Original: [Equation]
Revised: [Equation]

Original: 0 , σ ≥ 0 , K, m, { S 1 , S 2 , . . .
Revised: 0 , σ ≥ 0 , K, m, { S 1 , S 2 , . . .

Original: [Equation]
Revised: [Equation]

Original: S m
Revised: S m

Original: [Equation]
Revised: [Equation]

Original: 2: a 1 =
Revised: 2: a 1 =

Original: A 1 = 1 2ˆ Lm , q
Revised: A 1 = 1 2ˆ Lm , q

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ( z
Revised: ( z

Original: [Equation]
Revised: [Equation]

Original: + c ) .
Revised: + c ) .

Original: 3: for k = 1 , 2 , . . .
Revised: 3: for k = 1 , 2 , . . .

Original: , K do 4: x k = prox
Revised: , K do 4: x k = prox

Original: [Equation]
Revised: [Equation]

Original: A k r ( x 0 − 1 γ q k − 1 ) .
Revised: A k r ( x 0 − 1 γ q k − 1 ) .

Original: Pick j k uniformly at random in [ m ] .
Revised: Pick j k uniformly at random in [ m ] .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 8: z
Revised: 8: z

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: − y
Revised: − y

Original: [Equation]
Revised: [Equation]

Original: 9: q k = q k
Revised: 9: q k = q k

Original: [Equation]
Revised: [Equation]

Original: 1 + a k +1 (cid:0) z k + c (cid:1) + ma k ( z k − z k
Revised: 1 + a k +1 (cid:0) z k + c (cid:1) + ma k ( z k − z k

Original: [Equation]
Revised: [Equation]

Original: 10: end for 11: return ˜ x K = 1 A
Revised: 10: end for 11: return ˜ x K = 1 A

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( y
Revised: ( y

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: where ( ˆ u , ˆ v ) =
Revised: where ( ˆ u , ˆ v ) =

Original: arg sup ( u , v ) ∈W k {L ( ˜ x k , v ) − L ( u , ˜ y k ) } .
Revised: arg sup ( u , v ) ∈W k {L ( ˜ x k , v ) − L ( u , ˜ y k ) } .

Original: Furthermore,
Revised: Furthermore,

Original: E (cid:104) γ + σA k
Revised: E (cid:104) γ + σA k

Original: [Equation]
Revised: [Equation]

Original: Deﬁne K 

Revised: Deﬁne K

Original: = (cid:6) σ 18ˆ Lmγ (cid:7) .
Revised: = (cid:6) σ 18ˆ Lmγ (cid:7) .

Original: Then in the bounds above:
Revised: Then in the bounds above:

Original: [Equation]
Revised: [Equation]

Original: Observe that (ˆ u , ˆ v ) in the theorem statement exists because of compactness of W k and our assumptions on r ( · ) .
Revised: Observe that (ˆ u , ˆ v ) in the theorem statement exists because of compactness of W k and our assumptions on r ( · ) .

Original: The parameter γ can be tuned to balance the relative weights of primal and dualinitial quantities (cid:107) x ∗ − x 0 (cid:107) and (cid:107) y ∗ − y 0 (cid:107) (or estimates of these quantities), which can signiﬁcantlyinﬂuence practical performance of the method.
Revised: The parameter γ can be tuned to balance the relative weights of primal and dualinitial quantities (cid:107) x ∗ − x 0 (cid:107) and (cid:107) y ∗ − y 0 (cid:107) (or estimates of these quantities), which can signiﬁcantlyinﬂuence practical performance of the method.

Original: In addition to the guarantee on the variational form, due to the linear structure, we also provide explicitguarantees for both the objective and the constraints in (GLP), stated in the following corollary.
Revised: In addition to the guarantee on the variational form, due to the linear structure, we also provide explicitguarantees for both the objective and the constraints in (GLP), stated in the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: In Algorithm 1, for all k ≥ 1 , ˜ x k satisﬁes
Revised: In Algorithm 1, for all k ≥ 1 , ˜ x k satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 ]
Revised: (cid:107) 2 ]

Original: A k , where v = 2 (cid:107) y ∗ (cid:107) (cid:107)
Revised: A k , where v = 2 (cid:107) y ∗ (cid:107) (cid:107)

Original: A ˜ x k − b (cid:107) (
Revised: A ˜ x k − b (cid:107) (

Original: A ˜ x k − b ) .
Revised: A ˜ x k − b ) .

Original: In CLVR , we allow for arbitrary ( x 0 , y 0 )
Revised: In CLVR , we allow for arbitrary ( x 0 , y 0 )

Original: ∈
Revised: ∈

Original: X × R n .
Revised: X × R n .

Original: Nevertheless, by setting y = 0 n , we canobtain z 0 = 

Revised: Nevertheless, by setting y = 0 n , we canobtain z 0 =

Original: d at no cost — a useful strategy for large-scale problems since it avoids the (potentiallyexpensive) single matrix-vector multiplication w.r.t.
Revised: d at no cost — a useful strategy for large-scale problems since it avoids the (potentiallyexpensive) single matrix-vector multiplication w.r.t.

Original: A .
Revised: A .

Original: 3.2 Lazy Update for Sparse and Structured (PD-GLP)
Revised: 3.2 Lazy Update for Sparse and Structured (PD-GLP)

Original: In Algorithm 1, direct computation of the iterates ( x k , y k ) and the output points (˜ x k , ˜ y k ) can beexpensive.
Revised: In Algorithm 1, direct computation of the iterates ( x k , y k ) and the output points (˜ x k , ˜ y k ) can beexpensive.

Original: However, [17] showed that it is possible to only update the averaged vector in thecoordinate block chosen for that iteration.
Revised: However, [17] showed that it is possible to only update the averaged vector in thecoordinate block chosen for that iteration.

Original: This strategy requires us to record the most recent updatefor each coordinate block and update it only when it is selected again, which is tricky and needs tobe implemented carefully.
Revised: This strategy requires us to record the most recent updatefor each coordinate block and update it only when it is selected again, which is tricky and needs tobe implemented carefully.

Original: For sparse and block coordinate-separable instances of (PD-GLP), weshow that by introducing auxiliary variables that are sparsely connected, we can signiﬁcantly simplify
Revised: For sparse and block coordinate-separable instances of (PD-GLP), weshow that by introducing auxiliary variables that are sparsely connected, we can signiﬁcantly simplify

Original: CLVR and make its complexity scale independently of the ambient dimension n · d , instead scalingwith nnz( A ) .
Revised: CLVR and make its complexity scale independently of the ambient dimension n · d , instead scalingwith nnz( A ) .

Original: Due to space constraints, we defer technical details, including the lazy version of
Revised: Due to space constraints, we defer technical details, including the lazy version of

Original: CLVR and associated proofs, to Appendix A.
Revised: CLVR and associated proofs, to Appendix A.

Original: 3.3 Restart Scheme
Revised: 3.3 Restart Scheme

Original: We now propose a ﬁxed restart strategy with a ﬁxed number of iterations per each restart epoch anddiscuss an adaptive restart strategy for the special case of standard-form LP, which corresponds to(GLP) with r ( x ) ≡ 0 and X = { x : x i ≥ 0 , i ∈
Revised: We now propose a ﬁxed restart strategy with a ﬁxed number of iterations per each restart epoch anddiscuss an adaptive restart strategy for the special case of standard-form LP, which corresponds to(GLP) with r ( x ) ≡ 0 and X = { x : x i ≥ 0 , i ∈

Original: [ d ] } .
Revised: [ d ] } .

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: and the primal-dual form
Revised: and the primal-dual form

Original: [Equation]
Revised: [Equation]

Original: This problem has a sharpness property that can be used to obtain linear convergence in ﬁrst-ordermethods [9].
Revised: This problem has a sharpness property that can be used to obtain linear convergence in ﬁrst-ordermethods [9].

Original: For convenience, in the following, we deﬁne w =
Revised: For convenience, in the following, we deﬁne w =

Original: ( x , y ) , ˆ w = (ˆ x , ˆ y ) , ˜ w =
Revised: ( x , y ) , ˆ w = (ˆ x , ˆ y ) , ˜ w =

Original: (˜ x , ˜ y )and w ∗ =
Revised: (˜ x , ˜ y )and w ∗ =

Original: ( x ∗ , y ∗ ) .
Revised: ( x ∗ , y ∗ ) .

Original: Meanwhile, for γ > 0 , we denote the weighted norm (cid:107) w (cid:107) ( γ ) := 237 (cid:113) γ (cid:107) x − x ∗ (cid:107) 22 + 1 γ (cid:107) y − y ∗ (cid:107) 22 .
Revised: Meanwhile, for γ > 0 , we denote the weighted norm (cid:107) w (cid:107) ( γ ) := 237 (cid:113) γ (cid:107) x − x ∗ (cid:107) 22 + 1 γ (cid:107) y − y ∗ (cid:107) 22 .

Original: Further, we use W ∗ to denote the optimal solution set of the LPand deﬁne the distance to W ∗ by dist ( w , W ∗ ) ( γ ) =
Revised: Further, we use W ∗ to denote the optimal solution set of the LPand deﬁne the distance to W ∗ by dist ( w , W ∗ ) ( γ ) =

Original: min w ∗ ∈W ∗ (cid:107) w − w ∗ (cid:107) ( γ ) .
Revised: min w ∗ ∈W ∗ (cid:107) w − w ∗ (cid:107) ( γ ) .

Original: When γ = 1 , (cid:107) · (cid:107) ( γ )is the standard Euclidean norm.
Revised: When γ = 1 , (cid:107) · (cid:107) ( γ )is the standard Euclidean norm.

Original: Then based on (PD-LP), we can use the following classical LPmetric 3 to measure the progress of iterative algorithms for LP:
Revised: Then based on (PD-LP), we can use the following classical LPmetric 3 to measure the progress of iterative algorithms for LP:

Original: [Equation]
Revised: [Equation]

Original: which can be explicitly and directly computed.
Revised: which can be explicitly and directly computed.

Original: For the Euclidean case ( γ = 1 ), it is well-known [30]that there exists a Hoffman constant H 1 such that
Revised: For the Euclidean case ( γ = 1 ), it is well-known [30]that there exists a Hoffman constant H 1 such that

Original: [Equation]
Revised: [Equation]

Original: Using the equivalence of norms in ﬁnite dimensions, for general γ > 0 , we can conclude that thereexists another constant H γ (to which we refer as the generalized Hoffman’s constant) such that
Revised: Using the equivalence of norms in ﬁnite dimensions, for general γ > 0 , we can conclude that thereexists another constant H γ (to which we refer as the generalized Hoffman’s constant) such that

Original: [Equation]
Revised: [Equation]

Original: Using Eq.
Revised: Using Eq.

Original: (9) and Theorem 1, we then obtain the following bounds for distance and LPMetric.
Revised: (9) and Theorem 1, we then obtain the following bounds for distance and LPMetric.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the CLVR algorithm applied to the standard-form LP problem (PD-LP) , withinput w 0 and output ˜ w k .
Revised: Consider the CLVR algorithm applied to the standard-form LP problem (PD-LP) , withinput w 0 and output ˜ w k .

Original: Given γ > 0 , deﬁne w ∗ =
Revised: Given γ > 0 , deﬁne w ∗ =

Original: arg min w ∈W ∗ (cid:107) w 0 − w (cid:107) ( γ ) , and deﬁne
Revised: arg min w ∈W ∗ (cid:107) w 0 − w (cid:107) ( γ ) , and deﬁne

Original: C 0 =
Revised: C 0 =

Original: [Equation]
Revised: [Equation]

Original: γ ) + (cid:107)
Revised: γ ) + (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ∗ (cid:107) ( γ ) .
Revised: ∗ (cid:107) ( γ ) .

Original: Then for H γ deﬁned as in (9) , we have
Revised: Then for H γ deﬁned as in (9) , we have

Original: [Equation]
Revised: [Equation]

Original: As a result, by Theorem 2, if we know the values of ˆ L, (cid:107) w ∗ (cid:107) ( γ ) and H γ , then by setting k = 100ˆ LmC 

Revised: As a result, by Theorem 2, if we know the values of ˆ L, (cid:107) w ∗ (cid:107) ( γ ) and H γ , then by setting k = 100ˆ LmC

Original: H γ k ,we can halve the square root of the distance and the LPMetric in expectation.
Revised: H γ k ,we can halve the square root of the distance and the LPMetric in expectation.

Original: Thus we can obtainlinear convergence if we restart the CLVR algorithm after a ﬁxed number of iterations.
Revised: Thus we can obtainlinear convergence if we restart the CLVR algorithm after a ﬁxed number of iterations.

Original: However, thevalues of (cid:107) w ∗ (cid:107) ( γ ) and H γ are often unknown and thus make this strategy unrealistic in practice.
Revised: However, thevalues of (cid:107) w ∗ (cid:107) ( γ ) and H γ are often unknown and thus make this strategy unrealistic in practice.

Original: Compared with the above ﬁxed restart strategy, a natural strategy is to restart whenever the LPMetrichalves (summarized in Algorithm 4 in the appendix).
Revised: Compared with the above ﬁxed restart strategy, a natural strategy is to restart whenever the LPMetrichalves (summarized in Algorithm 4 in the appendix).

Original: Since LPMetric is easy to monitor and update,implementation of this strategy is straightforward.
Revised: Since LPMetric is easy to monitor and update,implementation of this strategy is straightforward.

Original: However, bounding the number of iterationsrequired to halve the metric (in expectation or with high probability) seems nontrivial.
Revised: However, bounding the number of iterationsrequired to halve the metric (in expectation or with high probability) seems nontrivial.

Original: What can besaid (based on Theorem 2 and denoting by K the number of iterations on CLVR between restarts) is that
Revised: What can besaid (based on Theorem 2 and denoting by K the number of iterations on CLVR between restarts) is that

Original: P [ K > 50ˆ LmC 

Revised: P [ K > 50ˆ LmC

Original: δ 2 H γ ] ≤ δ.
Revised: δ 2 H γ ] ≤ δ.

Original: This follows by Markov inequality, as P [ K > k ] =
Revised: This follows by Markov inequality, as P [ K > k ] =

Original: P (cid:104)(cid:112) LPMetric( ˜ w k ) >
Revised: P (cid:104)(cid:112) LPMetric( ˜ w k ) >

Original: [Equation]
Revised: [Equation]

Original: (cid:105) ≤ 5 (cid:113) 2 ˆ LmC
Revised: (cid:105) ≤ 5 (cid:113) 2 ˆ LmC

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: k .
Revised: k .

Original: Although we use adaptive restart in our experiments, we defer itsconvergence analysis to future work.
Revised: Although we use adaptive restart in our experiments, we defer itsconvergence analysis to future work.

Original: Finally, as an independent and parallel work to ours, [40]proposed a high probability guarantee for scheduled restart for stochastic extragradient-type methods.
Revised: Finally, as an independent and parallel work to ours, [40]proposed a high probability guarantee for scheduled restart for stochastic extragradient-type methods.

Original:  Application: DRO
Revised:  Application: DRO

Original: Consider sample vectors { a 1 , a 2 , . .
Revised: Consider sample vectors { a 1 , a 2 , . .

Original: . , a n } with labels { b 1 , b 2 , . . .
Revised: . , a n } with labels { b 1 , b 2 , . . .

Original: , b n } , where b i ∈ { 1 , − 1 } ( i ∈ [ n ]) .
Revised: , b n } , where b i ∈ { 1 , − 1 } ( i ∈ [ n ]) .

Original: The DRO problem with f -divergence based ambiguity set is
Revised: The DRO problem with f -divergence based ambiguity set is

Original: [Equation]
Revised: [Equation]

Original: where P ρ,n = (cid:8) p ∈ R n : (cid:80) ni =1 p i = 1 , p i ≥ 0 ( i ∈ [ n ]) , D f ( p (cid:107) 1 /n ) ≤ ρn (cid:9) is the ambiguity set, gis a convex loss function and D f is an f -divergence deﬁned by D f ( p (cid:107) q )
Revised: where P ρ,n = (cid:8) p ∈ R n : (cid:80) ni =1 p i = 1 , p i ≥ 0 ( i ∈ [ n ]) , D f ( p (cid:107) 1 /n ) ≤ ρn (cid:9) is the ambiguity set, gis a convex loss function and D f is an f -divergence deﬁned by D f ( p (cid:107) q )

Original: = (cid:80) i =1 q i f ( p i /q
Revised: = (cid:80) i =1 q i f ( p i /q

Original: i ) withp , q ∈ (cid:8) p ∈ R n : (cid:80) ni
Revised: i ) withp , q ∈ (cid:8) p ∈ R n : (cid:80) ni

Original: =1 p i = 1 , p i ≥ 0 (cid:9) and f being a convex function [44].
Revised: =1 p i = 1 , p i ≥ 0 (cid:9) and f being a convex function [44].

Original: The formulation(10) is a nonbilinearly coupled convex-concave min-max problem with constraint set P ρ,n for whichefﬁcient projections are not available in general.
Revised: The formulation(10) is a nonbilinearly coupled convex-concave min-max problem with constraint set P ρ,n for whichefﬁcient projections are not available in general.

Original: When g is a nonsmooth loss (e.g., the hinge loss),many well-known methods such as the extragradient [34, 45] cannot be used even if we could projectonto P ρ,n efﬁciently.
Revised: When g is a nonsmooth loss (e.g., the hinge loss),many well-known methods such as the extragradient [34, 45] cannot be used even if we could projectonto P ρ,n efﬁciently.

Original: However, by introducing auxiliary variables, additional linear constraints, andsimple convex constraints, we can make the interacting term between primal and dual variablesbilinear, as shown next.
Revised: However, by introducing auxiliary variables, additional linear constraints, andsimple convex constraints, we can make the interacting term between primal and dual variablesbilinear, as shown next.

Original: (See Appendix C for a proof.)
Revised: (See Appendix C for a proof.)

Original: Theorem 3.
Revised: Theorem 3.

Original: Let X be a compact convex set.
Revised: Let X be a compact convex set.

Original: Then the DRO problem in Eq.
Revised: Then the DRO problem in Eq.

Original: (10) is equivalent to
Revised: (10) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: In Theorem 3, the domain of the one-dimensional convex function f ∗ ( · ) is an interval such as [ a, b ] ,so that q i ∈ µ i dom( f ∗ ) denotes the inequality µ i a ≤ q i ≤ µ i b .
Revised: In Theorem 3, the domain of the one-dimensional convex function f ∗ ( · ) is an interval such as [ a, b ] ,so that q i ∈ µ i dom( f ∗ ) denotes the inequality µ i a ≤ q i ≤ µ i b .

Original: Since the perspective functionµf ∗ (cid:0) qµ (cid:1) is a simple convex function of two variables, we can assume that the proximal operatorfor this function on the domain { ( µ, q ) : q ∈ µ dom( f ∗ ) , µ > 0 } can be computed efﬁciently [11].
Revised: Since the perspective functionµf ∗ (cid:0) qµ (cid:1) is a simple convex function of two variables, we can assume that the proximal operatorfor this function on the domain { ( µ, q ) : q ∈ µ dom( f ∗ ) , µ > 0 } can be computed efﬁciently [11].

Original: Similarly, we can assume that the constraint g ( u ) ≤ w admits an efﬁciently computable projectionoperator.
Revised: Similarly, we can assume that the constraint g ( u ) ≤ w admits an efﬁciently computable projectionoperator.

Original: As a result, the formulation (10) can be solved by CLVR .
Revised: As a result, the formulation (10) can be solved by CLVR .

Original: When expressing (10) in theform of (PD-GLP), the primal and dual variable vectors have dimensions d + 1 + 4 n and 3 n − 1 ,respectively.
Revised: When expressing (10) in theform of (PD-GLP), the primal and dual variable vectors have dimensions d + 1 + 4 n and 3 n − 1 ,respectively.

Original: However, according to Table 1, provided that X is coordinate separable, the overallcomplexity of CLVR will only be O (cid:0) (nnz( A )+ n )( R +1)
Revised: However, according to Table 1, provided that X is coordinate separable, the overallcomplexity of CLVR will only be O (cid:0) (nnz( A )+ n )( R +1)

Original: (cid:15) (cid:1) .
Revised: (cid:15) (cid:1) .

Original: The original DRO problem with Wasserstein metric based ambiguity set is an inﬁnite -dimensionalnonbilinearly coupled convex-concave min-max problem deﬁned by
Revised: The original DRO problem with Wasserstein metric based ambiguity set is an inﬁnite -dimensionalnonbilinearly coupled convex-concave min-max problem deﬁned by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where a ∈ R d , b ∈ { 1 , − 1 } , P is a distribution on R d ×{ 1 , − 1 } , g is a convex loss function and P ρ,κis the Wasserstein metric-based ambiguity set [52].
Revised: where a ∈ R d , b ∈ { 1 , − 1 } , P is a distribution on R d ×{ 1 , − 1 } , g is a convex loss function and P ρ,κis the Wasserstein metric-based ambiguity set [52].

Original: Our reformulation for Eq.
Revised: Our reformulation for Eq.

Original: (11) is in Appendix C.2.
Revised: (11) is in Appendix C.2.

Original:  Numerical Experiments and Discussion
Revised:  Numerical Experiments and Discussion

Original: We provide experimental evaluations of our algorithm for the reformulation of the DRO with Wasserstein metric of (cid:96) 1 -norm (with κ = 0 .
Revised: We provide experimental evaluations of our algorithm for the reformulation of the DRO with Wasserstein metric of (cid:96) 1 -norm (with κ = 0 .

Original: 1 and ρ = 10 ) and hinge loss.
Revised: 1 and ρ = 10 ) and hinge loss.

Original: For its LP formulation (see
Revised: For its LP formulation (see

Original: Theorem 4 in the Appendix), we compare our CLVR method with three representative methods:
Revised: Theorem 4 in the Appendix), we compare our CLVR method with three representative methods:

Original: PDHG [14], SPDHG [15] and PURE - CD [5].
Revised: PDHG [14], SPDHG [15] and PURE - CD [5].

Original: For all algorithms we use LPMetric (7) as the performancemeasure and use a restart strategy of successive halving of LPMetric (Section 3.3) to obtain linearconvergence.
Revised: For all algorithms we use LPMetric (7) as the performancemeasure and use a restart strategy of successive halving of LPMetric (Section 3.3) to obtain linearconvergence.

Original: We implemented CLVR and other algorithms in Julia, optimizing all implementations tothe best of our ability.
Revised: We implemented CLVR and other algorithms in Julia, optimizing all implementations tothe best of our ability.

Original: Full details for the experimental setting can be found in Appendix D.
Revised: Full details for the experimental setting can be found in Appendix D.

Original: [Table]
Revised: [Table]

Original: Comparison with Primal-Dual Algorithms.
Revised: Comparison with Primal-Dual Algorithms.

Original: Figure 1 provides an algorithm comparison in termsof the number of data passes and wall-clock time.
Revised: Figure 1 provides an algorithm comparison in termsof the number of data passes and wall-clock time.

Original: The spikes in all the plots are due restarts; at thebeginning of each cycle, the value of LP metric increases signiﬁcantly, then decreases rapidly.
Revised: The spikes in all the plots are due restarts; at thebeginning of each cycle, the value of LP metric increases signiﬁcantly, then decreases rapidly.

Original: Forthe number of data passes (top row), CLVR with block size 1 and PURE - CD perform best on all threedatasets.
Revised: Forthe number of data passes (top row), CLVR with block size 1 and PURE - CD perform best on all threedatasets.

Original: CLVR with block size 10 and SPDHG with block size 50 have second-tier performance, while
Revised: CLVR with block size 10 and SPDHG with block size 50 have second-tier performance, while

Original: PDHG is worst.
Revised: PDHG is worst.

Original: For the CLVR algorithm, smaller block size corresponds to smaller ˆ L in Assumption 3,which corresponds to better complexity in terms of data passes by Theorem 1.
Revised: For the CLVR algorithm, smaller block size corresponds to smaller ˆ L in Assumption 3,which corresponds to better complexity in terms of data passes by Theorem 1.

Original: Nevertheless, the gapbetween empirical performance and theoretical guarantee for SPDHG and PURE - CD deserves furtherresearch because, to date, they have only been shown to have the same iteration complexity as PDHG .
Revised: Nevertheless, the gapbetween empirical performance and theoretical guarantee for SPDHG and PURE - CD deserves furtherresearch because, to date, they have only been shown to have the same iteration complexity as PDHG .

Original: Empirically on a9a , CLVR with block size 1 performs better than PURE - CD in terms of data passes.
Revised: Empirically on a9a , CLVR with block size 1 performs better than PURE - CD in terms of data passes.

Original: In terms of wall-clock time (bottom row), because of different per-iteration costs of each algorithmand instruction-level parallelism in modern processors [28], the plots differ signiﬁcantly from theplots for number of data passes.
Revised: In terms of wall-clock time (bottom row), because of different per-iteration costs of each algorithmand instruction-level parallelism in modern processors [28], the plots differ signiﬁcantly from theplots for number of data passes.

Original: Even with block size 50 , SPDHG spends the most wall-clock timefor one data pass and is the slowest on sparse datasets a9a and rcv1 , but is faster than PDHG onthe dense dataset gisette .
Revised: Even with block size 50 , SPDHG spends the most wall-clock timefor one data pass and is the slowest on sparse datasets a9a and rcv1 , but is faster than PDHG onthe dense dataset gisette .

Original: Meanwhile, while CLVR with block size 10 is not best in terms of datapasses, it remains fastest in terms of wall-clock time on all datasets due to cheaper per-iteration costand instruction-level parallelism.
Revised: Meanwhile, while CLVR with block size 10 is not best in terms of datapasses, it remains fastest in terms of wall-clock time on all datasets due to cheaper per-iteration costand instruction-level parallelism.

Original: On rcv1 , the per-iteration cost of PURE - CD is about 60% of that of
Revised: On rcv1 , the per-iteration cost of PURE - CD is about 60% of that of

Original: CLVR with block size 1 .
Revised: CLVR with block size 1 .

Original: Hence, despite having similar performance in terms of data passes, PURE - CDis faster than CLVR with block size 1 , but is still slower than CLVR with block size 10 .
Revised: Hence, despite having similar performance in terms of data passes, PURE - CDis faster than CLVR with block size 1 , but is still slower than CLVR with block size 10 .

Original: Comparison with Off-the-shelf Solvers.
Revised: Comparison with Off-the-shelf Solvers.

Original: Table 2 shows that the performance of CLVR is competitive against state-of-the-art solvers such as GLPK [1] and Gurobi [26].
Revised: Table 2 shows that the performance of CLVR is competitive against state-of-the-art solvers such as GLPK [1] and Gurobi [26].

Original: Moreover, we observethat CLVR reached accurate solutions signiﬁcantly faster than GLPK and Gurobi in the reformulatedproblems with gisette and rcv1 datasets.
Revised: Moreover, we observethat CLVR reached accurate solutions signiﬁcantly faster than GLPK and Gurobi in the reformulatedproblems with gisette and rcv1 datasets.

Original: Although CLVR is much slower than Gurobi(barrier) ona9a dataset, we believe that much of the performance gap in this case is due to the redundancy in the problem formulation with the a9a dataset, much of which is removed by Gurobi presolver 4 .
Revised: Although CLVR is much slower than Gurobi(barrier) ona9a dataset, we believe that much of the performance gap in this case is due to the redundancy in the problem formulation with the a9a dataset, much of which is removed by Gurobi presolver 4 .

Original: Weleave presolving and other heuristic speedups of CLVR for future work.
Revised: Weleave presolving and other heuristic speedups of CLVR for future work.

Original: CLVR is fastest in both the number of data passes and wall-clock time among all primal-dualalgorithms we compared against on these datasets and it is also competitive with off-the-shelf solvers.
Revised: CLVR is fastest in both the number of data passes and wall-clock time among all primal-dualalgorithms we compared against on these datasets and it is also competitive with off-the-shelf solvers.

Original: Since it has the best known theoretical guarantee, we believe that CLVR could be a method of choice.
Revised: Since it has the best known theoretical guarantee, we believe that CLVR could be a method of choice.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes] All theoretical claims are supported by proofs, which,due to space constraints, are provided in the appendix.
Revised: [Yes] All theoretical claims are supported by proofs, which,due to space constraints, are provided in the appendix.

Original: Numerical results are providedin Section 5, with full technical details of the experiments provided in Appendix D.(b)
Revised: Numerical results are providedin Section 5, with full technical details of the experiments provided in Appendix D.(b)

Original: Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] The three main examples are: (i)the discussion of lazy update strategy and guarantees for ˆ x k versus ˜ x k in Appendix A;(ii) the discussion of linear convergence under the restart strategy and theoreticalguarantees for scheduled restart versus empirical guarantees for adaptive restart in
Revised: [Yes] The three main examples are: (i)the discussion of lazy update strategy and guarantees for ˆ x k versus ˜ x k in Appendix A;(ii) the discussion of linear convergence under the restart strategy and theoreticalguarantees for scheduled restart versus empirical guarantees for adaptive restart in

Original: Section 3.3; and (iii) comparison to off-the-shelf solvers in Section 5 and use ofpreconditioning.(c)
Revised: Section 3.3; and (iii) comparison to off-the-shelf solvers in Section 5 and use ofpreconditioning.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: This workconcerns a generic solver for generalized linear programs and as such is not tied to anyspeciﬁc application that could have a potential negative societal impact.(d)
Revised: This workconcerns a generic solver for generalized linear programs and as such is not tied to anyspeciﬁc application that could have a potential negative societal impact.(d)

Original: Have you read the ethics review guidelines
Revised: Have you read the ethics review guidelines

Original: and ensured that your paper conforms to
Revised: and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See assumptionsin Section 2.
Revised: [Yes] See assumptionsin Section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B andC for complete proofs.
Revised: [Yes] See Appendix B andC for complete proofs.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We include ourcode in the supplementary material.
Revised: [Yes] We include ourcode in the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: In Section 5 and Appendix D.
Revised: In Section 5 and Appendix D.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running ex-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running ex-

Original: periments multiple times)?
Revised: periments multiple times)?

Original: [No] Error bars are not reported due to almost identical
Revised: [No] Error bars are not reported due to almost identical

Original: behaviors and results from the large number of stochastic iterations.
Revised: behaviors and results from the large number of stochastic iterations.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] In Appendix D.
Revised: [Yes] In Appendix D.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We used datasets from
Revised: [Yes] We used datasets from

Original: LibSVM, which is a open source machine learning library and cited the creators [16]
Revised: LibSVM, which is a open source machine learning library and cited the creators [16]

Original: in Appendix D which provides full experimental details.
Revised: in Appendix D which provides full experimental details.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] LibSVM datasets are distributed
Revised: [N/A] LibSVM datasets are distributed

Original: under the BSD license which is “a low restriction type of license for open source
Revised: under the BSD license which is “a low restriction type of license for open source

Original: software that does not put requirements on redistribution.”
Revised: software that does not put requirements on redistribution.”

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: Our code is provided in the supplementary material.
Revised: Our code is provided in the supplementary material.

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: As mentioned before, the datasets we use are open source.
Revised: As mentioned before, the datasets we use are open source.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: Coordinate Linear Variance Reduction
Revised: Coordinate Linear Variance Reduction

Original: for Generalized Linear Programming
Revised: for Generalized Linear Programming

Original: Appendix
Revised: Appendix

Original: Outline.
Revised: Outline.

Original: The appendix of this paper is organized as follows:
Revised: The appendix of this paper is organized as follows:

Original: • Section A contains details and the pseudo code for CLVR with lazy update.
Revised: • Section A contains details and the pseudo code for CLVR with lazy update.

Original: • Section B provides the proofs for Section 3.
Revised: • Section B provides the proofs for Section 3.

Original: • Section C provides the proofs for Section 4.•
Revised: • Section C provides the proofs for Section 4.•

Original: Section D provides the complementary details for Section 5.
Revised: Section D provides the complementary details for Section 5.

Original: A Lazy Update for Sparse and Structured Instances of (PD-GLP)
Revised: A Lazy Update for Sparse and Structured Instances of (PD-GLP)

Original: In Algorithm 1, for dense A , the O ( | S j k | d ) cost of Steps 6 and 8 dominates the O ( d ) cost of Steps 4and 9.
Revised: In Algorithm 1, for dense A , the O ( | S j k | d ) cost of Steps 6 and 8 dominates the O ( d ) cost of Steps 4and 9.

Original: However, when A is sparse and | S j k | is small, the cost of Steps 6 and 8 will be O (nnz( A S jk )) ,which may be less than the O ( d ) cost of Steps 4 and 9.
Revised: However, when A is sparse and | S j k | is small, the cost of Steps 6 and 8 will be O (nnz( A S jk )) ,which may be less than the O ( d ) cost of Steps 4 and 9.

Original: Using this observation, we show that the natureof the dual averaging update enables us to propose an efﬁcient implementation whose complexitydepends on nnz( A ) rather than n · d .
Revised: Using this observation, we show that the natureof the dual averaging update enables us to propose an efﬁcient implementation whose complexitydepends on nnz( A ) rather than n · d .

Original: Recall that we partition [ n ] into subsets { S 1 , S 2 , . . .
Revised: Recall that we partition [ n ] into subsets { S 1 , S 2 , . . .

Original: , S m } and use A S j ( j ∈ [ m ]) to denote the j throw block of A .
Revised: , S m } and use A S j ( j ∈ [ m ]) to denote the j throw block of A .

Original: For each block A S j , we use C j ⊂ [ d ] to denote the indices of those columns of A S jthat contain at least one nonzero element.
Revised: For each block A S j , we use C j ⊂ [ d ] to denote the indices of those columns of A S jthat contain at least one nonzero element.

Original: (Of course, { C 1 , . . .
Revised: (Of course, { C 1 , . . .

Original: , C m } is not in general a partition of[ d ] as different subsets S
Revised: , C m } is not in general a partition of[ d ] as different subsets S

Original: j may have nonzeros in the same columns.)
Revised: j may have nonzeros in the same columns.)

Original: We assume further that X and rare coordinate separable, that is, X
Revised: We assume further that X and rare coordinate separable, that is, X

Original: = X 1 × · · · × X d with X i ⊂ R for all i and r ( x ) :
Revised: = X 1 × · · · × X d with X i ⊂ R for all i and r ( x ) :

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =1 r ( x i )with x i ∈
Revised: =1 r ( x i )with x i ∈

Original: X i .
Revised: X i .

Original: In Step 4 of Algorithm 1, the separability of X and r means that an update to one coordinate block ofx — say the C j k block in the update from x k − 1 to x k , which requires a projection and an applicationof the proximal operator — does not inﬂuence other coordinates x ik for i / ∈ C j k .
Revised: In Step 4 of Algorithm 1, the separability of X and r means that an update to one coordinate block ofx — say the C j k block in the update from x k − 1 to x k , which requires a projection and an applicationof the proximal operator — does not inﬂuence other coordinates x ik for i / ∈ C j k .

Original: Moreover, wecan efﬁciently maintain an implicit representation of q k , in terms of a newly introduced auxiliaryvector r k ; see Lemma 8 of Appendix B.2.
Revised: Moreover, wecan efﬁciently maintain an implicit representation of q k , in terms of a newly introduced auxiliaryvector r k ; see Lemma 8 of Appendix B.2.

Original: Similarly, to update ˜ y k efﬁciently, we maintain an implicitrepresentation of ˜ y k via an auxiliary vector s k , as shown in Lemma 9 of Appendix B.2.
Revised: Similarly, to update ˜ y k efﬁciently, we maintain an implicitrepresentation of ˜ y k via an auxiliary vector s k , as shown in Lemma 9 of Appendix B.2.

Original: It is generally not possible to maintain ˜ x k efﬁciently since, in principle, all components of x k canchange on every iteration, and we wish to avoid the O ( d ) cost of evaluating every full x k .
Revised: It is generally not possible to maintain ˜ x k efﬁciently since, in principle, all components of x k canchange on every iteration, and we wish to avoid the O ( d ) cost of evaluating every full x k .

Original: We seekinstead to output a proxy ˆ x K for ˜ x K from Algorithm 1 such that E [ˆ x K ] =
Revised: We seekinstead to output a proxy ˆ x K for ˜ x K from Algorithm 1 such that E [ˆ x K ] =

Original: ˜ x K .
Revised: ˜ x K .

Original: One possible choiceis to pick an index k (cid:48) ∈
Revised: One possible choiceis to pick an index k (cid:48) ∈

Original: [ K ] from the weighted discrete distribution ( a 1 A K , a 2 A K , . . .
Revised: [ K ] from the weighted discrete distribution ( a 1 A K , a 2 A K , . . .

Original: , a K A K ) (computingthe scalar quantities a k and A k for k ∈
Revised: , a K A K ) (computingthe scalar quantities a k and A k for k ∈

Original: [ K ] in advance), then setting ˆ x K = x k (cid:48) .
Revised: [ K ] in advance), then setting ˆ x K = x k (cid:48) .

Original: A slightly moresophisticated strategy is to sample a predetermined number (cid:98) K of vectors x k , k ∈
Revised: A slightly moresophisticated strategy is to sample a predetermined number (cid:98) K of vectors x k , k ∈

Original: [ K ] , and deﬁneˆ x K to be the simple average of these vectors.
Revised: [ K ] , and deﬁneˆ x K to be the simple average of these vectors.

Original: Once again, the indices are chosen from the weighteddiscrete distribution ( a 1 A K , a 2 A K , . .
Revised: Once again, the indices are chosen from the weighteddiscrete distribution ( a 1 A K , a 2 A K , . .

Original: , a K A K ) .
Revised: , a K A K ) .

Original: Note that the total expected cost of the K iterations of thealgorithm (excluding the full-vector updates) is O ( K nnz( A ) /m ) , while the total cost of evaluatingthe (cid:98) K full vectors x k and accumulating them into ˆ x K is O ( (cid:98) Kd ) .
Revised: Note that the total expected cost of the K iterations of thealgorithm (excluding the full-vector updates) is O ( K nnz( A ) /m ) , while the total cost of evaluatingthe (cid:98) K full vectors x k and accumulating them into ˆ x K is O ( (cid:98) Kd ) .

Original: Thus, for the full-step iterationsnot to dominate the total cost, we can choose (cid:98) K to be O
Revised: Thus, for the full-step iterationsnot to dominate the total cost, we can choose (cid:98) K to be O

Original: ( K nnz( A ) / ( md )) .
Revised: ( K nnz( A ) / ( md )) .

Original: Finally, we note that
Revised: Finally, we note that

Original: Theorem 1 is for ˜ x k , not ˆ x K .
Revised: Theorem 1 is for ˜ x k , not ˆ x K .

Original: However, since E [ˆ x k ] = ˜ x k , we expect the convergence rates from thetheorem to hold for ˆ x K in practice.
Revised: However, since E [ˆ x k ] = ˜ x k , we expect the convergence rates from thetheorem to hold for ˆ x K in practice.

Original: An implementation of Algorithm 1 that exploits the form of q k in Lemma 8 and ˜ y k in Lemma 9, andevaluates explicitly only those components of x k needed to perform the rest of the iteration is givenas Algorithm 2.
Revised: An implementation of Algorithm 1 that exploits the form of q k in Lemma 8 and ˜ y k in Lemma 9, andevaluates explicitly only those components of x k needed to perform the rest of the iteration is givenas Algorithm 2.

Original: This version also incorporates the strategy for obtaining ˆ x K by sampling (cid:98) K iterateson which to evaluate the full vector x k .
Revised: This version also incorporates the strategy for obtaining ˆ x K by sampling (cid:98) K iterateson which to evaluate the full vector x k .

Original: Due to the efﬁcient implementation in Algorithm 2, to attain an (cid:15) -accurate solution in terms of theprimal-dual gap in Theorem 1, we need O ( nnz ( A )ˆ L (cid:15) ) FLOPS, which corresponds to O ( ˆ L(cid:15) ) data passes.
Revised: Due to the efﬁcient implementation in Algorithm 2, to attain an (cid:15) -accurate solution in terms of theprimal-dual gap in Theorem 1, we need O ( nnz ( A )ˆ L (cid:15) ) FLOPS, which corresponds to O ( ˆ L(cid:15) ) data passes.

Original: As a result, because a smaller batch size leads to a smaller ˆ L , we attain the best performance in terms of number of data passes when the batch size is set to one.
Revised: As a result, because a smaller batch size leads to a smaller ˆ L , we attain the best performance in terms of number of data passes when the batch size is set to one.

Original: However, as modern computerarchitecture has particular design for vectorized operations, lower runtime is obtained for a smallbatch size strictly larger than one (see Section 5).
Revised: However, as modern computerarchitecture has particular design for vectorized operations, lower runtime is obtained for a smallbatch size strictly larger than one (see Section 5).

Original: Remark 1.
Revised: Remark 1.

Original: While we consider the case of fully coordinate separable r and X for simplicity, ourlazy update approach is also applicable to the coordinate block partitioning case in which X =
Revised: While we consider the case of fully coordinate separable r and X for simplicity, ourlazy update approach is also applicable to the coordinate block partitioning case in which X =

Original: X 1 × · · · X m with X i ∈ R
Revised: X 1 × · · · X m with X i ∈ R

Original: d i ( i ∈ [ m ] , (cid:80) mi =1 d i = d ) and r ( x ) := (cid:80) mi =1 r ( x i ) with x i ∈
Revised: d i ( i ∈ [ m ] , (cid:80) mi =1 d i = d ) and r ( x ) := (cid:80) mi =1 r ( x i ) with x i ∈

Original: X i .
Revised: X i .

Original: Thedifference is that for each coordinate block A S j ( j ∈ [ m ]) , we overload C j ⊂ [ m ] to denote the setof blocks in A S j where each coordinate block contains at least one nonzero element.
Revised: Thedifference is that for each coordinate block A S j ( j ∈ [ m ]) , we overload C j ⊂ [ m ] to denote the setof blocks in A S j where each coordinate block contains at least one nonzero element.

Original: Remark 2.
Revised: Remark 2.

Original: Dual averaging has been shown to have signiﬁcant advantage in producing sparseriterates than mirror descent in the context of online learning [36, 59].
Revised: Dual averaging has been shown to have signiﬁcant advantage in producing sparseriterates than mirror descent in the context of online learning [36, 59].

Original: It further leads to betterbounds in well-conditioned ﬁnite-sum optimization [54].
Revised: It further leads to betterbounds in well-conditioned ﬁnite-sum optimization [54].

Original: In this work, we show that dual averagingoffers better ﬂexibility with sparse matrices than does mirror descent.
Revised: In this work, we show that dual averagingoffers better ﬂexibility with sparse matrices than does mirror descent.

Original: Algorithm
Revised: Algorithm

Original: 1: Input: ˜ x 0 = x 0 = x − 1 ∈ X , y 0 ∈ R n , z 

Revised: 1: Input: ˜ x 0 = x 0 = x − 1 ∈ X , y 0 ∈ R n , z

Original: =
Revised: =

Original: A T y 0 , γ > 0 , ˆ L > 0 , K , (cid:98) K , m , { S 1 , S 2 , . . .
Revised: A T y 0 , γ > 0 , ˆ L > 0 , K , (cid:98) K , m , { S 1 , S 2 , . . .

Original: , S m } , { C 1 , C 2 , . .
Revised: , S m } , { C 1 , C 2 , . .

Original: , C m } .
Revised: , C m } .

Original: 2: a 0 =
Revised: 2: a 0 =

Original: A 0 = 0 , a 1 =
Revised: A 0 = 0 , a 1 =

Original: A 1 = 1 2ˆ Lm , q 0 = a 1 ( z 0 + c ) , r 0 = 

Revised: A 1 = 1 2ˆ Lm , q 0 = a 1 ( z 0 + c ) , r 0 =

Original: d , s 0 = 0 n .
Revised: d , s 0 = 0 n .

Original: 3: for k = 1 , 2 , . . .
Revised: 3: for k = 1 , 2 , . . .

Original: , K − 1 do 4: a k +1 = √ 1+ σA k /γ 2ˆ Lm , A k
Revised: , K − 1 do 4: a k +1 = √ 1+ σA k /γ 2ˆ Lm , A k

Original: +1 =
Revised: +1 =

Original: A k + a k +1 .
Revised: A k + a k +1 .

Original: 5: end for 6: Choose indices { (cid:96) 1 , . . .
Revised: 5: end for 6: Choose indices { (cid:96) 1 , . . .

Original: , (cid:96) (cid:98) K } i.i
Revised: , (cid:96) (cid:98) K } i.i

Original: [Equation]
Revised: [Equation]

Original: d. from [ K ] according to the distribution (cid:110) a 1 A K , a 2 A K , . . .
Revised: d. from [ K ] according to the distribution (cid:110) a 1 A K , a 2 A K , . . .

Original: , a K A K (cid:111) .
Revised: , a K A K (cid:111) .

Original: 7: for k = 1 , 2 , . . .
Revised: 7: for k = 1 , 2 , . . .

Original: , K do 8:
Revised: , K do 8:

Original: Pick j k uniformly at random in [ m ] .
Revised: Pick j k uniformly at random in [ m ] .

Original: 9: if k = (cid:96) i for some i = 1 , 2 , . . .
Revised: 9: if k = (cid:96) i for some i = 1 , 2 , . . .

Original: , (cid:98) K then 10:
Revised: , (cid:98) K then 10:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 12: else 13:
Revised: 12: else 13:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 15: end if 16: y
Revised: 15: end if 16: y

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: B Omitted Proofs from Section 3B.1 Omitted Proofs from Section 3.1
Revised: B Omitted Proofs from Section 3B.1 Omitted Proofs from Section 3.1

Original: We state a version of CLVR in Algorithm 3 that is convenient for the analysis, and is equivalent to
Revised: We state a version of CLVR in Algorithm 3 that is convenient for the analysis, and is equivalent to

Original: Algorithm 1 in the main body of the paper.
Revised: Algorithm 1 in the main body of the paper.

Original: Before analyzing the convergence of CLVR , we justify ourclaim of equivalence in Proposition 1.
Revised: Before analyzing the convergence of CLVR , we justify ourclaim of equivalence in Proposition 1.

Original: Proposition 1.
Revised: Proposition 1.

Original: The iterates of Algorithm 1 and 3 are equivalent.
Revised: The iterates of Algorithm 1 and 3 are equivalent.

Original: Proof.
Revised: Proof.

Original: To argue equivalence, we show that the iterates of Algorithm 1 and 3 solve the same optimization problems.
Revised: To argue equivalence, we show that the iterates of Algorithm 1 and 3 solve the same optimization problems.

Original: To avoid ambiguity, here we will use ˆ x k , ˆ y k to denote the iterates x k , y k in
Revised: To avoid ambiguity, here we will use ˆ x k , ˆ y k to denote the iterates x k , y k in

Original: Algorithm 3, while we retain the notation x k , y k for the iterates of Algorithm 1.
Revised: Algorithm 3, while we retain the notation x k , y k for the iterates of Algorithm 1.

Original: Let us ﬁrst start by writing an equivalent deﬁnition of x k in Algorithm 1.
Revised: Let us ﬁrst start by writing an equivalent deﬁnition of x k in Algorithm 1.

Original: To do so, we ﬁrst unroll therecursive deﬁnitions of z k and q k .
Revised: To do so, we ﬁrst unroll therecursive deﬁnitions of z k and q k .

Original: We can observe that, since by deﬁnition y k and y k − 1 only differover the coordinate block S j k , we have
Revised: We can observe that, since by deﬁnition y k and y k − 1 only differover the coordinate block S j k , we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, using Eq.
Revised: On the other hand, using Eq.

Original: (12), the deﬁnition of q k implies
Revised: (12), the deﬁnition of q k implies

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition of the proximal operator (see Eq.
Revised: Using the deﬁnition of the proximal operator (see Eq.

Original: (1)) and the deﬁnition of x k in Step 4 of
Revised: (1)) and the deﬁnition of x k in Step 4 of

Original: Algorithm 1, we have
Revised: Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Now let us consider the optimization problem that deﬁnes ˆ x k .
Revised: Now let us consider the optimization problem that deﬁnes ˆ x k .

Original: Assume for now that the deﬁnitions ofy k and ˆ y k agree (we justify this claim below).
Revised: Assume for now that the deﬁnitions ofy k and ˆ y k agree (we justify this claim below).

Original: Observe ﬁrst that the minimization problem deﬁningˆ x k is independent of u , so by unrolling the recursion for φ k , we have
Revised: Observe ﬁrst that the minimization problem deﬁningˆ x k is independent of u , so by unrolling the recursion for φ k , we have

Original: [Equation]
Revised: [Equation]

Original: It remains to argue that the deﬁnitions of y k and ˆ y k agree.
Revised: It remains to argue that the deﬁnitions of y k and ˆ y k agree.

Original: First, observe that since the deﬁnitions ofψ k and ψ k − 1 differ only over block S j k , we have ˆ y S
Revised: First, observe that since the deﬁnitions ofψ k and ψ k − 1 differ only over block S j k , we have ˆ y S

Original: j k
Revised: j k

Original: = ˆ y S
Revised: = ˆ y S

Original: j k − 1 for all j (cid:54) =
Revised: j k − 1 for all j (cid:54) =

Original: j k .
Revised: j k .

Original: For j = j k , we haveby unrolling the recursive deﬁnition of ψ k that
Revised: For j = j k , we haveby unrolling the recursive deﬁnition of ψ k that

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: In the following three lemmas, we bound φ k ( x k ) and ψ k ( y k ) below and above, which is thensubsequently used to bound the primal-dual gap in Theorem 1.
Revised: In the following three lemmas, we bound φ k ( x k ) and ψ k ( y k ) below and above, which is thensubsequently used to bound the primal-dual gap in Theorem 1.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: For all steps of Algorithm 3 with k ≥ 1 , we have, ∀ ( u , v ) ∈
Revised: For all steps of Algorithm 3 with k ≥ 1 , we have, ∀ ( u , v ) ∈

Original: X × R n ,
Revised: X × R n ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the deﬁnitions of ψ k ( y ) and φ k ( x ) in Algorithm 3, it follows that, ∀ k ≥ 1 ,
Revised: By the deﬁnitions of ψ k ( y ) and φ k ( x ) in Algorithm 3, it follows that, ∀ k ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Observe that, as function of x , φ k ( x ) is ( γ + σA k ) -strongly convex.
Revised: Observe that, as function of x , φ k ( x ) is ( γ + σA k ) -strongly convex.

Original: As, by deﬁnition, x k =arg min x ∈X φ k ( x ) , it follows that
Revised: As, by deﬁnition, x k =arg min x ∈X φ k ( x ) , it follows that

Original: [Equation]
Revised: [Equation]

Original: Now, writing φ k ( u ) explicitly and rearranging the last inequality, the stated bound on φ k ( x k ) follows.
Revised: Now, writing φ k ( u ) explicitly and rearranging the last inequality, the stated bound on φ k ( x k ) follows.

Original: As a function of y , ψ k ( y ) is 1 /γ -strongly convex.
Revised: As a function of y , ψ k ( y ) is 1 /γ -strongly convex.

Original: The proof for the bound on ψ k uses the sameargument and is omitted.
Revised: The proof for the bound on ψ k uses the sameargument and is omitted.

Original: In the following proof, for k ≥ 1 , let F k denote the natural ﬁltration, containing all the randomnessin the algorithm up to and including iteration k. Recall that A =  
Revised: In the following proof, for k ≥ 1 , let F k denote the natural ﬁltration, containing all the randomnessin the algorithm up to and including iteration k. Recall that A =  

Original: A S...
Revised: A S...

Original: A Sm 
Revised: A Sm 

Original:  , and let A ¯ S j ( j ∈ [ m ])denote the matrix A with its S j block of rows replaced by a zero block.
Revised:  , and let A ¯ S j ( j ∈ [ m ])denote the matrix A with its S j block of rows replaced by a zero block.

Original: For convenience, for k = 1 , 2 , . . .
Revised: For convenience, for k = 1 , 2 , . . .

Original: , we deﬁne
Revised: , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then from the deﬁnition of y k in Algorithm 1, we have E [ y k − y k − 1 |F k − 1 ] = 1 m (ˆ y k − y k − 1 ) .
Revised: Then from the deﬁnition of y k in Algorithm 1, we have E [ y k − y k − 1 |F k − 1 ] = 1 m (ˆ y k − y k − 1 ) .

Original: Motivated by [4], we have the following result.
Revised: Motivated by [4], we have the following result.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given the sequences { y k } in Algorithm 3 and { ˆ y k } in Eq.
Revised: Given the sequences { y k } in Algorithm 3 and { ˆ y k } in Eq.

Original: (18) , we deﬁne the sequence{ ˇ v k } by
Revised: (18) , we deﬁne the sequence{ ˇ v k } by

Original: [Equation]
Revised: [Equation]

Original: Then for any v ∈ R d that may be correlated with the randomness in { ˇ v i } ki =1 , we have
Revised: Then for any v ∈ R d that may be correlated with the randomness in { ˇ v i } ki =1 , we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over all the randomness in the history.
Revised: where the expectation is taken over all the randomness in the history.

Original: Proof.
Revised: Proof.

Original: First, we prove a bound for E (cid:104) (cid:80) ki =
Revised: First, we prove a bound for E (cid:104) (cid:80) ki =

Original: [Equation]
Revised: [Equation]

Original: ˇ v i (cid:107) 2 (cid:105) .
Revised: ˇ v i (cid:107) 2 (cid:105) .

Original: By the fact that E [ y i − y i − 1 |F i − 1 ]
Revised: By the fact that E [ y i − y i − 1 |F i − 1 ]

Original: = 1 m (ˆ y
Revised: = 1 m (ˆ y

Original: i − y
Revised: i − y

Original: i − 1 ) , for each ˇ
Revised: i − 1 ) , for each ˇ

Original: v i , we have
Revised: v i , we have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation on all the randomness in the history, for E (cid:104)
Revised: Taking expectation on all the randomness in the history, for E (cid:104)

Original: [Equation]
Revised: [Equation]

Original: ki =
Revised: ki =

Original: [Equation]
Revised: [Equation]

Original: ˇ v i
Revised: ˇ v i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Then to prove (19), we deﬁne the sequence { ˇ y k } ∞ k =0 by ˇ y 

Revised: Then to prove (19), we deﬁne the sequence { ˇ y k } ∞ k =0 by ˇ y

Original: = y 0 and ˇ y k
Revised: = y 0 and ˇ y k

Original: = ˇ y k − 1 − ˇ v k fork = 1 , 2 , . . . .
Revised: = ˇ y k − 1 − ˇ v k fork = 1 , 2 , . . . .

Original: By expanding 12 (cid:107) ˇ y k − v (cid:107) 2 ,
Revised: By expanding 12 (cid:107) ˇ y k − v (cid:107) 2 ,

Original: [Equation]
Revised: [Equation]

Original: By summing this expression and telescoping
Revised: By summing this expression and telescoping

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the ﬁrst inequality is by Eq.
Revised: where the ﬁrst inequality is by Eq.

Original: (21) and the second inequality is by Eq.
Revised: (21) and the second inequality is by Eq.

Original: To obtain the result(19) , we use the facts that E [ˇ v i | F i − 1 ]
Revised: To obtain the result(19) , we use the facts that E [ˇ v i | F i − 1 ]

Original: = 0 and that ˇ y i − 1 ∈ F i − 1 to obtain E (cid:104) (cid:80) k
Revised: = 0 and that ˇ y i − 1 ∈ F i − 1 to obtain E (cid:104) (cid:80) k

Original: i =1 (cid:104)
Revised: i =1 (cid:104)

Original: ˇ y
Revised: ˇ y

Original: i − 1 , ˇ v i (cid:105) (cid:105)
Revised: i − 1 , ˇ v i (cid:105) (cid:105)

Original: =0 .
Revised: =0 .

Original: We emphasize that Lemma 2 holds for any v that may be even correlated with the randomness in thealgorithm.
Revised: We emphasize that Lemma 2 holds for any v that may be even correlated with the randomness in thealgorithm.

Original: Lemma 3.
Revised: Lemma 3.

Original: For all steps of Algorithm 3 with k ≥ 1 , we have for all ( u , v ) ∈
Revised: For all steps of Algorithm 3 with k ≥ 1 , we have for all ( u , v ) ∈

Original: X × R n that
Revised: X × R n that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the ﬁrst claim
Revised: For the ﬁrst claim

Original: [Equation]
Revised: [Equation]

Original: we have from the deﬁnition of φ k ( x k ) , using that φ k − 1 ( x k − 1 ) is ( γ +σA k − 1 ) -strongly convex and minimized at x k − 1 , that
Revised: we have from the deﬁnition of φ k ( x k ) , using that φ k − 1 ( x k − 1 ) is ( γ +σA k − 1 ) -strongly convex and minimized at x k − 1 , that

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Meanwhile, by the deﬁnition of { ¯ y k } (using y − 1 = y 0 for the case of k = 1 ), we have that
Revised: Meanwhile, by the deﬁnition of { ¯ y k } (using y − 1 = y 0 for the case of k = 1 ), we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The claimed lower bound on φ k ( x k ) follows when we combine (23) and (24).
Revised: The claimed lower bound on φ k ( x k ) follows when we combine (23) and (24).

Original: For the second claim, we have by the deﬁnition of ψ k that
Revised: For the second claim, we have by the deﬁnition of ψ k that

Original: [Equation]
Revised: [Equation]

Original: To obtain the claimed lower bound on ψ k , we proceed to bound the terms on the right-hand side in(25).
Revised: To obtain the claimed lower bound on ψ k , we proceed to bound the terms on the right-hand side in(25).

Original: First, since ψ k − 1 is (1 /γ )
Revised: First, since ψ k − 1 is (1 /γ )

Original: -strongly convex and minimized at y k − 1 , we have
Revised: -strongly convex and minimized at y k − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Second, by using the deﬁnition of A S jk and A ¯ S jk , and by using several times that y k − 1 and y kdiffer only in their S j k components, we have
Revised: Second, by using the deﬁnition of A S jk and A ¯ S jk , and by using several times that y k − 1 and y kdiffer only in their S j k components, we have

Original: [Equation]
Revised: [Equation]

Original: where in the last equality we used y S jk k
Revised: where in the last equality we used y S jk k

Original: [Equation]
Revised: [Equation]

Original: 1 =γma k ( Ax k − b ) , which both hold by deﬁnitions of ˆ y k and y k .
Revised: 1 =γma k ( Ax k − b ) , which both hold by deﬁnitions of ˆ y k and y k .

Original: Finally, combining (25)–(27), we have the bound on ψ k ( y k ) from the statement of the lemma.
Revised: Finally, combining (25)–(27), we have the bound on ψ k ( y k ) from the statement of the lemma.

Original: By combining the two lower bounds in Lemma 3, we obtain the following result.
Revised: By combining the two lower bounds in Lemma 3, we obtain the following result.

Original: Lemma 4.
Revised: Lemma 4.

Original: For any ( u , v ) ∈ X × R n , the sum of ψ k ( y k )
Revised: For any ( u , v ) ∈ X × R n , the sum of ψ k ( y k )

Original: + φ k ( x k ) can be bounded as follows: forall k ≥ 1 ,
Revised: + φ k ( x k ) can be bounded as follows: forall k ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: (28) where
Revised: (28) where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Before our proof, as A − 1 and A 0 are not used in Algorithm 3, without loss of generality, weset A − 1 =
Revised: Before our proof, as A − 1 and A 0 are not used in Algorithm 3, without loss of generality, weset A − 1 =

Original: A 0 = 0 .
Revised: A 0 = 0 .

Original: Fix any ( u , v ) ∈
Revised: Fix any ( u , v ) ∈

Original: X × R n .
Revised: X × R n .

Original: By combining the bounds on φ k ( x k ) and ψ k ( y k )from Lemma 3, we have ∀ k ≥ 1 that
Revised: By combining the bounds on φ k ( x k ) and ψ k ( y k )from Lemma 3, we have ∀ k ≥ 1 that

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: and Q k is deﬁned in Eq.
Revised: and Q k is deﬁned in Eq.

Original: To ﬁnd a lower bound on P k we start by bounding the magnitude of the inner product term in (31).
Revised: To ﬁnd a lower bound on P k we start by bounding the magnitude of the inner product term in (31).

Original: Recall that y k − 2 and y k − 1 differ only on the coordinate block S j k − 1 .
Revised: Recall that y k − 2 and y k − 1 differ only on the coordinate block S j k − 1 .

Original: We thus have:
Revised: We thus have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second inequality is by Young’s inequality, and the third inequality is by Assumption 3where (cid:107) A S jk − 1 ( x k − x k − 1 ) (cid:107) ≤
Revised: where the second inequality is by Young’s inequality, and the third inequality is by Assumption 3where (cid:107) A S jk − 1 ( x k − x k − 1 ) (cid:107) ≤

Original: ˆ L (cid:107) x k − x k − 1 (cid:107) .
Revised: ˆ L (cid:107) x k − x k − 1 (cid:107) .

Original: With our setting a k − 1 = √
Revised: With our setting a k − 1 = √

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: /γ
Revised: /γ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 2 m ˆ L , for all k ≥ 1 , we have
Revised: 2 m ˆ L , for all k ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: By substituting this equality into (32) and then combining with (31), we obtain
Revised: By substituting this equality into (32) and then combining with (31), we obtain

Original: [Equation]
Revised: [Equation]

Original: We complete the proof by combining Eqs.
Revised: We complete the proof by combining Eqs.

Original: (30), (31) and the bound Eq.
Revised: (30), (31) and the bound Eq.

Original: (33) for P k .
Revised: (33) for P k .

Original: By telescoping the inequality in Lemma 4, and using Lemmas 1 and 2, we obtain the next result.
Revised: By telescoping the inequality in Lemma 4, and using Lemmas 1 and 2, we obtain the next result.

Original: Lemma 5.
Revised: Lemma 5.

Original: For all ( u , v ) ∈
Revised: For all ( u , v ) ∈

Original: X × R n , we have
Revised: X × R n , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˇ v i is deﬁned in Lemma 2.
Revised: where ˇ v i is deﬁned in Lemma 2.

Original: Proof.
Revised: Proof.

Original: Telescoping the inequality in Lemma 4, we have
Revised: Telescoping the inequality in Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last equality is by the fact
Revised: where the last equality is by the fact

Original: that ψ 0 ( y 0 ) =
Revised: that ψ 0 ( y 0 ) =

Original: φ 0 ( x 0 ) = 0 , a 0 = 0 , and our convention thaty − 1 :=
Revised: φ 0 ( x 0 ) = 0 , a 0 = 0 , and our convention thaty − 1 :=

Original: y 0 .
Revised: y 0 .

Original: Then by the convexity of r ( · ) and the deﬁnition of { Q i } , we have
Revised: Then by the convexity of r ( · ) and the deﬁnition of { Q i } , we have

Original: [Equation]
Revised: [Equation]

Original: where ˜ x k = 

Revised: where ˜ x k =

Original: A k (cid:80) ki
Revised: A k (cid:80) ki

Original: =1 a i x i , ˜ y k
Revised: =1 a i x i , ˜ y k

Original: = 1 A k (cid:80) ki
Revised: = 1 A k (cid:80) ki

Original: =1 ( a i y i + ( m − 1) a
Revised: =1 ( a i y i + ( m − 1) a

Original: i
Revised: i

Original: ( y i − y
Revised: ( y i − y

Original: i − 1 ))
Revised: i − 1 ))

Original: (as deﬁned in (4))
Revised: (as deﬁned in (4))

Original: andthe last equality is by the deﬁnition of the Lagrangian L ( · , · ) and { ˇ v i } in Lemma 2.
Revised: andthe last equality is by the deﬁnition of the Lagrangian L ( · , · ) and { ˇ v i } in Lemma 2.

Original: Then by combining Eqs.
Revised: Then by combining Eqs.

Original: (35)-(36) and Lemma 1, we have
Revised: (35)-(36) and Lemma 1, we have

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have from the fact that y k and y k − 1 differ in only the S j k components, Young’s inequality
Revised: we have from the fact that y k and y k − 1 differ in only the S j k components, Young’s inequality

Original: [Equation]
Revised: [Equation]

Original: and the deﬁnition of a k that
Revised: and the deﬁnition of a k that

Original: [Equation]
Revised: [Equation]

Original: leading to
Revised: leading to

Original: [Equation]
Revised: [Equation]

Original: and completing the proof.
Revised: and completing the proof.

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose that ( x ∗ , y ∗ ) is a Nash point for (PD-GLP) .
Revised: Suppose that ( x ∗ , y ∗ ) is a Nash point for (PD-GLP) .

Original: Then the iterates x k , y k from
Revised: Then the iterates x k , y k from

Original: Algorithm 1 satisfy
Revised: Algorithm 1 satisfy

Original: [Equation]
Revised: [Equation]

Original: (37) where the expectation is w.r.t.
Revised: (37) where the expectation is w.r.t.

Original: all the randomness in the algorithm.
Revised: all the randomness in the algorithm.

Original: Proof.
Revised: Proof.

Original: Note that the existence of a Nash point is assumed in Assumption 2.
Revised: Note that the existence of a Nash point is assumed in Assumption 2.

Original: With ( u , v ) =
Revised: With ( u , v ) =

Original: ( x ∗ , y ∗ ) ,by the deﬁnition of Nash equilibrium, we have
Revised: ( x ∗ , y ∗ ) ,by the deﬁnition of Nash equilibrium, we have

Original: [Equation]
Revised: [Equation]

Original: By setting ( u , v ) =
Revised: By setting ( u , v ) =

Original: ( x ∗ , y ∗ ) in the result of Lemma 5, using (38) to eliminate the ﬁrst term on theleft-hand side of the inequality, and rearranging, we obtain
Revised: ( x ∗ , y ∗ ) in the result of Lemma 5, using (38) to eliminate the ﬁrst term on theleft-hand side of the inequality, and rearranging, we obtain

Original: [Equation]
Revised: [Equation]

Original: The result will follow when we show that the expectation (with respect to all the randomness) of thelast two terms on the right-hand side is zero.
Revised: The result will follow when we show that the expectation (with respect to all the randomness) of thelast two terms on the right-hand side is zero.

Original: Since y i − 1 ∈ F i − 1 , we have
Revised: Since y i − 1 ∈ F i − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: which takes care of the second-last term.
Revised: which takes care of the second-last term.

Original: For any ﬁxed v ∈ R n , we also have
Revised: For any ﬁxed v ∈ R n , we also have

Original: [Equation]
Revised: [Equation]

Original: which takes care of the last term, and completes the proof.
Revised: which takes care of the last term, and completes the proof.

Original: Next we state a technical lemma, proved in an earlier paper, which provides the ﬁnal ingredient forthe proof of Theorem 1.
Revised: Next we state a technical lemma, proved in an earlier paper, which provides the ﬁnal ingredient forthe proof of Theorem 1.

Original: Lemma 7.
Revised: Lemma 7.

Original: [[55]] Let { A k } k ≥ 0 be a sequence of nonnegative real numbers such that A 0 = 0 and
Revised: [[55]] Let { A k } k ≥ 0 be a sequence of nonnegative real numbers such that A 0 = 0 and

Original: A k is deﬁned recursively via A k =
Revised: A k is deﬁned recursively via A k =

Original: A k − 1 + (cid:112) c 2 1 + c 2 A k − 1 , where c 1 > 0 , and c 2 ≥ 0 .
Revised: A k − 1 + (cid:112) c 2 1 + c 2 A k − 1 , where c 1 > 0 , and c 2 ≥ 0 .

Original: Deﬁne
Revised: Deﬁne

Original: K 0 = (cid:100) c
Revised: K 0 = (cid:100) c

Original: 2 9 c 1 (cid:101) .
Revised: 2 9 c 1 (cid:101) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: if c 2 > 0 and k > K 0 , c
Revised: if c 2 > 0 and k > K 0 , c

Original: [Equation]
Revised: [Equation]

Original: k, otherwise .
Revised: k, otherwise .

Original: We are now ready to prove our main result, which we restate here.
Revised: We are now ready to prove our main result, which we restate here.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let x k , y k , k ∈ [ K ] , be the iterates of Algorithm 1 and let ˜ x k , ˜
Revised: Let x k , y k , k ∈ [ K ] , be the iterates of Algorithm 1 and let ˜ x k , ˜

Original: y k be deﬁned by
Revised: y k be deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for k ∈ [ K ] .
Revised: for k ∈ [ K ] .

Original: Let W k ⊂ X × R n , k ∈ [ K ] , be a sequence of compact convex sets such that(˜ x k ,
Revised: Let W k ⊂ X × R n , k ∈ [ K ] , be a sequence of compact convex sets such that(˜ x k ,

Original: ˜ y k ) ∈ W k
Revised: ˜ y k ) ∈ W k

Original: ⊂ W ⊂ X
Revised: ⊂ W ⊂ X

Original: × R n , where W is also convex and compact.
Revised: × R n , where W is also convex and compact.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: where (ˆ u , ˆ v ) =
Revised: where (ˆ u , ˆ v ) =

Original: arg sup ( u , v ) ∈W k {L (˜ x k , v ) − L ( u , ˜ y k ) } .
Revised: arg sup ( u , v ) ∈W k {L (˜ x k , v ) − L ( u , ˜ y k ) } .

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: Deﬁne K 

Revised: Deﬁne K

Original: = (cid:6)
Revised: = (cid:6)

Original: [Equation]
Revised: [Equation]

Original: 18ˆ Lmγ (cid:7) .
Revised: 18ˆ Lmγ (cid:7) .

Original: Then in the bounds above:
Revised: Then in the bounds above:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To provide a guarantee in terms of primal-dual gap, we need to take the supremum on u , vof {L (˜ x k , v ) − L ( u , ˜ y k )) } over W k ; we denote the arg sup by (ˆ u , ˆ v ) .
Revised: To provide a guarantee in terms of primal-dual gap, we need to take the supremum on u , vof {L (˜ x k , v ) − L ( u , ˜ y k )) } over W k ; we denote the arg sup by (ˆ u , ˆ v ) .

Original: When we subsequentlytake the expectation, we have to account for the fact that ˆ v will be correlated with the randomnessin the iteration history.
Revised: When we subsequentlytake the expectation, we have to account for the fact that ˆ v will be correlated with the randomnessin the iteration history.

Original: We can however, use Lemmas 2 and 6 to give the upper bound on E (cid:104) −(cid:80)
Revised: We can however, use Lemmas 2 and 6 to give the upper bound on E (cid:104) −(cid:80)

Original: [Equation]
Revised: [Equation]

Original: From (34), using the fact that 12 γ (cid:107) v − y k (cid:107) 2 + γ + σA k
Revised: From (34), using the fact that 12 γ (cid:107) v − y k (cid:107) 2 + γ + σA k

Original: [Equation]
Revised: [Equation]

Original: u − x k
Revised: u − x k

Original: [Equation]
Revised: [Equation]

Original: 2 ≥ 0 , we have
Revised: 2 ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by Eq.
Revised: where the ﬁrst inequality is by Eq.

Original: (34), the second inequality is by Lemma 2 and (39), and the lastinequality is by Lemma 6.
Revised: (34), the second inequality is by Lemma 2 and (39), and the lastinequality is by Lemma 6.

Original: This proves the ﬁrst claim (5).
Revised: This proves the ﬁrst claim (5).

Original: The second claim (6) follows from Lemma6 with the fact that 1 4 γ E (cid:104) (cid:80) ki =1 (cid:107)
Revised: The second claim (6) follows from Lemma6 with the fact that 1 4 γ E (cid:104) (cid:80) ki =1 (cid:107)

Original: y i − y
Revised: y i − y

Original: i − 1 (cid:107) 2 (cid:105) ≥ 0 .
Revised: i − 1 (cid:107) 2 (cid:105) ≥ 0 .

Original: The ﬁnal claim concerning A k follows from
Revised: The ﬁnal claim concerning A k follows from

Original: Lemma 7 when we set
Revised: Lemma 7 when we set

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: In Algorithm 1, for all k ≥ 1 , ˜ x k satisﬁes
Revised: In Algorithm 1, for all k ≥ 1 , ˜ x k satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where v = 2 (cid:107) y ∗ (cid:107) (cid:107)
Revised: where v = 2 (cid:107) y ∗ (cid:107) (cid:107)

Original: A ˜ x k − b (cid:107) (
Revised: A ˜ x k − b (cid:107) (

Original: A ˜ x k − b ) .
Revised: A ˜ x k − b ) .

Original: Proof.
Revised: Proof.

Original: Assume that (cid:107)
Revised: Assume that (cid:107)

Original: A ˜ x k − b (cid:107) (cid:54) = 0 , as otherwise the ﬁrst bound follows trivially.
Revised: A ˜ x k − b (cid:107) (cid:54) = 0 , as otherwise the ﬁrst bound follows trivially.

Original: Let u = x ∗ andv =
Revised: Let u = x ∗ andv =

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For any ﬁxed u
Revised: For any ﬁxed u

Original: [Equation]
Revised: [Equation]

Original: and any v ∈ R n possibly depending on the randomness in the algorithm, we havefrom Lemma 5, taking expectation over all the randomness, that
Revised: and any v ∈ R n possibly depending on the randomness in the algorithm, we havefrom Lemma 5, taking expectation over all the randomness, that

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, we have
Revised: Meanwhile, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by Lemma 2 and the second inequality is by Lemma 6.
Revised: where the ﬁrst inequality is by Lemma 2 and the second inequality is by Lemma 6.

Original: Since y i − 1 ∈
Revised: Since y i − 1 ∈

Original: F i − 1 , we have as in (39) that
Revised: F i − 1 , we have as in (39) that

Original: [Equation]
Revised: [Equation]

Original: By combining Eq.
Revised: By combining Eq.

Original: (42)-(44) with u = x ∗ , we have
Revised: (42)-(44) with u = x ∗ , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the KKT condition of (PD-GLP) and the optimality of ( x ∗ , y ∗ ) , we have for all x ∈ X that
Revised: By the KKT condition of (PD-GLP) and the optimality of ( x ∗ , y ∗ ) , we have for all x ∈ X that

Original: [Equation]
Revised: [Equation]

Original: 46) and thus
Revised: 46) and thus

Original: [Equation]
Revised: [Equation]

Original: By combining Eq.
Revised: By combining Eq.

Original: (45) and Eq.
Revised: (45) and Eq.

Original: (47), we have
Revised: (47), we have

Original: [Equation]
Revised: [Equation]

Original: proving our ﬁrst claim.
Revised: proving our ﬁrst claim.

Original: The second claim is obtained by combining Eqs.
Revised: The second claim is obtained by combining Eqs.

Original: (45), (47), and (48).
Revised: (45), (47), and (48).

Original: B.2 Omitted Proof from Section 3.

Revised: B.2 Omitted Proof from Section 3.

Original: We need to compute explicitly only those components of q k − 1 and x k that are needed to update y k .
Revised: We need to compute explicitly only those components of q k − 1 and x k that are needed to update y k .

Original: Lemma 8.
Revised: Lemma 8.

Original: For { q k } deﬁned in Algorithm 1, we have
Revised: For { q k } deﬁned in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: where r 0 = 0 and for all k = 1 , 2 , . . .
Revised: where r 0 = 0 and for all k = 1 , 2 , . . .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is by induction.
Revised: The proof is by induction.

Original: For k = 0 , we have q 

Revised: For k = 0 , we have q

Original: =
Revised: =

Original: A 1 ( c + z 0 ) which is true by deﬁnition.
Revised: A 1 ( c + z 0 ) which is true by deﬁnition.

Original: Assuming that the result holds for some index k , we show that it continues to hold for k + 1 .
Revised: Assuming that the result holds for some index k , we show that it continues to hold for k + 1 .

Original: Wehave from Step 9 of Algorithm 1, then using the inductive assumption, that
Revised: Wehave from Step 9 of Algorithm 1, then using the inductive assumption, that

Original: [Equation]
Revised: [Equation]

Original: as required.
Revised: as required.

Original: This lemma indicates that we can reconstruct q k (or any subvector of q k that we need, on demand),provided we maintain z k and r k .
Revised: This lemma indicates that we can reconstruct q k (or any subvector of q k that we need, on demand),provided we maintain z k and r k .

Original: Note that the update from z k to z k +1 is sparse; these two vectorsdiffer only in the components corresponding to the block C j k .
Revised: Note that the update from z k to z k +1 is sparse; these two vectorsdiffer only in the components corresponding to the block C j k .

Original: To obtain r k +1 from r k , we need toadd a scalar times the sparse vector
Revised: To obtain r k +1 from r k , we need toadd a scalar times the sparse vector

Original: z k +1 − z k , so this update is also efﬁcient.
Revised: z k +1 − z k , so this update is also efﬁcient.

Original: We can also maintain an implicit representation of the averaged vector ˜ y
Revised: We can also maintain an implicit representation of the averaged vector ˜ y

Original: k efﬁciently, as shown in thefollowing lemma.
Revised: k efﬁciently, as shown in thefollowing lemma.

Original: We defer the proofs of Lemmas 8 and 9 to Appendix B.
Revised: We defer the proofs of Lemmas 8 and 9 to Appendix B.

Original: Lemma 9.
Revised: Lemma 9.

Original: For { ˜ y k } deﬁned in (4) , we have
Revised: For { ˜ y k } deﬁned in (4) , we have

Original: [Equation]
Revised: [Equation]

Original: where s 

Revised: where s

Original: = 0 and for all k
Revised: = 0 and for all k

Original: = 1 , 2 , . . . ,
Revised: = 1 , 2 , . . . ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that ˜ y K
Revised: Recall that ˜ y K

Original: ( K ≥ 1) is deﬁned in Step 11 of Algorithm 1.
Revised: ( K ≥ 1) is deﬁned in Step 11 of Algorithm 1.

Original: The proof is by induction.
Revised: The proof is by induction.

Original: Fork = 1 , ˜ y 1 = y
Revised: Fork = 1 , ˜ y 1 = y

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: m − 1) a 1 ( y 1 − y 0 ) =
Revised: m − 1) a 1 ( y 1 − y 0 ) =

Original: y 1 + 1 A 1 s 1 which is true by the deﬁnition of s 1 and
Revised: y 1 + 1 A 1 s 1 which is true by the deﬁnition of s 1 and

Original: A 0 .
Revised: A 0 .

Original: Next, we assume that the result holds for some k ≥ 2 and show that it continues to hold fork + 1 .
Revised: Next, we assume that the result holds for some k ≥ 2 and show that it continues to hold fork + 1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: s k +1 .
Revised: s k +1 .

Original: as required.
Revised: as required.

Original: B.3 Omitted Proof from Section 3. 
Revised: B.3 Omitted Proof from Section 3.

Original: The standard-form LP (PD-LP) is derived by setting r ( x ) ≡ 0 and X = { x ∈ R
Revised: The standard-form LP (PD-LP) is derived by setting r ( x ) ≡ 0 and X = { x ∈ R

Original: d : x i ≥ 0 , i ∈
Revised: d : x i ≥ 0 , i ∈

Original: [ d ] }in (PD-GLP).
Revised: [ d ] }in (PD-GLP).

Original: Given any w ∈ X × R n , we deﬁne a compact convex subset W ζ,γ ( w ) of X × R n asfollows:
Revised: Given any w ∈ X × R n , we deﬁne a compact convex subset W ζ,γ ( w ) of X × R n asfollows:

Original: [Equation]
Revised: [Equation]

Original: where we have deﬁned (cid:107) · (cid:107) ( γ ) by (cid:107) w (cid:107) ( γ )
Revised: where we have deﬁned (cid:107) · (cid:107) ( γ ) by (cid:107) w (cid:107) ( γ )

Original: = (cid:113) γ (cid:107) x (cid:107) 2 + 1 γ (cid:107) y (cid:107) 2 in Section 3.3.
Revised: = (cid:113) γ (cid:107) x (cid:107) 2 + 1 γ (cid:107) y (cid:107) 2 in Section 3.3.

Original: Lemma 10.
Revised: Lemma 10.

Original: Consider the standard-form LP (LP) .
Revised: Consider the standard-form LP (LP) .

Original: Given τ > 0 and w ∈ X × R n , if (cid:107) w (cid:107) ( γ ) ≤ τand ζ ≤ τ , then
Revised: Given τ > 0 and w ∈ X × R n , if (cid:107) w (cid:107) ( γ ) ≤ τand ζ ≤ τ , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let F ( w ) =
Revised: Let F ( w ) =

Original: (cid:20)
Revised: (cid:20)

Original: A T y + c b − Ax (cid:21) .
Revised: A T y + c b − Ax (cid:21) .

Original: Then we haveρ ζ,γ ( w
Revised: Then we haveρ ζ,γ ( w

Original: [Equation]
Revised: [Equation]

Original: :=
Revised: :=

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from w ∈ W ζ,γ ( w ) .
Revised: where the inequality follows from w ∈ W ζ,γ ( w ) .

Original: First, we prove
Revised: First, we prove

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) max {− A T y − c , 0 }(cid:107)> 0 , for ˆ w= (ˆ x 1 , ˆ y 1 ) , let ˆ x= x +ζ γ (cid:107) max {− A T y − c , 0 }(cid:107) 2 max {− A T y − c , 0 } ∈ X and ˆ y 1 =
Revised: If (cid:107) max {− A T y − c , 0 }(cid:107)> 0 , for ˆ w= (ˆ x 1 , ˆ y 1 ) , let ˆ x= x +ζ γ (cid:107) max {− A T y − c , 0 }(cid:107) 2 max {− A T y − c , 0 } ∈ X and ˆ y 1 =

Original: y .
Revised: y .

Original: Then we have (cid:107) ˆ w 1 − w (cid:107) ( γ ) =
Revised: Then we have (cid:107) ˆ w 1 − w (cid:107) ( γ ) =

Original: ζ andthus ˆ w 1 ∈ W ζ,γ ( w ) .
Revised: ζ andthus ˆ w 1 ∈ W ζ,γ ( w ) .

Original: So, as ρ ζ,γ ( w ) ≥ F ( w ) T ( w − ˆ w 1 ) , with the deﬁnition of ˆ w 1 , Eq.
Revised: So, as ρ ζ,γ ( w ) ≥ F ( w ) T ( w − ˆ w 1 ) , with the deﬁnition of ˆ w 1 , Eq.

Original: (52) holds.
Revised: (52) holds.

Original: If (cid:107) max {− A T y − c , 0 }(cid:107) 2 = 0 , then Eq.
Revised: If (cid:107) max {− A T y − c , 0 }(cid:107) 2 = 0 , then Eq.

Original: (52) holds trivially.
Revised: (52) holds trivially.

Original: Second, we prove
Revised: Second, we prove

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If (cid:107) Ax − b (cid:107) 2 > 0 , for ˆ w 2 = (ˆ x 2 , ˆ y 2 ) , let ˆ x 2 = x
Revised: If (cid:107) Ax − b (cid:107) 2 > 0 , for ˆ w 2 = (ˆ x 2 , ˆ y 2 ) , let ˆ x 2 = x

Original: and ˆ y 2 =
Revised: and ˆ y 2 =

Original: y + ζγ (cid:107)
Revised: y + ζγ (cid:107)

Original: Ax − b (cid:107) ( Ax − b ) .
Revised: Ax − b (cid:107) ( Ax − b ) .

Original: Then we have(cid:107)
Revised: Then we have(cid:107)

Original: ˆ w 2 − w (cid:107) ( γ ) =
Revised: ˆ w 2 − w (cid:107) ( γ ) =

Original: ζ and thus ˆ w 2 ∈ W ζ,γ ( w ) .
Revised: ζ and thus ˆ w 2 ∈ W ζ,γ ( w ) .

Original: Then as ρ ζ,γ ( w ) ≥ F ( w ) T ( w − ˆ w 2 ) , Eq.
Revised: Then as ρ ζ,γ ( w ) ≥ F ( w ) T ( w − ˆ w 2 ) , Eq.

Original: (53) holds.
Revised: (53) holds.

Original: If (cid:107) Ax − b (cid:107) 2 = 0 , then Eq.
Revised: If (cid:107) Ax − b (cid:107) 2 = 0 , then Eq.

Original: (53) holds trivially.
Revised: (53) holds trivially.

Original: Third, we prove that
Revised: Third, we prove that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the inequality holds trivially if c T x + b T y ≤ 0 , so we assume that c T x + b T y > 0 ,and note that F ( w ) T w =
Revised: Note that the inequality holds trivially if c T x + b T y ≤ 0 , so we assume that c T x + b T y > 0 ,and note that F ( w ) T w =

Original: c T x + b T y > 0 in this case.
Revised: c T x + b T y > 0 in this case.

Original: This condition implies that w (cid:54) = 0 , sowe can deﬁne ˆ w 3 = w − min (cid:8)
Revised: This condition implies that w (cid:54) = 0 , sowe can deﬁne ˆ w 3 = w − min (cid:8)

Original: [Equation]
Revised: [Equation]

Original: w (cid:107) ( γ ) , 1 (cid:9) w .
Revised: w (cid:107) ( γ ) , 1 (cid:9) w .

Original: Then we have (cid:107) ˆ w 3 − w (cid:107) ( γ ) ≤
Revised: Then we have (cid:107) ˆ w 3 − w (cid:107) ( γ ) ≤

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w (cid:107) ( γ ) ≤ ζ.
Revised: (cid:107) w (cid:107) ( γ ) ≤ ζ.

Original: Meanwhile, we also have ˆ x 3 ≥ x 3 − x 3 = 0 , so that ˆ w 3 ∈ X × R n .
Revised: Meanwhile, we also have ˆ x 3 ≥ x 3 − x 3 = 0 , so that ˆ w 3 ∈ X × R n .

Original: Thus, we have ˆ w 3 ∈ W ζ,γ ( w ) .
Revised: Thus, we have ˆ w 3 ∈ W ζ,γ ( w ) .

Original: Then, with the assumptions ζ ≤ τ and (cid:107) w (cid:107) ( γ ) ≤ τ, together with F ( w ) T w > 0 , we haveρ ζ
Revised: Then, with the assumptions ζ ≤ τ and (cid:107) w (cid:107) ( γ ) ≤ τ, together with F ( w ) T w > 0 , we haveρ ζ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: completing the proof of the claim (54).
Revised: completing the proof of the claim (54).

Original: By combining Eqs.
Revised: By combining Eqs.

Original: (52), (53) and (54), we obtain
Revised: (52), (53) and (54), we obtain

Original: [Equation]
Revised: [Equation]

Original: 56) from which the result follows.
Revised: 56) from which the result follows.

Original: Lemma 11.
Revised: Lemma 11.

Original: Consider the standard-form LP (LP) , and let w ∗ =
Revised: Consider the standard-form LP (LP) , and let w ∗ =

Original: ( x ∗ , y ∗ ) be a Nash point (that is, asolution of the primal-dual form (PD-LP) ).
Revised: ( x ∗ , y ∗ ) be a Nash point (that is, asolution of the primal-dual form (PD-LP) ).

Original: For a starting point w 0 , deﬁne ζ = (cid:107) w 0 − w ∗ (cid:107) ( γ ) andchoose γ > 0 .
Revised: For a starting point w 0 , deﬁne ζ = (cid:107) w 0 − w ∗ (cid:107) ( γ ) andchoose γ > 0 .

Original: Then for all k = 1 , 2 , . .
Revised: Then for all k = 1 , 2 , . .

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken w.r.t.
Revised: where the expectation is taken w.r.t.

Original: all the randomness up to iteration k .
Revised: all the randomness up to iteration k .

Original: Further, for W ζ,γ ( ˜ w k )deﬁned as in (49) , we have
Revised: Further, for W ζ,γ ( ˜ w k )deﬁned as in (49) , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: By Theorem 1, we have
Revised: By Theorem 1, we have

Original: [Equation]
Revised: [Equation]

Original: by the deﬁnition of (cid:107) · (cid:107) ( γ ) .
Revised: by the deﬁnition of (cid:107) · (cid:107) ( γ ) .

Original: Using this deﬁnition again, we have that the left-hand side in thisexpression is bounded below by 14 (cid:107) w k − w ∗ (cid:107) 2( γ ) , so that
Revised: Using this deﬁnition again, we have that the left-hand side in thisexpression is bounded below by 14 (cid:107) w k − w ∗ (cid:107) 2( γ ) , so that

Original: [Equation]
Revised: [Equation]

Original: By convexity of (cid:107) · (cid:107) 2 , we obtain
Revised: By convexity of (cid:107) · (cid:107) 2 , we obtain

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst claim of the lemma now follows by applying Jensen’s inequality to this bound.
Revised: The ﬁrst claim of the lemma now follows by applying Jensen’s inequality to this bound.

Original: For ˆ w = ( ˆ x , ˆ y ) ∈ W ζ,γ ( ˜ w k ) , we have the following bound on E (cid:104) γ (cid:107) ˆ x −
Revised: For ˆ w = ( ˆ x , ˆ y ) ∈ W ζ,γ ( ˜ w k ) , we have the following bound on E (cid:104) γ (cid:107) ˆ x −

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 60) where the ﬁrst inequality is by the triangle inequality, the second inequality is by the arithmeticinequality, the third inequality is by Eq.
Revised: 60) where the ﬁrst inequality is by the triangle inequality, the second inequality is by the arithmeticinequality, the third inequality is by Eq.

Original: (59) and our assumption ˆ w ∈ W ζ,γ ( ˜ w k ) with ζ = (cid:107) w 0 −w ∗ (cid:107) ( γ ) .
Revised: (59) and our assumption ˆ w ∈ W ζ,γ ( ˜ w k ) with ζ = (cid:107) w 0 −w ∗ (cid:107) ( γ ) .

Original: For the case of linear programming, the strong convexity constant is σ = 0 , so that A k =k 2ˆ Lm in CLVR .
Revised: For the case of linear programming, the strong convexity constant is σ = 0 , so that A k =k 2ˆ Lm in CLVR .

Original: Thus, by applying Theorem 1 with W k = W ζ,γ ( ˜ w k ) and ˆ w = (ˆ x , ˆ y ) =
Revised: Thus, by applying Theorem 1 with W k = W ζ,γ ( ˜ w k ) and ˆ w = (ˆ x , ˆ y ) =

Original: arg sup w =( x , y ) ∈W ζ,γ ( ˜ w k ) {
Revised: arg sup w =( x , y ) ∈W ζ,γ ( ˜ w k ) {

Original: L (˜ x k , y ) − L ( x , ˜ y k ) } , and using the deﬁnition of (cid:107) · (cid:107) ( γ ) and Eq.
Revised: L (˜ x k , y ) − L ( x , ˜ y k ) } , and using the deﬁnition of (cid:107) · (cid:107) ( γ ) and Eq.

Original: (60),we have
Revised: (60),we have

Original: [Equation]
Revised: [Equation]

Original: Then with Theorem 1, Lemmas 10 and 11, we give our theorem for the ﬁxed restart strategy.
Revised: Then with Theorem 1, Lemmas 10 and 11, we give our theorem for the ﬁxed restart strategy.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the CLVR algorithm applied to the standard-form LP problem (PD-LP) , withinput w 0 and output ˜ w k .
Revised: Consider the CLVR algorithm applied to the standard-form LP problem (PD-LP) , withinput w 0 and output ˜ w k .

Original: Given γ > 0 , deﬁne w
Revised: Given γ > 0 , deﬁne w

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: , and deﬁne
Revised: , and deﬁne

Original: C 

Revised: C

Original: = γ + 1 /γ
Revised: = γ + 1 /γ

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: √ 2 + 1) (cid:107) w 0 − w ∗ (cid:107) ( γ ) + (cid:107) w ∗ (cid:107) ( γ ) .
Revised: √ 2 + 1) (cid:107) w 0 − w ∗ (cid:107) ( γ ) + (cid:107) w ∗ (cid:107) ( γ ) .

Original: Then for H γ deﬁned as in (9) , we have
Revised: Then for H γ deﬁned as in (9) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Applying Lemma 10 with w = ˜ w k , τ = (cid:107) ˜ w k (cid:107) ( γ ) + (cid:107) w 0 − w ∗ (cid:107) ( γ ) and ζ = (cid:107) w 0 − w ∗ (cid:107) ( γ ) ≤τ , we have
Revised: Applying Lemma 10 with w = ˜ w k , τ = (cid:107) ˜ w k (cid:107) ( γ ) + (cid:107) w 0 − w ∗ (cid:107) ( γ ) and ζ = (cid:107) w 0 − w ∗ (cid:107) ( γ ) ≤τ , we have

Original: [Equation]
Revised: [Equation]

Original: As x 2 y is jointly convex in x, y on the domain { ( x, y ) : x ∈ R , y > 0 } [11, Example 3.18] using
Revised: As x 2 y is jointly convex in x, y on the domain { ( x, y ) : x ∈ R , y > 0 } [11, Example 3.18] using

Original: Jensen’s inequality, we have that E [ x 2 y ] ≥ ( E [ x ]) 2 E
Revised: Jensen’s inequality, we have that E [ x 2 y ] ≥ ( E [ x ]) 2 E

Original: [ y ] .
Revised: [ y ] .

Original: (In our case, this simply follows as x, y depend onthe same source of randomness.)
Revised: (In our case, this simply follows as x, y depend onthe same source of randomness.)

Original: Applying this inequality to (62) with x = (cid:112) LPMetric ( ˜ w k ) andy
Revised: Applying this inequality to (62) with x = (cid:112) LPMetric ( ˜ w k ) andy

Original: = γ + 1 /γ
Revised: = γ + 1 /γ

Original: + (cid:107) ˜
Revised: + (cid:107) ˜

Original: w k
Revised: w k

Original: [Equation]
Revised: [Equation]

Original: γ )
Revised: γ )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: − w
Revised: − w

Original: [Equation]
Revised: [Equation]

Original: ( γ ) , we obtain
Revised: ( γ ) , we obtain

Original: [Equation]
Revised: [Equation]

Original: sup
Revised: sup

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using Lemma
Revised: Using Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By combining Eqs.
Revised: By combining Eqs.

Original: (63) and (64) and using the deﬁnition of C 0 , we have
Revised: (63) and (64) and using the deﬁnition of C 0 , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by the deﬁnition of Hoffman constant in Eq.
Revised: where the last inequality is by the deﬁnition of Hoffman constant in Eq.

Original: Meanwhile, by Lemma11, we have:
Revised: Meanwhile, by Lemma11, we have:

Original: [Equation]
Revised: [Equation]

Original: Now, recalling that w ∗ =
Revised: Now, recalling that w ∗ =

Original: arg min w ∈W ∗
Revised: arg min w ∈W ∗

Original: [Equation]
Revised: [Equation]

Original: − w (cid:107) ( γ ) and using Eq.
Revised: − w (cid:107) ( γ ) and using Eq.

Original: (9), we have
Revised: (9), we have

Original: [Equation]
Revised: [Equation]

Original: By combining Eqs.
Revised: By combining Eqs.

Original: (65)-(67), we have
Revised: (65)-(67), we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the sharpness property, (cid:107) ˜
Revised: By the sharpness property, (cid:107) ˜

Original: w k − w ∗ (cid:107) ( γ ) ≤ LPMetric( ˜ w k ) H γ .
Revised: w k − w ∗ (cid:107) ( γ ) ≤ LPMetric( ˜ w k ) H γ .

Original: Since one call to CLVR betweenrestarts halves the LPMetric, we have LPMetric( ˜ w k ) ≤ 12 LPMetric( w 0 ) .
Revised: Since one call to CLVR betweenrestarts halves the LPMetric, we have LPMetric( ˜ w k ) ≤ 12 LPMetric( w 0 ) .

Original: Combining these twoinequalities leads to the claimed bound.
Revised: Combining these twoinequalities leads to the claimed bound.

Original: Finally, we summarize the adaptive restart strategy in Algorithm 4.
Revised: Finally, we summarize the adaptive restart strategy in Algorithm 4.

Original: Algorithm
Revised: Algorithm

Original: 1: Input: (cid:15) > 0 , x 0 ∈ X , y 0 ∈ R n , γ > 0 , ˆ L > 0 , K , (cid:98) K , m , { S 1 , S 2 , . .
Revised: 1: Input: (cid:15) > 0 , x 0 ∈ X , y 0 ∈ R n , γ > 0 , ˆ L > 0 , K , (cid:98) K , m , { S 1 , S 2 , . .

Original: , S m } , { C 1 , C 2 , . .
Revised: , S m } , { C 1 , C 2 , . .

Original: , C m } 2: t = 0 , x (0)0 = x 0 , y
Revised: , C m } 2: t = 0 , x (0)0 = x 0 , y

Original: (0)0 =
Revised: (0)0 =

Original: y 0 , w (0)
Revised: y 0 , w (0)

Original: =
Revised: =

Original: ( x (0)0 , y (0)0 ) 3: repeat 4:
Revised: ( x (0)0 , y (0)0 ) 3: repeat 4:

Original: Run Lazy CLVR (Algorithm 2) until LPMetric( w ( t +1) ) ≤ 12 LPMetric( w ( t ) ) where, w ( t +1)
Revised: Run Lazy CLVR (Algorithm 2) until LPMetric( w ( t +1) ) ≤ 12 LPMetric( w ( t ) ) where, w ( t +1)

Original: =
Revised: =

Original: ˜ w ( t ) K
Revised: ˜ w ( t ) K

Original: =
Revised: =

Original: (˜ x ( t ) K , ˜ y
Revised: (˜ x ( t ) K , ˜ y

Original: ( t ) K ) and ˜ x ( t ) K , ˜ y
Revised: ( t ) K ) and ˜ x ( t ) K , ˜ y

Original: ( t ) K are the output of Lazy CLVR 5: t =
Revised: ( t ) K are the output of Lazy CLVR 5: t =

Original: t +6: until LPMetric( w ( t ) ) ≤ (cid:15) 7: Return: w ( t )
Revised: t +6: until LPMetric( w ( t ) ) ≤ (cid:15) 7: Return: w ( t )

Original: C Omitted
Revised: C Omitted

Original: Proofs from Section 

Revised: Proofs from Section

Original: C.1 DRO with f -divergence-based Ambiguity Set
Revised: C.1 DRO with f -divergence-based Ambiguity Set

Original: Theorem 3.
Revised: Theorem 3.

Original: Let X be a compact convex set.
Revised: Let X be a compact convex set.

Original: Then the DRO problem in Eq.
Revised: Then the DRO problem in Eq.

Original: (10) is equivalent tomin x , u , v , w , µ , q ,γ (cid:110) γ + ρµn + n n (cid:88)
Revised: (10) is equivalent tomin x , u , v , w , µ , q ,γ (cid:110) γ + ρµn + n n (cid:88)

Original: i = µ i f ∗ (cid:16) q i µ i (cid:17)(cid:111) s .
Revised: i = µ i f ∗ (cid:16) q i µ i (cid:17)(cid:111) s .

Original: t .
Revised: t .

Original: w + v − q n − γ 1 n = 0 n , u
Revised: w + v − q n − γ 1 n = 0 n , u

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since the context is clear, to simplify the notation, in the following we use P to denote P ρ,n .
Revised: Since the context is clear, to simplify the notation, in the following we use P to denote P ρ,n .

Original: First, using Sion’s minimax theorem, we have that
Revised: First, using Sion’s minimax theorem, we have that

Original: [Equation]
Revised: [Equation]

Original: Introducing auxiliary variables w and u , the problem is further equivalent to
Revised: Introducing auxiliary variables w and u , the problem is further equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the last equivalence is by applying minimax equality, which holds due to compactness of
Revised: where the last equivalence is by applying minimax equality, which holds due to compactness of

Original: P [56].
Revised: P [56].

Original: Hence, we can conclude that
Revised: Hence, we can conclude that

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed tuple ( x , w , u ) , using Lagrange multipliers to enforce the constraints from P , we canfurther write
Revised: For a ﬁxed tuple ( x , w , u ) , using Lagrange multipliers to enforce the constraints from P , we canfurther write

Original: [Equation]
Revised: [Equation]

Original: Now, using the deﬁnitions of D f and the convex conjugate of f , we have
Revised: Now, using the deﬁnitions of D f and the convex conjugate of f , we have

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality is by Eq.
Revised: where the ﬁrst equality is by Eq.

Original: (70), the third one is by the variable substitution
Revised: (70), the third one is by the variable substitution

Original: q i
Revised: q i

Original: =
Revised: =

Original: nµν i , andthe last one is by introducing µ 1 , µ 2 , . . . , µ n to replace µ .
Revised: nµν i , andthe last one is by introducing µ 1 , µ 2 , . . . , µ n to replace µ .

Original: Then each nµ i f ∗ (cid:0) q i nµ i (cid:1) ( i ∈ [ n ]) is aperspective function of f ∗ [11], which is jointly convex w.r.t.
Revised: Then each nµ i f ∗ (cid:0) q i nµ i (cid:1) ( i ∈ [ n ]) is aperspective function of f ∗ [11], which is jointly convex w.r.t.

Original: ( µ i , q i ) .
Revised: ( µ i , q i ) .

Original: Hence, we can conclude that
Revised: Hence, we can conclude that

Original: [Equation]
Revised: [Equation]

Original: where the last line is by strong duality.
Revised: where the last line is by strong duality.

Original: Thus, combining with Eq.
Revised: Thus, combining with Eq.

Original: (69), we conclude that the original
Revised: (69), we conclude that the original

Original: DRO problem with f -divergence based ambiguity set is equivalent to the following problem:
Revised: DRO problem with f -divergence based ambiguity set is equivalent to the following problem:

Original: [Equation]
Revised: [Equation]

Original: Finally, noticing that the maximization problem over p ∈ R n enforces the equality constraintw + v − q n − γ 1 n = 0 n , we obtain
Revised: Finally, noticing that the maximization problem over p ∈ R n enforces the equality constraintw + v − q n − γ 1 n = 0 n , we obtain

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: Example: Conditional Value at Risk (CVaR) with Hinge Loss.
Revised: Example: Conditional Value at Risk (CVaR) with Hinge Loss.

Original: As a speciﬁc example of anapplication of Theorem 3, we consider CVaR at level α ∈ (0 , 1) , which leads to the optimizationproblem:
Revised: As a speciﬁc example of anapplication of Theorem 3, we consider CVaR at level α ∈ (0 , 1) , which leads to the optimizationproblem:

Original: [Equation]
Revised: [Equation]

Original: where g ( u i ) = max { 0 , 1 − u i } is the hinge loss.
Revised: where g ( u i ) = max { 0 , 1 − u i } is the hinge loss.

Original: Here the ambiguity set constraint reduces to simplebounds p i ≤ 1 αn for i ∈ [ n ] , so the reformulation based on the convex perspective function can beavoided altogether and replaced by simple Lagrange multipliers for this linear constraint.
Revised: Here the ambiguity set constraint reduces to simplebounds p i ≤ 1 αn for i ∈ [ n ] , so the reformulation based on the convex perspective function can beavoided altogether and replaced by simple Lagrange multipliers for this linear constraint.

Original: In particular,in the proof of Theorem 3, we can write
Revised: In particular,in the proof of Theorem 3, we can write

Original: [Equation]
Revised: [Equation]

Original: Following the argument from the proof of Theorem 3, and expressing w i ≥ g ( u i ) equivalently as thepair of constraints w i ≥ 0 and w i ≥ 1 − u i , the problem reduces to
Revised: Following the argument from the proof of Theorem 3, and expressing w i ≥ g ( u i ) equivalently as thepair of constraints w i ≥ 0 and w i ≥ 1 − u i , the problem reduces to

Original: [Equation]
Revised: [Equation]

Original: which is a linear program.
Revised: which is a linear program.

Original: To write it in the standard form, we further introduce slack variabless ∈ R n , s ≥ 0 , to replace inequality constraints w i ≥
Revised: To write it in the standard form, we further introduce slack variabless ∈ R n , s ≥ 0 , to replace inequality constraints w i ≥

Original: 1 − u i by s i − u
Revised: 1 − u i by s i − u

Original: i − w i = − 1 .
Revised: i − w i = − 1 .

Original: Forimplementation purposes, we deﬁne X to be the set of simple non-negativity constraints ( w i
Revised: Forimplementation purposes, we deﬁne X to be the set of simple non-negativity constraints ( w i

Original: ≥0 , s i ≥ 0 , , v i ≥ 0 , µ i , ≥ 0 , ∀ i ∈ [ n ] ) from Eq.
Revised: ≥0 , s i ≥ 0 , , v i ≥ 0 , µ i , ≥ 0 , ∀ i ∈ [ n ] ) from Eq.

Original: The problem then becomes
Revised: The problem then becomes

Original: [Equation]
Revised: [Equation]

Original: , µ
Revised: , µ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , i ∈ [ n ] , s − u − w = − 1 n , w ≥ 0 n , v ≥ 0 n , µ ≥ 0 n , s ≥ 0 n .
Revised: , i ∈ [ n ] , s − u − w = − 1 n , w ≥ 0 n , v ≥ 0 n , µ ≥ 0 n , s ≥ 0 n .

Original: C.2 DRO with
Revised: C.2 DRO with

Original: Wasserstein Metric-based Ambiguity Set
Revised: Wasserstein Metric-based Ambiguity Set

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let µ and ν be two probability distributions supported on Θ = R d × { 1 , − 1 } andlet Π( µ , ν ) denote the set of all joint distributions between µ and ν .
Revised: Let µ and ν be two probability distributions supported on Θ = R d × { 1 , − 1 } andlet Π( µ , ν ) denote the set of all joint distributions between µ and ν .

Original: Then the Wasserstein metricbetween µ and ν is deﬁned by
Revised: Then the Wasserstein metricbetween µ and ν is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where ξ , ξ (cid:48) ∈ Θ and ζ ( · , · ) : Θ × Θ → R + is a convex cost function deﬁned by
Revised: where ξ , ξ (cid:48) ∈ Θ and ζ ( · , · ) : Θ × Θ → R + is a convex cost function deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) denotes a general norm and κ > 0 is used to balance the feature mismatch and labeluncertainty.
Revised: where (cid:107) · (cid:107) denotes a general norm and κ > 0 is used to balance the feature mismatch and labeluncertainty.

Original: Let Q = 1 n (cid:80) ni =1 δ ( a i ,b i ) , where δ ( a i ,b i ) is the Dirac Delta function (or a point mass) atpoint ( a i , b i ) .
Revised: Let Q = 1 n (cid:80) ni =1 δ ( a i ,b i ) , where δ ( a i ,b i ) is the Dirac Delta function (or a point mass) atpoint ( a i , b i ) .

Original: Then, the Wasserstein metric based ambiguity set is deﬁned as
Revised: Then, the Wasserstein metric based ambiguity set is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Assumption 5.
Revised: Assumption 5.

Original: R (cid:51) M
Revised: R (cid:51) M

Original: = sup
Revised: = sup

Original: [Equation]
Revised: [Equation]

Original: | θ |
Revised: | θ |

Original: [Equation]
Revised: [Equation]

Original: To obtain our main result in Theorem 12, we ﬁrst show the following auxiliary lemma that issubsequently used in the proof of Theorem 4.
Revised: To obtain our main result in Theorem 12, we ﬁrst show the following auxiliary lemma that issubsequently used in the proof of Theorem 4.

Original: A similar result for the special case of a logistic lossfunction can be found in [52].
Revised: A similar result for the special case of a logistic lossfunction can be found in [52].

Original: Lemma 12.
Revised: Lemma 12.

Original: Let ( a (cid:48) , b (cid:48) )
Revised: Let ( a (cid:48) , b (cid:48) )

Original: ∈
Revised: ∈

Original: R d × { 1 , − 1 } be a given pair of data sample and label.
Revised: R d × { 1 , − 1 } be a given pair of data sample and label.

Original: Then, for everyλ > 0 , we have
Revised: Then, for everyλ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since g is assumed to be proper, convex
Revised: Since g is assumed to be proper, convex

Original: [Equation]
Revised: [Equation]

Original: and lower semicontinuous
Revised: and lower semicontinuous

Original: [Equation]
Revised: [Equation]

Original: by the Fenchel-Moreautheorem
Revised: by the Fenchel-Moreautheorem

Original: [Equation]
Revised: [Equation]

Original: it is equal to its biconjugate.
Revised: it is equal to its biconjugate.

Original: Applying this property
Revised: Applying this property

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Applying the change of variable v := a − a (cid:48) , we further have
Revised: Applying the change of variable v := a − a (cid:48) , we further have

Original: [Equation]
Revised: [Equation]

Original: = (cid:26)
Revised: = (cid:26)

Original: [Equation]
Revised: [Equation]

Original: w ) , if sup θ ∈ dom( g ∗ ) (cid:107) θb (cid:48) w (cid:107) ∗ ≤ λ + ∞ , otherwise , where the second equality is by the convex conjugate of a norm (cid:107) · (cid:107) being equal to the convexindicator of the unit ball w.r.t.
Revised: w ) , if sup θ ∈ dom( g ∗ ) (cid:107) θb (cid:48) w (cid:107) ∗ ≤ λ + ∞ , otherwise , where the second equality is by the convex conjugate of a norm (cid:107) · (cid:107) being equal to the convexindicator of the unit ball w.r.t.

Original: the dual norm (cid:107) · (cid:107) ∗ .
Revised: the dual norm (cid:107) · (cid:107) ∗ .

Original: Finally, it remains to use that, by assumption,sup θ ∈ dom( g ∗ ) | θ | = M.
Revised: Finally, it remains to use that, by assumption,sup θ ∈ dom( g ∗ ) | θ | = M.

Original: Then following [52, Theorem 1], we provide reformulations of the problem from Eq.
Revised: Then following [52, Theorem 1], we provide reformulations of the problem from Eq.

Original: (11) that can beaddressed by computationally efﬁcient solvers.
Revised: (11) that can beaddressed by computationally efﬁcient solvers.

Original: Theorem 4.
Revised: Theorem 4.

Original: The optimization problem from Eq.
Revised: The optimization problem from Eq.

Original: (11) is equivalent to:
Revised: (11) is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let z
Revised: Let z

Original: =
Revised: =

Original: ( a , b ) ∈ Θ := R d × { 1 , − 1 } and let h w ( z ) :
Revised: ( a , b ) ∈ Θ := R d × { 1 , − 1 } and let h w ( z ) :

Original: = g ( b a T w ) .
Revised: = g ( b a T w ) .

Original: Then by the deﬁnition ofthe Wasserstein metric,
Revised: Then by the deﬁnition ofthe Wasserstein metric,

Original: [Equation]
Revised: [Equation]

Original: Assume that the conditional distribution of z given z (cid:48) =
Revised: Assume that the conditional distribution of z given z (cid:48) =

Original: ( a i , b i ) is P i , for all i ∈ [ n ] .
Revised: ( a i , b i ) is P i , for all i ∈ [ n ] .

Original: Then, based onthe deﬁnition of P n = 1 n (cid:80) ni =1 δ
Revised: Then, based onthe deﬁnition of P n = 1 n (cid:80) ni =1 δ

Original: ( a i ,b i ) , we have
Revised: ( a i ,b i ) , we have

Original: [Equation]
Revised: [Equation]

Original: As a result, the problem from Eq.
Revised: As a result, the problem from Eq.

Original: (77) is equivalent to
Revised: (77) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Then substituting in z = ( a , b ) , using that the domain of y is { 1 , − 1 } , and decomposing P i intounnormalized measures P ± 1 (d a ) =
Revised: Then substituting in z = ( a , b ) , using that the domain of y is { 1 , − 1 } , and decomposing P i intounnormalized measures P ± 1 (d a ) =

Original: P i (d a , { b = ± 1 } ) supported on R d , the RHS of Eq.
Revised: P i (d a , { b = ± 1 } ) supported on R d , the RHS of Eq.

Original: (79) canbe simpliﬁed to
Revised: (79) canbe simpliﬁed to

Original: [Equation]
Revised: [Equation]

Original: With the deﬁnition of the cost function ζ (( a , b ) , ( a (cid:48) , b (cid:48) ))
Revised: With the deﬁnition of the cost function ζ (( a , b ) , ( a (cid:48) , b (cid:48) ))

Original: = (cid:107) a − a (cid:48) (cid:107) + κ | b − b (cid:48) | , it follows that
Revised: = (cid:107) a − a (cid:48) (cid:107) + κ | b − b (cid:48) | , it follows that

Original: [Equation]
Revised: [Equation]

Original: d a
Revised: d a

Original: [Equation]
Revised: [Equation]

Original: =n
Revised: =n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: The above problem w.r.t.
Revised: The above problem w.r.t.

Original: P i ± 1 is an inﬁnite-dimensional linear program with a ﬁnite number ofconstraints.
Revised: P i ± 1 is an inﬁnite-dimensional linear program with a ﬁnite number ofconstraints.

Original: By [53, Proposition 3.4], we get the following equivalent dual formulation:
Revised: By [53, Proposition 3.4], we get the following equivalent dual formulation:

Original: [Equation]
Revised: [Equation]

Original: Then, recalling that h w ( a , ± 1) is short for g ( ± a T w ) , by Lemma 12, we havesup a ∈ Rd
Revised: Then, recalling that h w ( a , ± 1) is short for g ( ± a T w ) , by Lemma 12, we havesup a ∈ R

Original: [Equation]
Revised: [Equation]

Original: w ( a ,± [Equation] ) , if sup
Revised: a ,

Original: [Equation]
Revised: [Equation]

Original: w
Revised: if

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: , otherwise.
Revised: otherwise.

Original: Finally, the resulting reformulation is
Revised: Finally, the resulting reformulation is

Original: [Equation]
Revised: [Equation]

Original: Finally, recalling that, by assumption, sup θ ∈ dom( g ∗ ) | θ | =
Revised: Finally, recalling that, by assumption, sup θ ∈ dom( g ∗ ) | θ | =

Original: M, it follows that the constraintsup θ ∈ dom( g ∗ ) (cid:107) θ w (cid:107) ∗ ≤ λ is equivalent to (cid:107) w (cid:107) ∗ ≤ λ/M.
Revised: M, it follows that the constraintsup θ ∈ dom( g ∗ ) (cid:107) θ w (cid:107) ∗ ≤ λ is equivalent to (cid:107) w (cid:107) ∗ ≤ λ/M.

Original: Meanwhile, by introducing u i
Revised: Meanwhile, by introducing u i

Original: =
Revised: =

Original: b i a T i w , v i = − u
Revised: b i a T i w , v i = − u

Original: i ( i ∈ [ n ]) and s i , t i ( i ∈ [ n ] , we obtain Theorem 4.
Revised: i ( i ∈ [ n ]) and s i , t i ( i ∈ [ n ] , we obtain Theorem 4.

Original: In Theorem 4, when we assume that the conic constraints g ( u ) ≤ s and (cid:107) w (cid:107) ∗ ≤ λ/M in Eq.
Revised: In Theorem 4, when we assume that the conic constraints g ( u ) ≤ s and (cid:107) w (cid:107) ∗ ≤ λ/M in Eq.

Original: (76)admit efﬁcient proximal operators, we can formulate this problem as (PD-GLP) and apply CLVR .
Revised: (76)admit efﬁcient proximal operators, we can formulate this problem as (PD-GLP) and apply CLVR .

Original: Theresulting complexity bounds are similar to those discussed above for the f -divergence formulation.
Revised: Theresulting complexity bounds are similar to those discussed above for the f -divergence formulation.

Original: D Experiment Details
Revised: D Experiment Details

Original: D.1 Comparisons between  L and R
Revised: D.1 Comparison between the values of L and R

Original: As described in Section 1, one of the main advantages of CLVR over previous methods is that thecomplexity of CLVR depends on the max row norm R instead of the spectral norm L , which in theworst case for ill-conditioned problems can have a factor of √ n improvement.
Revised: As described in Section 1, one of the main advantages of CLVR over previous methods is that thecomplexity of CLVR depends on the max row norm R instead of the spectral norm L , which in theworst case for ill-conditioned problems can have a factor of √ n improvement.

Original: We claimed thatin practical problems where the problem instances are highly structure (e.g., reformulated DROproblems), R can be much smaller than L and in Table 3 we provide empirical evidence for suchclaims.
Revised: We claimed thatin practical problems where the problem instances are highly structured (e.g., reformulated DROproblems), R can be much smaller than L and in Table 3 we provide empirical evidence for suchclaims.

Original: Inour all  experiments, we normalize each rows of A to R = 1 as stated in Assumption 1 sothe values of L demonstrate the theoretical improvements.
Revised: In all our experiments, we normalize each rows of A to R = 1 as stated in Assumption 1 sothe values of L demonstrate the theoretical improvements for the experiments described in Section 5.

Original: [Table]
Revised: [Table]

Original: D.2 Experiment Details
Revised: D.2 Comparison of adaptive restart schemes

Revised: [Table] We now provide a brief empirical comparison between our adaptive restart scheme that uses LP metricand the adaptive restart scheme using the normalized duality gap proposed in [9]. We compared theperformance of PDHG on benchmark problem sets qap10 , qap15 , nug

Original: [Table]
Revised: [Table]

Original: For the DRO with Wasserstein metric of (cid:96) 1 -norm, since the dual norm of the (cid:96) 1 norm is the (cid:96) ∞ norm,the reformulation for this problem in Theorem 4 is a standard-form LP.
Revised: and nug20 used in [9],using the two adaptive restart criteria. We ran PDHG until reaching accuracy as described in [9] (i [Table] e.,until normalized duality gap is at most 1e-6 and primal & dual infeasibility is at most 1e-8)

Revised: [Table]

Revised: [Table]

Revised: As can be seen from Table

Revised: [Table]

Revised: the two restart criteria give similar performance in terms of iterationcomplexity: there is no clear winner, as normalized gap is better on qap10 and nug08 , while LP

Revised: Metric is better on qap15 and nug20 .

Revised: On the other hand, as Figure 2 shows, the two adaptive restartschemes lead to similar performance of PDHG over iterations.

Revised: A bottom line is that our restartcriterion does not seem to hurt the performance in terms of iteration complexity.

Revised: Further empiricalevaluation will be carried out to compare the performance of two restart criteria in terms of wall clocktime.

Revised: [Table]

Revised: D.3 Experiment Details

Revised: For the DRO with Wasserstein metric of (cid:96) 1 -norm, since the dual norm of the (cid:96) 1 norm is the (cid:96) ∞ norm,the reformulation for this problem in Theorem 4 is a standard-form LP.

Original: For the LP formulation, wecompare our CLVR method with three representative methods: PDHG [14], SPDHG [15] and PURECD [5].
Revised: For the LP formulation, wecompare our CLVR method with three representative methods: PDHG [14], SPDHG [15] and PURECD [5].

Original: For all algorithms we use LPMetric (7) as the performance measure and use a restart strategyof successive halving of LPMetric (Section 3.3) to obtain linear convergence.
Revised: For all algorithms we use LPMetric (7) as the performance measure and use a restart strategyof successive halving of LPMetric (Section 3.3) to obtain linear convergence.

Original: We implemented
Revised: We implemented

Original: CLVR and other algorithms in Julia, optimizing all implementations to the best of our ability.
Revised: CLVR and other algorithms in Julia, optimizing all implementations to the best of our ability.

Original: For
Revised: For

Original: SPDHG , whose per-iteration cost is at least O ( d ) , we consider a large batch size 50 to balance theeffect of the O ( d ) cost and improve the overall efﬁciency.
Revised: SPDHG , whose per-iteration cost is at least O ( d ) , we consider a large batch size 50 to balance theeffect of the O ( d ) cost and improve the overall efﬁciency.

Original: Meanwhile, PURE - CD with block size 1 isalready well suited to sparsity.
Revised: Meanwhile, PURE - CD with block size 1 isalready well suited to sparsity.

Original: For CLVR , we experiment with block sizes 1 and 10 .
Revised: For CLVR , we experiment with block sizes 1 and 10 .

Original: We conductedour experiments on LibSVM [16] datasets a9a , gisette , and rcv1.binary , each with differentsparsity levels.
Revised: We conductedour experiments on LibSVM [16] datasets a9a , gisette , and rcv1.binary , each with differentsparsity levels.

Original: For fair comparison in terms of wall-clock time, we implement all the algorithms inthe Julia programming language 5 and run each algorithm using one CPU core, all the experiments arerun on a Linux machine with a second generation Intel Xeon Scalable Processor (Cascade Lake-SP)with 128 GB of RAM.
Revised: For fair comparison in terms of wall-clock time, we implement all the algorithms inthe Julia programming language 5 and run each algorithm using one CPU core, all the experiments arerun on a Linux machine with a second generation Intel Xeon Scalable Processor (Cascade Lake-SP)with 128 GB of RAM.

Original: As the weight parameter γ (see Theorem 1) between primal and dual variablesstrongly inﬂuences the empirical performance, we tune it for all datasets by trying the values { 10 − i }for i ∈ Z , for each of the methods.
Revised: As the weight parameter γ (see Theorem 1) between primal and dual variablesstrongly inﬂuences the empirical performance, we tune it for all datasets by trying the values { 10 − i }for i ∈ Z , for each of the methods.

Original: We set the Lipschitz constant of PDHG to be the largest singularvalue of the constraint matrix in the LP formulation.
Revised: We set the Lipschitz constant of PDHG to be the largest singularvalue of the constraint matrix in the LP formulation.

Original: For PURE - CD and CLVR with block size 1 ,because the rows of the matrix are normalized, we set the Lipschitz constant to 1 .
Revised: For PURE - CD and CLVR with block size 1 ,because the rows of the matrix are normalized, we set the Lipschitz constant to 1 .

Original: For CLVR withblock size 10 and SPDHG with block size 50 , the Lipschitz constants are tuned to 3 and 9 , respectively.
Revised: For CLVR withblock size 10 and SPDHG with block size 50 , the Lipschitz constants are tuned to 3 and 9 , respectively.

Original: Remark 3 (Comparisons of using different block sizes) .
Revised: Remark 3 (Comparisons of using different block sizes) .

Original: We conducted experiments to compare thepractical performance of CLVR against different choices of block sizes and the results are shown in
Revised: We conducted experiments to compare thepractical performance of CLVR against different choices of block sizes and the results are shown in

Original: Table 2.
Revised: Table 3.

Original: We ran the DRO with Wasserstein metric using the same setup as described in Section 5, onthe rcv1 dataset and using an early stopping criterion of LP Metric at 10 − 1 .
Revised: We ran the DRO with Wasserstein metric using the same setup as described in Section 5, onthe rcv1 dataset and using an early stopping criterion of LP Metric at 10 − 1 .

Original: In the plot, we cansee that CLVR converges to an approximate solution fastest when the block size is set to 10 .
Revised: In the plot, we cansee that CLVR converges to an approximate solution fastest when the block size is set to 10 .

Original: Therefore,this plot provides support for our choice of block size of 10 in Section 5.
Revised: Therefore,this plot provides support for our choice of block size of 10 in Section 5.

Original: As illustrated in Figure 1,
Revised: As illustrated in Figure 1,

Original: CLVR is most efﬁcient in terms of the number of data passes when the block size is 1.
Revised: CLVR is most efﬁcient in terms of the number of data passes when the block size is 1.

Original: However,we observe that when evaluated in terms of the execution time, running CLVR with larger blocksize yields better performance even though it exhibits worse performance in terms of the numberof data passes.
Revised: However,we observe that when evaluated in terms of the execution time, running CLVR with larger blocksize yields better performance even though it exhibits worse performance in terms of the numberof data passes.

Original: We attribute this phenomenon to the instructional-level parallelism [28] in modernprocessors, allowing more computations to be completed in the same number of clock cycles.
Revised: We attribute this phenomenon to the instructional-level parallelism [28] in modernprocessors, allowing more computations to be completed in the same number of clock cycles.

Original: In Table 4, we list information about the three datasets and the corresponding matrices in thereformulations.
Revised: In Table 5, we list information about the three datasets and the corresponding matrices in thereformulations.

Original: As we see, due to the sparse connectivity of auxiliary variables, all the matrices inreformulations are quite sparse.
Revised: As we see, due to the sparse connectivity of auxiliary variables, all the matrices inreformulations are quite sparse.

Original: As a commonly adopted preprocessing step for LP, we normalizeeach row of the matrix in the standard-form LP with Euclidean norm one.[Table]
Revised: As a commonly adopted preprocessing step for LP, we normalizeeach row of the matrix in the standard-form LP with Euclidean norm one.

Original: Remark 4 (Performance comparison using multiple cores) .
Revised: Remark 4 (Performance comparison using multiple cores) .

Original: We conducted further experiments toexamine the effects of allowing the algorithms to run on more computing cores.
Revised: We conducted further experiments toexamine the effects of allowing the algorithms to run on more computing cores.

Original: However, we did notobserve any meaningful difference in performances in terms of wall-clock time when we repeated theexperiments described above and in Section 5 using 2 CPU cores per algorithm.
Revised: However, we did not [Table] observe any meaningful difference in performances in terms of wall-clock time when we repeated theexperiments described above and in Section 5 using 2 CPU cores per algorithm.

Original: Our interpretation ofthis observation is that because most of the steps within CLVR and other algorithms we are comparingagainst are simple and cheap, involving very few large matrix-vector multiplications and no matrixfactorization, the practical performance of algorithms becomes memory-bound, hence additionalcomputational resources do not beneﬁt them much if at all.
Revised: Our interpretation ofthis observation is that because most of the steps within CLVR and other algorithms we are comparingagainst are simple and cheap, involving very few large matrix-vector multiplications and no matrixfactorization, the practical performance of algorithms becomes memory-bound, hence additionalcomputational resources do not beneﬁt them much if at all.