Original: S TOCHASTIC P ROJECTIVE S PLITTING : S OLVING S ADDLE -P OINT P ROBLEMS WITH M ULTIPLE R EGULARIZERS
Revised: S TOCHASTIC P ROJECTIVE S PLITTING : S OLVING S ADDLE -P OINT P ROBLEMS WITH M ULTIPLE R EGULARIZERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a new, stochastic variant of the projective splitting (PS) family of algorithms for monotone inclusion problems.
Revised: We present a new, stochastic variant of the projective splitting (PS) family of algorithms for monotone inclusion problems.

Original: It can solve min-max and noncooperative game formulations arising in applications such as robust ML without the convergence issues associated with gradient descent-ascent, the current de facto standard approach in ML applications.
Revised: It can solve min-max and noncooperative game formulations arising in applications such as robust ML without the convergence issues associated with gradient descent-ascent, the current de facto standard approach in ML applications.

Original: Our proposal is the ﬁrst version of PS able to use stochastic gradient oracles.
Revised: Our proposal is the ﬁrst version of PS able to use stochastic gradient oracles.

Original: It can solve min-max games while handling multiple constraints and nonsmooth regularizers via projection and proximal operators.
Revised: It can solve min-max games while handling multiple constraints and nonsmooth regularizers via projection and proximal operators.

Original: Unlike other stochastic splitting methods that can solve such problems, our method does not rely on a product-space reformulation of the original problem.
Revised: Unlike other stochastic splitting methods that can solve such problems, our method does not rely on a product-space reformulation of the original problem.

Original: We prove almost-sure convergence of the iterates to the solution and a convergence rate for the expected residual.
Revised: We prove almost-sure convergence of the iterates to the solution and a convergence rate for the expected residual.

Original: By working with monotone inclusions rather than variational inequalities, our analysis avoids the drawbacks of measuring convergence through the restricted gap function.
Revised: By working with monotone inclusions rather than variational inequalities, our analysis avoids the drawbacks of measuring convergence through the restricted gap function.

Original: We close with numerical experiments on a distributionally robust sparse logistic regression problem.
Revised: We close with numerical experiments on a distributionally robust sparse logistic regression problem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The most prominent application of optimization in ML is empirical risk minimization.
Revised: The most prominent application of optimization in ML is empirical risk minimization.

Original: However, inspired by the success of GANs (Goodfellow et al., 2014).
Revised: However, inspired by the success of GANs (Goodfellow et al., 2014).

Original: , ML practitioners have developed more complicated min-max and adversarial optimization formulations (Yu et al., 2021; Kuhn et al., 2019;
Revised: , ML practitioners have developed more complicated min-max and adversarial optimization formulations (Yu et al., 2021; Kuhn et al., 2019;

Original: Shaﬁeezadeh-Abadeh et al., 2015; Sinha et al., 2018;
Revised: Shaﬁeezadeh-Abadeh et al., 2015; Sinha et al., 2018;

Original: Lin et al., 2020; Namkoong & Duchi, 2016;
Revised: Lin et al., 2020; Namkoong & Duchi, 2016;

Original: Huang et al., 2017; Wadsworth et al., 2018; Zhang et al., 2018; Edwards & Storkey, 2015; Celis & Keswani, 2019).
Revised: Huang et al., 2017; Wadsworth et al., 2018; Zhang et al., 2018; Edwards & Storkey, 2015; Celis & Keswani, 2019).

Original: Solving these multi-player games leads to issues not seen when minimizing a single-player loss function.
Revised: Solving these multi-player games leads to issues not seen when minimizing a single-player loss function.

Original: The competitive nature of a game leads to rotational dynamics that can cause intuitive gradient-based methods to fail to converge (Gidel et al., 2019;
Revised: The competitive nature of a game leads to rotational dynamics that can cause intuitive gradient-based methods to fail to converge (Gidel et al., 2019;

Original: Daskalakis et al., 2018;
Revised: Daskalakis et al., 2018;

Original: Hsieh et al., 2020).
Revised: Hsieh et al., 2020).

Original: A mathematical framework underlying both convex optimization and saddle-point problems is the monotone inclusion problem ; see Ryu & Boyd (2016) for an introduction.
Revised: A mathematical framework underlying both convex optimization and saddle-point problems is the monotone inclusion problem ; see Ryu & Boyd (2016) for an introduction.

Original: Methods developed for monotone inclusions will converge for convex-concave, games as they are explicitly designed to handle such problems’ governing dynamics.
Revised: Methods developed for monotone inclusions will converge for convex-concave, games as they are explicitly designed to handle such problems’ governing dynamics.

Original: In recent years, monotone inclusion methods and theory have started to receive attention in the ML community (Diakonikolas, 2020; Liu et al., 2021;
Revised: In recent years, monotone inclusion methods and theory have started to receive attention in the ML community (Diakonikolas, 2020; Liu et al., 2021;

Original: Ryu et al., 2020; Pathak & Wainwright, 2020), with a focus on monotone variational inequalities, which form a special case of monotone inclusions (Antonakopoulos et al., 2019; Gidel et al., 2019;
Revised: Ryu et al., 2020; Pathak & Wainwright, 2020), with a focus on monotone variational inequalities, which form a special case of monotone inclusions (Antonakopoulos et al., 2019; Gidel et al., 2019;

Original: Daskalakis et al., 2018; Hsieh et al., 2020; Mertikopoulos et al., 2019).
Revised: Daskalakis et al., 2018; Hsieh et al., 2020; Mertikopoulos et al., 2019).

Original: The most prevalent methods for solving min-max games in ML are variants of gradient descent-ascent (GDA).
Revised: The most prevalent methods for solving min-max games in ML are variants of gradient descent-ascent (GDA).

Original: This method alternates between a gradient-descent step for the minimizing player and a gradient-ascent step for the maximizing player.
Revised: This method alternates between a gradient-descent step for the minimizing player and a gradient-ascent step for the maximizing player.

Original: Unfortunately, GDA requires additional assumptions to converge on convex-concave games, and it even fails for some simple 2D bilinear games (Gidel et al., 2019, Prop. 1).
Revised: Unfortunately, GDA requires additional assumptions to converge on convex-concave games, and it even fails for some simple 2D bilinear games (Gidel et al., 2019, Prop. 1).

Original: While there have been several approaches to modify either GDA (Chavdarova et al., 2021; Grnarova et al., 2021; Balduzzi et al., 2018) or the underlying game objective (Mescheder et al., 2018; Nagarajan & Kolter, 2017; Mescheder et al., 2017) to ensure convergence, this paper instead develops a method for solving monotone inclusions that can naturally handle game dynamics.
Revised: While there have been several approaches to modify either GDA (Chavdarova et al., 2021; Grnarova et al., 2021; Balduzzi et al., 2018) or the underlying game objective (Mescheder et al., 2018; Nagarajan & Kolter, 2017; Mescheder et al., 2017) to ensure convergence, this paper instead develops a method for solving monotone inclusions that can naturally handle game dynamics.

Original: Our approach builds upon the recently proposed projective splitting (PS) method with forward steps (Johnstone & Eckstein, 2020b).
Revised: Our approach builds upon the recently proposed projective splitting (PS) method with forward steps (Johnstone & Eckstein, 2020b).

Original: PS is designed speciﬁcally for solving monotone inclusions, thus does not fall prey to the convergence issues that plague GDA, at least for convex-concave games.
Revised: PS is designed speciﬁcally for solving monotone inclusions, thus does not fall prey to the convergence issues that plague GDA, at least for convex-concave games.

Original: PS is within the general class of projective splitting methods invented by Eckstein & Svaiter (2008) and developed further in Eckstein & Svaiter (2009);
Revised: PS is within the general class of projective splitting methods invented by Eckstein & Svaiter (2008) and developed further in Eckstein & Svaiter (2009);

Original: Alotaibi et al.
Revised: Alotaibi et al.

Original: ; Combettes & Eckstein (2018); Eckstein (2017); Johnstone & Eckstein (2019; 2021; 2020a).
Revised: ; Combettes & Eckstein (2018); Eckstein (2017); Johnstone & Eckstein (2019; 2021; 2020a).

Original: These methods work by creating a separating hyperplane between the current iterate and the solution and then moving closer to the solution by projecting the current iterate onto this hyperplane (see Section 3 for an overview).
Revised: These methods work by creating a separating hyperplane between the current iterate and the solution and then moving closer to the solution by projecting the current iterate onto this hyperplane (see Section 3 for an overview).

Original: Other than being able to natively handle game dynamics, the primary advantage of PS is that it fully splits problems involving an arbitrary number of regularizers and constraints.
Revised: Other than being able to natively handle game dynamics, the primary advantage of PS is that it fully splits problems involving an arbitrary number of regularizers and constraints.

Original: “Full splitting” means that the method can handle multiple regularizers and constraints through their respective individual proximal and projection operators, along with the smooth terms via gradients.
Revised: “Full splitting” means that the method can handle multiple regularizers and constraints through their respective individual proximal and projection operators, along with the smooth terms via gradients.

Original: What makes this useful is that many of the regularizers used in ML have proximal operators that are relatively easy to compute; see for example Parikh & Boyd (2013).
Revised: What makes this useful is that many of the regularizers used in ML have proximal operators that are relatively easy to compute; see for example Parikh & Boyd (2013).

Original: Despite these advantages, the preexisting PS framework has a signiﬁcant drawback: it requires deterministic gradient oracles.
Revised: Despite these advantages, the preexisting PS framework has a signiﬁcant drawback: it requires deterministic gradient oracles.

Original: This feature makes it impractical for application to large datasets for which stochastic oracles may be the only feasible option.
Revised: This feature makes it impractical for application to large datasets for which stochastic oracles may be the only feasible option.

Original: Contributions
Revised: Contributions

Original: The primary contribution of this work is a new projective splitting algorithm that allows for a stochastic gradient oracle.
Revised: The primary contribution of this work is a new projective splitting algorithm that allows for a stochastic gradient oracle.

Original: We call the method stochastic projective splitting (SPS).
Revised: We call the method stochastic projective splitting (SPS).

Original: Our method “fully splits” the monotone inclusion problem
Revised: Our method “fully splits” the monotone inclusion problem

Original: [Equation]
Revised: [Equation]

Original: (1) where B is monotone and L -Lipschitz and each A i is maximal monotone and typically set valued, usually arising from a constraint or a nonsmooth regularizer in the underlying optimization problem or game; see for example Ryu & Boyd (2016) for deﬁnitions.
Revised: (1) where B is monotone and L -Lipschitz and each A i is maximal monotone and typically set valued, usually arising from a constraint or a nonsmooth regularizer in the underlying optimization problem or game; see for example Ryu & Boyd (2016) for deﬁnitions.

Original: For some example ML applications of (1), see Section 2 and Appendix A. Here, an algorithm that “fully splits” (1) means one whose computational steps each involve only the individual operators
Revised: For some example ML applications of (1), see Section 2 and Appendix A. Here, an algorithm that “fully splits” (1) means one whose computational steps each involve only the individual operators

Original: A 1 , . . . , A n , B .
Revised: A 1 , . . . , A n , B .

Original: Ours is the ﬁrst method that can accomplish full splitting without a product-space reformulation that recasts (1) as a two-operator problem on a higher-dimensional space, a tactic whose disadvantages are discussed in
Revised: Ours is the ﬁrst method that can accomplish full splitting without a product-space reformulation that recasts (1) as a two-operator problem on a higher-dimensional space, a tactic whose disadvantages are discussed in

Original: Appendix F.7.
Revised: Appendix F.7.

Original: Our method interrogates the Lipschitz operator B through a stochastic oracle.
Revised: Our method interrogates the Lipschitz operator B through a stochastic oracle.

Original: Previous methods splitting (1) have either required a deterministic oracle for B , or have made far more restrictive assumptions on the noise or the operators (Briceño-Arias & Combettes, 2011; Combettes &
Revised: Previous methods splitting (1) have either required a deterministic oracle for B , or have made far more restrictive assumptions on the noise or the operators (Briceño-Arias & Combettes, 2011; Combettes &

Original: Pesquet, 2012; Malitsky & Tam, 2020; Bot et al., 2019; Van Dung & Vu, 2021) than we will require below.
Revised: Pesquet, 2012; Malitsky & Tam, 2020; Bot et al., 2019; Van Dung & Vu, 2021) than we will require below.

Original: However, the stochastic methods of Alacaoglu et al.
Revised: However, the stochastic methods of Alacaoglu et al.

Original: and Böhm et al.
Revised: and Böhm et al.

Original: (2020), when combined with a product-space reformulation, can solve (1) when all the A i are subdifferentials of convex functions; see Section 6.
Revised: (2020), when combined with a product-space reformulation, can solve (1) when all the A i are subdifferentials of convex functions; see Section 6.

Original: When moving away from a deterministic gradient oracle in projective splitting, a key difﬁculty is that the generated hyperplanes do not guarantee separation between the solution and the current point.
Revised: When moving away from a deterministic gradient oracle in projective splitting, a key difﬁculty is that the generated hyperplanes do not guarantee separation between the solution and the current point.

Original: We solve this issue by relaxing the projection: we only update each iterate in the direction of the noisy projection and scale its movement by a decreasing stepsize that allows for control of the stochastic error.
Revised: We solve this issue by relaxing the projection: we only update each iterate in the direction of the noisy projection and scale its movement by a decreasing stepsize that allows for control of the stochastic error.

Original: Using the framework of stochastic quasi-Fejér monotonicity (Combettes & Pesquet, 2015), we prove almost-sure convergence of the ﬁnal iterate and do not require averaging of the iterates (Theorem 1, Section 5).
Revised: Using the framework of stochastic quasi-Fejér monotonicity (Combettes & Pesquet, 2015), we prove almost-sure convergence of the ﬁnal iterate and do not require averaging of the iterates (Theorem 1, Section 5).

Original: We also provide a non-asymptotic convergence rate for the approximation residual (Theorem 2, Section 5).
Revised: We also provide a non-asymptotic convergence rate for the approximation residual (Theorem 2, Section 5).

Original: A special case of SPS is the recently-developed Double Stepsize Extragradient Method (DSEG) (Hsieh et al., 2020).
Revised: A special case of SPS is the recently-developed Double Stepsize Extragradient Method (DSEG) (Hsieh et al., 2020).

Original: When n = 0 and therefore only B is present in (1), DSEG and SPS coincide.
Revised: When n = 0 and therefore only B is present in (1), DSEG and SPS coincide.

Original: Thus, our method extends DSEG to allow for regularizers and constraints.
Revised: Thus, our method extends DSEG to allow for regularizers and constraints.

Original: Our analysis also provides a new interpretation for DSEG as a special case of projective splitting.
Revised: Our analysis also provides a new interpretation for DSEG as a special case of projective splitting.

Original: Our nonasymptotic convergence rate for SPS also applies to DSEG under no additional assumptions.
Revised: Our nonasymptotic convergence rate for SPS also applies to DSEG under no additional assumptions.

Original: By contrast, the original convergence rate analysis for DSEG requires either strong monotonicity or an error bound.
Revised: By contrast, the original convergence rate analysis for DSEG requires either strong monotonicity or an error bound.

Original: We close with numerical experiments on a distributionally robust sparse logistic regression problem.
Revised: We close with numerical experiments on a distributionally robust sparse logistic regression problem.

Original: This is a nonsmooth convex-concave min-max problem which can be converted to (1) with n =set-valued operators.
Revised: This is a nonsmooth convex-concave min-max problem which can be converted to (1) with n =set-valued operators.

Original: On this problems class, SPS compares well to the possible alternative splitting methods.
Revised: On this problems class, SPS compares well to the possible alternative splitting methods.

Original: Non-monotone problems
Revised: Non-monotone problems

Original: The work of Hsieh et al.
Revised: The work of Hsieh et al.

Original: (2020) included a local convergence analysis for DSEG applied to locally monotone problems.
Revised: (2020) included a local convergence analysis for DSEG applied to locally monotone problems.

Original: For min-max problems, if the objective is locally convex-concave at a solution and DSEG is initialized in close proximity, then for small enough stepsizes it converges to the solution with high probability.
Revised: For min-max problems, if the objective is locally convex-concave at a solution and DSEG is initialized in close proximity, then for small enough stepsizes it converges to the solution with high probability.

Original: It is possible to extend this result to SPS, along with our convergence rate analysis.
Revised: It is possible to extend this result to SPS, along with our convergence rate analysis.

Original: This result is beyond the scope of this work, but Appendix
Revised: This result is beyond the scope of this work, but Appendix

Original: J provides a proof sketch.
Revised: J provides a proof sketch.

Original:  B ACKGROUND ON M ONOTONE I NCLUSIONS
Revised:  B ACKGROUND ON M ONOTONE I NCLUSIONS

Original: Since they are so important to SPS, this section provides some background material regarding monotone inclusions, along with their connections to convex optimization, games, and ML.
Revised: Since they are so important to SPS, this section provides some background material regarding monotone inclusions, along with their connections to convex optimization, games, and ML.

Original: Appendix G discusses their connections to variational inequalities.
Revised: Appendix G discusses their connections to variational inequalities.

Original: For a more thorough treatment, we refer to Bauschke & Combettes (2017).
Revised: For a more thorough treatment, we refer to Bauschke & Combettes (2017).

Original: See Appendix A for a longer discussion of the applications of monotone inclusions to ML along with several examples.
Revised: See Appendix A for a longer discussion of the applications of monotone inclusions to ML along with several examples.

Original: Fundamentals Let f : R d → R ∪ {∞} be closed, convex, and proper (CCP).
Revised: Fundamentals Let f : R d → R ∪ {∞} be closed, convex, and proper (CCP).

Original: Recall that its subdifferential ∂f is given by ∂f ( x ) .
Revised: Recall that its subdifferential ∂f is given by ∂f ( x ) .

Original: = { g : f ( y ) ≥ f ( x ) + g (cid:62) ( y − x ) } .
Revised: = { g : f ( y ) ≥ f ( x ) + g (cid:62) ( y − x ) } .

Original: The map ∂f has the property
Revised: The map ∂f has the property

Original: [Equation]
Revised: [Equation]

Original: and any point-to-set map having this property is called a monotone operator .
Revised: and any point-to-set map having this property is called a monotone operator .

Original: A monotone operator
Revised: A monotone operator

Original: T is called maximal if no additional points can be included in the image T ( x ) of any x ∈ R d without violating the above property (Bauschke & Combettes, 2017, Def. 20.20).
Revised: T is called maximal if no additional points can be included in the image T ( x ) of any x ∈ R d without violating the above property (Bauschke & Combettes, 2017, Def. 20.20).

Original: Subgradient maps of CCP functions are maximal (Bauschke & Combettes, 2017, Thm. 20.25).
Revised: Subgradient maps of CCP functions are maximal (Bauschke & Combettes, 2017, Thm. 20.25).

Original: A minimizer of f is any x ∗ such that 0 ∈ ∂f ( x ∗ ) .
Revised: A minimizer of f is any x ∗ such that 0 ∈ ∂f ( x ∗ ) .

Original: This is perhaps the simplest example of a monotone inclusion , the problem of ﬁnding x such that 0 ∈ T ( x ) , where T is a monotone operator.
Revised: This is perhaps the simplest example of a monotone inclusion , the problem of ﬁnding x such that 0 ∈ T ( x ) , where T is a monotone operator.

Original: If f is smooth, then ∂f ( x ) = {∇ f ( x ) } for all x , and the monotone inclusion 0 ∈ ∂f ( x ) is equivalent to the ﬁrst-order optimality condition 

Revised: If f is smooth, then ∂f ( x ) = {∇ f ( x ) } for all x , and the monotone inclusion 0 ∈ ∂f ( x ) is equivalent to the ﬁrst-order optimality condition

Original: = ∇ f ( x ) .
Revised: = ∇ f ( x ) .

Original: Under certain regularity conditions (Bauschke & Combettes, 2017, Cor. 16.5), minimizing a sum of
Revised: Under certain regularity conditions (Bauschke & Combettes, 2017, Cor. 16.5), minimizing a sum of

Original: CCP functions f 1 , . .
Revised: CCP functions f 1 , . .

Original: . , f n is equivalent to solving the monotone inclusion formed from the sum of their subdifferentials:
Revised: . , f n is equivalent to solving the monotone inclusion formed from the sum of their subdifferentials:

Original: [Equation]
Revised: [Equation]

Original: As throughout this paper for all set addition operations, the summation on the right-hand side of (2) is the Minkowski sum (cid:80) ni =1 S i
Revised: As throughout this paper for all set addition operations, the summation on the right-hand side of (2) is the Minkowski sum (cid:80) ni =1 S i

Original: = { (cid:80) ni
Revised: = { (cid:80) ni

Original: =

Revised: =

Original: s i | s
Revised: s i | s

Original: i ∈ S
Revised: i ∈ S

Original: i ∀ i ∈ 1 ..n } .
Revised: i ∀ i ∈ 1 ..n } .

Original: For a convex set X , a constraint x ∈ C for some convex set C may be imposed by setting one of the f i to be the indicator function ι C , deﬁned by ι C ( x ) = 0 for x ∈ C and ι C ( x ) =
Revised: For a convex set X , a constraint x ∈ C for some convex set C may be imposed by setting one of the f i to be the indicator function ι C , deﬁned by ι C ( x ) = 0 for x ∈ C and ι C ( x ) =

Original: + ∞ for x (cid:54)∈ C .
Revised: + ∞ for x (cid:54)∈ C .

Original: Indicator functions of closed convex sets are CCP (Bauschke & Combettes, 2017, Ex. 1.25), and the subgradient map of ι C is also referred to as the normal cone map N C of C (Bauschke & Combettes, 2017, Def. 6.37).
Revised: Indicator functions of closed convex sets are CCP (Bauschke & Combettes, 2017, Ex. 1.25), and the subgradient map of ι C is also referred to as the normal cone map N C of C (Bauschke & Combettes, 2017, Def. 6.37).

Original: Multiple constraints may be imposed by including multiple indicator functions in (2).
Revised: Multiple constraints may be imposed by including multiple indicator functions in (2).

Original: ML applications The form (2) can be used to model ML problems with multiple constraints and/or nonsmooth regularizers, including sparse and overlapping group lasso (Jacob et al., 2009), sparse and low-rank matrix estimation problems (Richard et al., 2012), and rare feature selection (Yan & Bien, 2020); see Pedregosa & Gidel (2018) for an overview.
Revised: ML applications The form (2) can be used to model ML problems with multiple constraints and/or nonsmooth regularizers, including sparse and overlapping group lasso (Jacob et al., 2009), sparse and low-rank matrix estimation problems (Richard et al., 2012), and rare feature selection (Yan & Bien, 2020); see Pedregosa & Gidel (2018) for an overview.

Original: Games Consider a two-player noncooperative game in which each player tries to selﬁshly minimize its own loss, with each loss depending on the actions of both players.
Revised: Games Consider a two-player noncooperative game in which each player tries to selﬁshly minimize its own loss, with each loss depending on the actions of both players.

Original: Typically, the goal is to ﬁnd a
Revised: Typically, the goal is to ﬁnd a

Original: Nash equilibrium, in which neither player can improve its loss by changing strategy:
Revised: Nash equilibrium, in which neither player can improve its loss by changing strategy:

Original: [Equation]
Revised: [Equation]

Original: Assuming that the admissible strategy sets Θ ⊆ R d x and Ω ⊆
Revised: Assuming that the admissible strategy sets Θ ⊆ R d x and Ω ⊆

Original: R d y are closed and convex and that
Revised: R d y are closed and convex and that

Original: F and G are differentiable,  the ﬁrst-order necessary conditions for solving the Nash equilibrium problem are
Revised: F and G are differentiable, then writing the ﬁrst-order necessary conditions for each optimization problem in (3) yields

Original: [Equation]
Revised: [Equation]

Original: If G = − F , then (3) is a min-max game.
Revised: If G = − F , then (3) is a min-max game.

Original: If F is also convex in x and concave in y , then B : ( x, y ) (cid:55)→ ( ∇ x F ( x, y ) , −∇ y F ( x, y ))
Revised: If F is also convex in x and concave in y , then B : ( x, y ) (cid:55)→ ( ∇ x F ( x, y ) , −∇ y F ( x, y ))

Original: (cid:62) is monotone 1 on R d x + d y (Rockafellar, 1970).
Revised: (cid:62) is monotone 1 on R d x + d y (Rockafellar, 1970).

Original: In many applications, B is also Lipschitz continuous.
Revised: In many applications, B is also Lipschitz continuous.

Original: In this situation, (4) is a monotone inclusion involving two operators
Revised: In this situation, (4) is a monotone inclusion involving two operators

Original: B and N Θ × Ω , with B being Lipschitz.
Revised: B and N Θ × Ω , with B being Lipschitz.

Original: Using the simultaneous version of GDA on (3) is equivalent to applying the forward-backward method (FB)
Revised: Using the simultaneous version of GDA on (3) is equivalent to applying the forward-backward method (FB)

Original: (Bauschke & Combettes, 2017, Thm. 26.14) to (4).
Revised: (Bauschke & Combettes, 2017, Thm. 26.14) to (4).

Original: However, convergence of FB requires that the operator B be cocoercive (Bauschke & Combettes, 2017, Def. 4.10), and not merely Lipschitz (Bauschke & Combettes, 2017, Thm. 26.14).
Revised: However, convergence of FB requires that the operator B be cocoercive (Bauschke & Combettes, 2017, Def. 4.10), and not merely Lipschitz (Bauschke & Combettes, 2017, Thm. 26.14).

Original: Thus, simultaneous GDA fails to converge for (3) without additional assumptions; see Gidel et al.
Revised: Thus, simultaneous GDA fails to converge for (3) without additional assumptions; see Gidel et al.

Original: Prop. 1) for a simple counterexample.
Revised: Prop. 1) for a simple counterexample.

Original: Regularizers and further constraints may be imposed by adding more operators to (4).
Revised: Regularizers and further constraints may be imposed by adding more operators to (4).

Original: For example, if one wished to apply a (nonsmooth) convex regularizer r : R d x → R ∪ { + ∞} to the x variables and a similar regularizer d : R d y → R ∪ { + ∞} to the y variables, one would add the operator
Revised: For example, if one wished to apply a (nonsmooth) convex regularizer r : R d x → R ∪ { + ∞} to the x variables and a similar regularizer d : R d y → R ∪ { + ∞} to the y variables, one would add the operator

Original: A 2 : ( x, y )
Revised: A 2 : ( x, y )

Original: (cid:55)→ ∂r ( x ) × ∂d ( y ) to the right-hand side of (4).
Revised: (cid:55)→ ∂r ( x ) × ∂d ( y ) to the right-hand side of (4).

Original: ML applications of games Distributionally robust supervised learning (DRSL) is an emerging framework for improving the stability and reliability of ML models in the face of distributional shifts (Yu et al., 2021;
Revised: ML applications of games Distributionally robust supervised learning (DRSL) is an emerging framework for improving the stability and reliability of ML models in the face of distributional shifts (Yu et al., 2021;

Original: Kuhn et al., 2019; Shaﬁeezadeh-Abadeh et al., 2015; Sinha et al., 2018; Lin et al., 2020; Namkoong & Duchi, 2016).
Revised: Kuhn et al., 2019; Shaﬁeezadeh-Abadeh et al., 2015; Sinha et al., 2018; Lin et al., 2020; Namkoong & Duchi, 2016).

Original: Common approaches to DRSL formulate the problem as a min-max game between a learner selecting the model parameters and an adversary selecting a worst-case distribution subject to some ambiguity set around the observed empirical distribution.
Revised: Common approaches to DRSL formulate the problem as a min-max game between a learner selecting the model parameters and an adversary selecting a worst-case distribution subject to some ambiguity set around the observed empirical distribution.

Original: This min-max problem is often further reduced to either a ﬁnite-dimensional saddlepoint problem or a convex optimization problem.
Revised: This min-max problem is often further reduced to either a ﬁnite-dimensional saddlepoint problem or a convex optimization problem.

Original: DRSL is a source of games with multiple constraints/regularizers.
Revised: DRSL is a source of games with multiple constraints/regularizers.

Original: One such formulation, based on
Revised: One such formulation, based on

Original: Yu et al.
Revised: Yu et al.

Original: (2021), is discussed in the experiments below.
Revised: (2021), is discussed in the experiments below.

Original: The work in Namkoong & Duchi (2016) uses an ambiguity set based on f -divergences, while Sinha et al.
Revised: The work in Namkoong & Duchi (2016) uses an ambiguity set based on f -divergences, while Sinha et al.

Original: (2018) introduce a Lagrangian relaxation of the Wasserstein ball.
Revised: (2018) introduce a Lagrangian relaxation of the Wasserstein ball.

Original: When applied to models utilizing multiple regularizers (Jacob et al., 2009;
Revised: When applied to models utilizing multiple regularizers (Jacob et al., 2009;

Original: Richard et al., 2012; Yan & Bien, 2020), both of these approaches lead to min-max problems with multiple regularizers.
Revised: Richard et al., 2012; Yan & Bien, 2020), both of these approaches lead to min-max problems with multiple regularizers.

Original: Other applications of games in ML, although typically nonconvex, include generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017), fair classiﬁcation (Wadsworth et al., 2018; Zhang et al., 2018; Edwards & Storkey, 2015; Celis & Keswani, 2019), and adversarial privacy (Huang et al., 2017).
Revised: Other applications of games in ML, although typically nonconvex, include generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017; Loizou et al., 2020; 2021; Mishchenko et al., 2020), fair classiﬁcation (Wadsworth et al., 2018; Zhang et al., 2018; Edwards & Storkey, 2015; Celis & Keswani, 2019), and adversarial privacy (Huang et al., 2017).

Original: Resolvents, proximal operators, and projections A fundamental computational primitive for solving monotone inclusions is the resolvent .
Revised: Resolvents, proximal operators, and projections A fundamental computational primitive for solving monotone inclusions is the resolvent .

Original: The resolvent of a monotone operator A is deﬁned to be J A .
Revised: The resolvent of a monotone operator A is deﬁned to be J A .

Original: = ( I + A ) − 1 , where I is the identity operator and the inverse of any operator T is simply
Revised: = ( I + A ) − 1 , where I is the identity operator and the inverse of any operator T is simply

Original: T − 1 : x (cid:55)→ { y : Ty (cid:51) x } .
Revised: T − 1 : x (cid:55)→ { y : Ty (cid:51) x } .

Original: If A is maximal monotone, then for any ρ > 0 , J ρA is single valued, nonexpansive, and has domain equal to R d (Bauschke & Combettes, 2017, Thm.
Revised: If A is maximal monotone, then for any ρ > 0 , J ρA is single valued, nonexpansive, and has domain equal to R d (Bauschke & Combettes, 2017, Thm.

Original: 21.1 and Prop.
Revised: 21.1 and Prop.

Original: Resolvents generalize proximal operators of convex functions: the proximal operator of a CCP function f is 
Revised: Resolvents generalize proximal operators of convex functions: the proximal operator of a CCP function f is prox

Original: [Equation]
Revised: [Equation]

Original: It is easily proved that prox ρf =
Revised: It is easily proved that prox ρf =

Original: J ρ∂f .
Revised: J ρ∂f .

Original: Like proximal operators, resolvents generalize projection onto convex sets: if f = ι C , then J ρN C = prox ρf =
Revised: Like proximal operators, resolvents generalize projection onto convex sets: if f = ι C , then J ρN C = prox ρf =

Original: proj C for any ρ > 0 .
Revised: proj C for any ρ > 0 .

Original: In many ML applications, proximal operators, and hence resolvents, are relatively straightforward to compute.
Revised: In many ML applications, proximal operators, and hence resolvents, are relatively straightforward to compute.

Original: For examples, see Parikh & Boyd (2013, Sec. 6).
Revised: For examples, see Parikh & Boyd (2013, Sec. 6).

Original: Operator splitting methods Operator splitting methods attempt to solve monotone inclusions such as (1) by a sequence of operations that each involve only one of the operators
Revised: Operator splitting methods Operator splitting methods attempt to solve monotone inclusions such as (1) by a sequence of operations that each involve only one of the operators

Original: A 1 , . . . , A n , B .
Revised: A 1 , . . . , A n , B .

Original: Such methods are often presented in the context of convex optimization problems like (2), but typically apply more generally to monotone inclusions such as (1).
Revised: Such methods are often presented in the context of convex optimization problems like (2), but typically apply more generally to monotone inclusions such as (1).

Original: In the speciﬁc context of (1), each iteration of such a method ideally handles each A i via its resolvent and the Lipschitz operator B by explicit (not stochastic) evaluation.
Revised: In the speciﬁc context of (1), each iteration of such a method ideally handles each A i via its resolvent and the Lipschitz operator B by explicit (not stochastic) evaluation.

Original: This is a feasible approach if the original problem can be decomposed in such a way that the resolvents of each A i are relatively inexpensive to compute, and full evaluations of B are possible.
Revised: This is a feasible approach if the original problem can be decomposed in such a way that the resolvents of each A i are relatively inexpensive to compute, and full evaluations of B are possible.

Original: Although not discussed here, more general formulations in which matrices couple the arguments of the operators can broaden the applicability of operator splitting methods.
Revised: Although not discussed here, more general formulations in which matrices couple the arguments of the operators can broaden the applicability of operator splitting methods.

Original:  T HE P ROJECTIVE S PLITTING F RAMEWORK
Revised:  T HE P ROJECTIVE S PLITTING F RAMEWORK

Original: Before introducing our proposed method, we give a brief introduction to the projective splitting class of methods.
Revised: Before introducing our proposed method, we give a brief introduction to the projective splitting class of methods.

Original: The extended solution set Projective splitting is a primal-dual framework and operates in an extended space of primal and dual variables.
Revised: The extended solution set Projective splitting is a primal-dual framework and operates in an extended space of primal and dual variables.

Original: Rather than directly ﬁnding a solution to (1), we ﬁnd a point in the extended solution set (or Kuhn-Tucker set )
Revised: Rather than directly ﬁnding a solution to (1), we ﬁnd a point in the extended solution set (or Kuhn-Tucker set )

Original: [Equation]
Revised: [Equation]

Original: Given p ∗ =
Revised: Given p ∗ =

Original: ( z ∗ , w ∗ 1 . . .
Revised: ( z ∗ , w ∗ 1 . . .

Original: , w ∗ n +1 ) ∈ S , it is straightforward to see that z ∗ solves (1).
Revised: , w ∗ n +1 ) ∈ S , it is straightforward to see that z ∗ solves (1).

Original: Conversely, given a solution z ∗ to (1), there must exist w ∗ 1 , . . .
Revised: Conversely, given a solution z ∗ to (1), there must exist w ∗ 1 , . . .

Original: , w ∗ n +1 such that ( z ∗ , w ∗ 1 , . . .
Revised: , w ∗ n +1 such that ( z ∗ , w ∗ 1 , . . .

Original: , w ∗ n +1 ) ∈ S .
Revised: , w ∗ n +1 ) ∈ S .

Original: Suppose p ∗ =
Revised: Suppose p ∗ =

Original: ( z ∗ , w ∗ 1 . . .
Revised: ( z ∗ , w ∗ 1 . . .

Original: , w ∗ n +1 ) ∈ S .
Revised: , w ∗ n +1 ) ∈ S .

Original: Since z ∗ solves (1), z ∗ is typically referred to as a primal solution .
Revised: Since z ∗ solves (1), z ∗ is typically referred to as a primal solution .

Original: The vectors w ∗ 1 , . .
Revised: The vectors w ∗ 1 , . .

Original: . , w ∗ n +1 solve a dual inclusion not described here, and are therefore called a dual solution .
Revised: . , w ∗ n +1 solve a dual inclusion not described here, and are therefore called a dual solution .

Original: It can be shown that S is closed and convex; see for example Johnstone & Eckstein (2020b).
Revised: It can be shown that S is closed and convex; see for example Johnstone & Eckstein (2020b).

Original: We will assume throughout that a solution to (1) exists, therefore the set S is nonempty.
Revised: We will assume throughout that a solution to (1) exists, therefore the set S is nonempty.

Original: Separator-projection framework Projective splitting methods are instances of the general separator-projection algorithmic framework for locating a member of a closed convex set S within a linear space P .
Revised: Separator-projection framework Projective splitting methods are instances of the general separator-projection algorithmic framework for locating a member of a closed convex set S within a linear space P .

Original: Each iteration k of algorithms drawn from this framework operates by ﬁnding a set H k that separates the current iterate p k ∈ P from S , meaning that S is entirely in the set and p k typically is not.
Revised: Each iteration k of algorithms drawn from this framework operates by ﬁnding a set H k that separates the current iterate p k ∈ P from S , meaning that S is entirely in the set and p k typically is not.

Original: One then attempts to “move closer" to S by projecting the p k onto H k .
Revised: One then attempts to “move closer" to S by projecting the p k onto H k .

Original: In the particular case of projective splitting applied to the problem (1) using (5), we select the space P to be
Revised: In the particular case of projective splitting applied to the problem (1) using (5), we select the space P to be

Original: [Equation]
Revised: [Equation]

Original: and each separating set H k to be the half space { p ∈ P | ϕ k ( p ) ≤ 0 } generated by an afﬁne function ϕ
Revised: and each separating set H k to be the half space { p ∈ P | ϕ k ( p ) ≤ 0 } generated by an afﬁne function ϕ

Original: k : P → R .
Revised: k : P → R .

Original: The general intention is to construct ϕ k such that ϕ k ( p k ) > 0 , but ϕ k ( p ∗ ) ≤ 0 for all p ∗ ∈ S .
Revised: The general intention is to construct ϕ k such that ϕ k ( p k ) > 0 , but ϕ k ( p ∗ ) ≤ 0 for all p ∗ ∈ S .

Original: The construction employed for ϕ k in the case of (1) and (5) is of the form
Revised: The construction employed for ϕ k in the case of (1) and (5) is of the form

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: for some points ( x ki , y ki ) ∈ R 2 d , i ∈ 1 .. ( n + 1) , that must be carefully chosen (see below).
Revised: for some points ( x ki , y ki ) ∈ R 2 d , i ∈ 1 .. ( n + 1) , that must be carefully chosen (see below).

Original: Any function of the form (7) can be shown to be afﬁne when restricted to P .
Revised: Any function of the form (7) can be shown to be afﬁne when restricted to P .

Original: As mentioned above, the standard separator-projection algorithm obtains its next iterate p k
Revised: As mentioned above, the standard separator-projection algorithm obtains its next iterate p k

Original: +1 by projecting p k onto H k .
Revised: +1 by projecting p k onto H k .

Original: This calculation involves the usual projection step for a half space, namely
Revised: This calculation involves the usual projection step for a half space, namely

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and the gradient ∇ ϕ k is computed relative to P , thus resulting in p k +1 ∈ P (over- or under-relaxed variants of this step are also possible).
Revised: and the gradient ∇ ϕ k is computed relative to P , thus resulting in p k +1 ∈ P , i.e. ∇ ϕ k = (cid:16)(cid:80) n +1 i =1 y ki , x k 1 − ¯ x k , . . . , x n +1 − ¯ x k (cid:17) where ¯ x k = 1 n +1 (cid:80) n +1 i =1 x ki .

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: The proposed method is given in Algorithm 1 and called Stochastic Projective Splitting (SPS).
Revised: The proposed method is given in Algorithm 1 and called Stochastic Projective Splitting (SPS).

Original: Unlike prior versions of projective splitting, SPS does not employ the stepsize α k of (8) that places the next iterate exactly on the hyperplane given by ϕ k ( p ) = 0 .
Revised: Unlike prior versions of projective splitting, SPS does not employ the stepsize α k of (8) that places the next iterate exactly on the hyperplane given by ϕ k ( p ) = 0 .

Original: Instead, it simply moves in the direction −∇ ϕ k with a pre-deﬁned stepsize { α k } .
Revised: Instead, it simply moves in the direction −∇ ϕ k with a pre-deﬁned stepsize { α k } .

Original: This fundamental change is required to deal with the stochastic noise on lines 6 and 8.
Revised: This fundamental change is required to deal with the stochastic noise on lines 6 and 8.

Original: This noise could lead to the usual choice of α k deﬁned in (8) being unstable and difﬁcult to analyze.
Revised: This noise could lead to the usual choice of α k deﬁned in (8) being unstable and difﬁcult to analyze.

Original: In order to guarantee convergence, the parameters α k and ρ k must be chosen to satisfy certain conditions given below.
Revised: In order to guarantee convergence, the parameters α k and ρ k must be chosen to satisfy certain conditions given below.

Original: Note that the gradient is calculated with respect to the subspace P deﬁned in (6); since the algorithm is initialized within P , it remains in P , within which ϕ k is afﬁne.
Revised: Note that the gradient is calculated with respect to the subspace P deﬁned in (6); since the algorithm is initialized within P , it remains in P , within which ϕ k is afﬁne.

Original: Collectively, the updates on lines 9-10 are equivalent to p k
Revised: Collectively, the updates on lines 9-10 are equivalent to p k

Original: +

Revised: +

Original: =
Revised: =

Original: p k − α k ∇ ϕ k , where p k
Revised: p k − α k ∇ ϕ k , where p k

Original: =
Revised: =

Original: ( z k , w k 1 , . . . , w k n +1 ) .
Revised: ( z k , w k 1 , . . . , w k n +1 ) .

Original: Note that SPS does not explicitly evaluate ϕ k , which is only used in the analysis, but it does keep track of ( x ki , y ki ) for i ∈ 1 .. ( n + 1) .
Revised: Note that SPS does not explicitly evaluate ϕ k , which is only used in the analysis, but it does keep track of ( x ki , y ki ) for i ∈ 1 .. ( n + 1) .

Original: The algorithm’s memory requirements scale linearly with the number of nonsmooth operators n in the inclusion (1), with the simplest implementation storing (3 n +5)
Revised: The algorithm’s memory requirements scale linearly with the number of nonsmooth operators n in the inclusion (1), with the simplest implementation storing (3 n +5)

Original: d working-vector elements.
Revised: d working-vector elements.

Original: This requirement can be reduced to ( n +7)
Revised: This requirement can be reduced to ( n +7)

Original: d through a technique discussed in Appendix H. In most applications, n will be small, for example 2 or 3 .
Revised: d through a technique discussed in Appendix H. In most applications, n will be small, for example 2 or 3 .

Original: Updating ( x ki , y ki )
Revised: Updating ( x ki , y ki )

Original: The variables ( x ki , y ki ) are updated on lines 3-8 of Algorithm 1, in which e k and (cid:15) k are R d -valued random variables deﬁned on a probability space (Ω , F , P ) .
Revised: The variables ( x ki , y ki ) are updated on lines 3-8 of Algorithm 1, in which e k and (cid:15) k are R d -valued random variables deﬁned on a probability space (Ω , F , P ) .

Original: For B we use a new, noisy version of the two-forward-step procedure from Johnstone & Eckstein (2020b).
Revised: For B we use a new, noisy version of the two-forward-step procedure from Johnstone & Eckstein (2020b).

Original: For each
Revised: For each

Original: A i , i ∈ 1 ..n , we use the same resolvent step used in previous projective splitting papers, originating with (Eckstein & Svaiter, 2008).
Revised: A i , i ∈ 1 ..n , we use the same resolvent step used in previous projective splitting papers, originating with (Eckstein & Svaiter, 2008).

Original: In the case (cid:15) k
Revised: In the case (cid:15) k

Original: = e k = 0 , the selection of the ( x ki , y ki ) is identical to that proposed by Johnstone & Eckstein (2020b), resulting in the hyperplane { p : ϕ k ( p )
Revised: = e k = 0 , the selection of the ( x ki , y ki ) is identical to that proposed by Johnstone & Eckstein (2020b), resulting in the hyperplane { p : ϕ k ( p )

Original: = 0 } strictly separating p k from S .
Revised: = 0 } strictly separating p k from S .

Original: SPS achieves full splitting of (1): each A i is processed separately using a resolvent and the Lipschitz term B is processed via a stochastic gradient oracle.
Revised: SPS achieves full splitting of (1): each A i is processed separately using a resolvent and the Lipschitz term B is processed via a stochastic gradient oracle.

Original: When the A i arise from regularizers or constraints, as discussed in Section 2, their resolvents can be readily computed so long as their respective proximal/projection operators have a convenient form.
Revised: When the A i arise from regularizers or constraints, as discussed in Section 2, their resolvents can be readily computed so long as their respective proximal/projection operators have a convenient form.

Original: Noise assumptions Let F k .
Revised: Noise assumptions Let F k .

Original: =
Revised: =

Original: σ ( p 1 , . . .
Revised: σ ( p 1 , . . .

Original: , p k ) and E k .
Revised: , p k ) and E k .

Original: =
Revised: =

Original: σ ( (cid:15) k ) .
Revised: σ ( (cid:15) k ) .

Original: The stochastic estimators for the gradients, r k and y kn +1 , are assumed to be unbiased, that is, the noise terms have mean 0 conditioned on the past:
Revised: The stochastic estimators for the gradients, r k and y kn +1 , are assumed to be unbiased, that is, the noise terms have mean 0 conditioned on the past:

Original: [Equation]
Revised: [Equation]

Original: We impose the following mild assumptions on the variance of the noise:
Revised: We impose the following mild assumptions on the variance of the noise:

Original: [Equation]
Revised: [Equation]

Original: where 0 ≤ N 1 , N 2 , N 3 , N 4 < ∞ .
Revised: where 0 ≤ N 1 , N 2 , N 3 , N 4 < ∞ .

Original: We do not require e k and (cid:15) k to be independent of one another.
Revised: We do not require e k and (cid:15) k to be independent of one another.

Original: Stepsize choices The stepsizes ρ k and α k are assumed to be deterministic.
Revised: Stepsize choices The stepsizes ρ k and α k are assumed to be deterministic.

Original: A constant stepsize choice which attains a non-asymptotic convergence rate will be considered in the next section (Theorem 2).
Revised: A constant stepsize choice which attains a non-asymptotic convergence rate will be considered in the next section (Theorem 2).

Original: The stepsize conditions we will impose to guarantee almost-sure convergence (Theorem 1) are
Revised: The stepsize conditions we will impose to guarantee almost-sure convergence (Theorem 1) are

Original: [Equation]
Revised: [Equation]

Original: For example, in the case L = 1 , a particular choice which satisﬁes these constraints is α k = k − 0 .
Revised: For example, in the case L = 1 , a particular choice which satisﬁes these constraints is α k = k − 0 .

Original: 5 − p for 0 < p < 0 .
Revised: 5 − p for 0 < p < 0 .

Original: 5 , and ρ k = k − 0 .
Revised: 5 , and ρ k = k − 0 .

Original: 5+ t for p ≤ t < 0 .
Revised: 5+ t for p ≤ t < 0 .

Original: 5 p + 0 .
Revised: 5 p + 0 .

Original: For simplicity, the stepsizes τ used for the resolvent updates in lines 3-5 are ﬁxed, but they could be allowed to vary with both i and k so long as they have ﬁnite positive lower and upper bounds.
Revised: For simplicity, the stepsizes τ used for the resolvent updates in lines 3-5 are ﬁxed, but they could be allowed to vary with both i and k so long as they have ﬁnite positive lower and upper bounds.

Original: Algorithm
Revised: Algorithm

Original: Input : p
Revised: Input : p

Original: [Equation]
Revised: [Equation]

Original: 1 i = 0 , { k k } k
Revised: 1 i = 0 , { k k } k

Original: , τ > t ki =
Revised: , τ > t ki =

Original: z k + τw ki ki
Revised: z k + τw ki ki

Original: J ki ki τ t ki
Revised: J ki ki τ t ki

Original: // (cid:15) k is unknown noise term
Revised: // (cid:15) k is unknown noise term

Original: e k is unknown noise term ki
Revised: e k is unknown noise term ki

Original:  M AIN T HEORETICAL R ESULTS
Revised:  M AIN T HEORETICAL R ESULTS

Original: Theorem 1.
Revised: Theorem 1.

Revised: Suppose A 1 , . . .

Revised: , A n are maximal monotone, B is L -Lipschitz and monotone, and a solution to (1) exists.

Original: For Algorithm 1, suppose (9) - (12) hold.
Revised: For Algorithm 1, suppose (9) - (12) hold.

Original: Then with probability one it holds that z k → z ∗ , where z ∗ solves (1) .
Revised: Then with probability one it holds that z k → z ∗ , where z ∗ solves (1) .

Original: Proof sketch Theorem 1 is proved in Appendix C, but we provide a brief sketch here.
Revised: Proof sketch Theorem 1 is proved in Appendix C, but we provide a brief sketch here.

Original: The proof begins by deriving a simple recursion inspired by the analysis of SGD (Robbins & Monro, 1951).
Revised: The proof begins by deriving a simple recursion inspired by the analysis of SGD (Robbins & Monro, 1951).

Original: Since p k +

Revised: Since p k +

Original: = p k − α k ∇ ϕ k , a step of projective splitting can be viewed as GD applied to the afﬁne hyperplane generator function ϕ k .
Revised: = p k − α k ∇ ϕ k , a step of projective splitting can be viewed as GD applied to the afﬁne hyperplane generator function ϕ k .

Original: Thus, for any p ∗ ∈ P ,
Revised: Thus, for any p ∗ ∈ P ,

Original: [Equation]
Revised: [Equation]

Original: where in the second equation we have used that ϕ k ( p ) is afﬁne on P .
Revised: where in the second equation we have used that ϕ k ( p ) is afﬁne on P .

Original: The basic strategy is to show that, for any p ∗ ∈ S ,
Revised: The basic strategy is to show that, for any p ∗ ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: for some C 1 , C 2 > 0 .
Revised: for some C 1 , C 2 > 0 .

Original: This condition allows one to establish stochastic quasi-Fejér monotonicity (SQFM) (Combettes & Pesquet, 2015, Proposition 2.3) of the iterates to S .
Revised: This condition allows one to establish stochastic quasi-Fejér monotonicity (SQFM) (Combettes & Pesquet, 2015, Proposition 2.3) of the iterates to S .

Original: One consequence of
Revised: One consequence of

Original: SQFM is that with probability one there exists a subsequence v k such that ϕ v k ( p v k ) − ϕ v k ( p ∗ ) converges to 0 .
Revised: SQFM is that with probability one there exists a subsequence v k such that ϕ v k ( p v k ) − ϕ v k ( p ∗ ) converges to 0 .

Original: Furthermore, roughly speaking, we show that ϕ k ( p k ) − ϕ k
Revised: Furthermore, roughly speaking, we show that ϕ k ( p k ) − ϕ k

Original: ( p ∗ ) provides an upper bound on the following “approximation residual" for SPS:
Revised: ( p ∗ ) provides an upper bound on the following “approximation residual" for SPS:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: G k provides an approximation error for SPS, as formalized in the following lemma:
Revised: G k provides an approximation error for SPS, as formalized in the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: For SPS, p k
Revised: For SPS, p k

Original: =
Revised: =

Original: ( z k , w k 1 , . . . , w kn +1 ) ∈ S
Revised: ( z k , w k 1 , . . . , w kn +1 ) ∈ S

Original: if and only if G k = 0 .
Revised: if and only if G k = 0 .

Original: Since y ki ∈ A i ( x ki ) for i ∈ 1 ..n , having G k = 0 implies that z k = x ki , w ki = y ki , and thus w ki ∈
Revised: Since y ki ∈ A i ( x ki ) for i ∈ 1 ..n , having G k = 0 implies that z k = x ki , w ki = y ki , and thus w ki ∈

Original: A i ( z k ) for i ∈ 1 ..n .
Revised: A i ( z k ) for i ∈ 1 ..n .

Original: Since w kn +1 = B ( z k ) and (cid:80) n +1 i =1 w ki = 0 , it follows that z k solves (1).
Revised: Since w kn +1 = B ( z k ) and (cid:80) n +1 i =1 w ki = 0 , it follows that z k solves (1).

Original: The reverse direction is proved in Appendix D.
Revised: The reverse direction is proved in Appendix D.

Original: The quantity G k generalizes the role played by the norm of the gradient in algorithms for smooth optimization.
Revised: The quantity G k generalizes the role played by the norm of the gradient in algorithms for smooth optimization.

Original: In particular, in the special case where n = 0 and B ( z ) = ∇ f ( z ) for some smooth convex function f , one has G k
Revised: In particular, in the special case where n = 0 and B ( z ) = ∇ f ( z ) for some smooth convex function f , one has G k

Original: =
Revised: =

Original: (cid:107)∇ f ( z k ) (cid:107) 2 .
Revised: (cid:107)∇ f ( z k ) (cid:107) 2 .

Original: Combining the properties of G k with other results following from SQFM (such as boundedness) will allow us to derive almost-sure convergence of the iterates to a solution of (1).
Revised: Combining the properties of G k with other results following from SQFM (such as boundedness) will allow us to derive almost-sure convergence of the iterates to a solution of (1).

Original: Convergence rate We can also establish non-asymptotic convergence rates for the approximation residual G k :
Revised: Convergence rate We can also establish non-asymptotic convergence rates for the approximation residual G k :

Original: Theorem 2.
Revised: Theorem 2.

Original: Fix the total iterations K ≥ 1 of Algorithm 1 and set
Revised: Fix the total iterations K ≥ 1 of Algorithm 1 and set

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for some C f > 0 .
Revised: for some C f > 0 .

Original: Suppose (9) - (11) hold.
Revised: Suppose (9) - (11) hold.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where the constants are given (along with the proof) in Appendix E.
Revised: where the constants are given (along with the proof) in Appendix E.

Original: Theorem 2 implies that if we pick an iterate J uniformly at random from 1 ..K , then the expected value of G J is O ( K − 1 / 4 ) .
Revised: Theorem 2 implies that if we pick an iterate J uniformly at random from 1 ..K , then the expected value of G J is O ( K − 1 / 4 ) .

Original: As far as we know, this is the ﬁrst convergence rate for a stochastic fullsplitting method solving (1) in the general discontinuous (i.e. set-valued) monotone inclusion case, and it is not clear whether it can be improved, either by a better analysis or a better method.
Revised: As far as we know, this is the ﬁrst convergence rate for a stochastic fullsplitting method solving (1) in the general discontinuous (i.e. set-valued) monotone inclusion case, and it is not clear whether it can be improved, either by a better analysis or a better method.

Original: Faster rates are certainly possible for deterministic methods under various continuity assumptions; Tseng’s method obtains O ( K − 1 ) rate (Monteiro & Svaiter, 2010) and the accelerated Halpern iteration under
Revised: Faster rates are certainly possible for deterministic methods under various continuity assumptions; Tseng’s method obtains O ( K − 1 ) rate (Monteiro & Svaiter, 2010) and the accelerated Halpern iteration under

Original: Lipschitz continuity obtains O ( K − 2 ) rate (Diakonikolas, 2020).
Revised: Lipschitz continuity obtains O ( K − 2 ) rate (Diakonikolas, 2020).

Original: While our rate may seem slow, it is worth remembering that (1) features n discontinuous operators
Revised: While our rate may seem slow, it is worth remembering that (1) features n discontinuous operators

Original: A i , so we expect rates at least as slow as nonsmooth convex optimization, but perhaps worse because (1) is far more general than convex optimization.
Revised: A i , so we expect rates at least as slow as nonsmooth convex optimization, but perhaps worse because (1) is far more general than convex optimization.

Original: For a different error metric, the restricted gap function, in the special case of variational inequalities, faster rates have been established in Juditsky et al.
Revised: For a different error metric, the restricted gap function, in the special case of variational inequalities, faster rates have been established in Juditsky et al.

Original: (2011) and Böhm et al.
Revised: (2011) and Böhm et al.

Original: However, it is unclear how to relate the restricted gap function to G k , so these rates may not be directly comparable to Theorem 2.
Revised: However, it is unclear how to relate the restricted gap function to G k , so these rates may not be directly comparable to Theorem 2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Arguably the three most popular classes of operator splitting algorithms are forward-backward splitting (FB) (Combettes & Pesquet, 2011), Douglas-Rachford splitting (DR) (Lions & Mercier, 1979), and Tseng’s method (Tseng, 2000).
Revised: Arguably the three most popular classes of operator splitting algorithms are forward-backward splitting (FB) (Combettes & Pesquet, 2011), Douglas-Rachford splitting (DR) (Lions & Mercier, 1979), and Tseng’s method (Tseng, 2000).

Original: The extragradient method (EG) is similar to Tseng’s method, but has more projection steps per iteration and only applies to variational inequalities (Korpelevich, 1977; Nemirovski, 2004).
Revised: The extragradient method (EG) is similar to Tseng’s method, but has more projection steps per iteration and only applies to variational inequalities (Korpelevich, 1977; Nemirovski, 2004; Li et al., 2021).

Original: The popular Alternating Direction Method of Multipliers (ADMM), in its standard form, is a dual application of DR (Gabay, 1983).
Revised: The popular Alternating Direction Method of Multipliers (ADMM), in its standard form, is a dual application of DR (Gabay, 1983).

Original: The three-operator splitting method (Davis & Yin, 2017) can only be applied to (1) if B is cocoercive rather than merely Lipchitz, and thus its usefulness is mostly limited to optimization applications and not games.
Revised: The three-operator splitting method (Davis & Yin, 2017) can only be applied to (1) if B is cocoercive rather than merely Lipchitz, and thus its usefulness is mostly limited to optimization applications and not games.

Original: FB, DR, and Tseng’s method apply to monotone inclusions involving two operators, with varying assumptions on one of the operators.
Revised: FB, DR, and Tseng’s method apply to monotone inclusions involving two operators, with varying assumptions on one of the operators.

Original: It is possible to derive splitting methods for the more complicated inclusion (1), involving more than two operators, by applying an appropriate 2 -operator splitting method such as Tseng’s method to a product-space reformulation (PSR)
Revised: It is possible to derive splitting methods for the more complicated inclusion (1), involving more than two operators, by applying an appropriate 2 -operator splitting method such as Tseng’s method to a product-space reformulation (PSR)

Original: (Briceño-Arias &Combettes, 2011;
Revised: (Briceño-Arias &

Original:  Combettes & Pesquet, 2012) (for more on PSR, see Appendix F).
Revised: Combettes, 2011; Combettes & Pesquet, 2012) (for more on PSR, see Appendix F).

Original: The recently developed forwardreﬂected-backward (FRB) method (Malitsky & Tam, 2020) can be used in the same way.
Revised: The recently developed forward-reﬂected-backward (FRB) method (Malitsky & Tam, 2020) can be used in the same way.

Original: However, there are several disadvantages to using a PSR, as discussed in Appendix F.7.
Revised: However, there are several disadvantages to using a PSR, as discussed in Appendix F.7.

Original: By using a PSR, the stochastic methods of Alacaoglu et al.
Revised: By using a PSR, the stochastic methods of Alacaoglu et al.

Original: and Böhm et al.
Revised: and Böhm et al.

Original: (2020) can be applied to (1) in the case that each A i is a subdifferential.
Revised: (2020) can be applied to (1) in the case that each A i is a subdifferential.

Original: Both of these methods are analyzed in terms of the restricted gap function .
Revised: Both of these methods are analyzed in terms of the restricted gap function .

Original: This merit function has a drawback compared with our approximation residual in that it requires one to ﬁnd a bound for the iterates.
Revised: This merit function has a drawback compared with our approximation residual in that it requires one to ﬁnd a bound for the iterates.

Original: However, Alacaoglu et al.
Revised: However, Alacaoglu et al.

Original: (2021) and
Revised: (2021) and

Original: Böhm et al.
Revised: Böhm et al.

Original: (2020) do not provide such a bound, meaning that their convergence rate results are somewhat incomplete.
Revised: (2020) do not provide such a bound, meaning that their convergence rate results are somewhat incomplete.

Original: We discuss this issue in Appendix G.
Revised: We discuss this issue in Appendix G.

Original: Theoretical convergence of the method of Böhm et al.
Revised: Theoretical convergence of the method of Böhm et al.

Original: (2020) requires the use of averaging, since the ﬁnal iterate does not converge for certain problems (Hsieh et al., 2020).
Revised: (2020) requires the use of averaging, since the ﬁnal iterate does not converge for certain problems (Hsieh et al., 2020).

Original: Empirically, averaging tends to be slow and to destroy regularizer-induced structural properties such as sparsity or low matrix rank, so its utility is largely theoretical and it is usually avoided in practice.
Revised: Empirically, averaging tends to be slow and to destroy regularizer-induced structural properties such as sparsity or low matrix rank, so its utility is largely theoretical and it is usually avoided in practice.

Original: Furthermore, averaging loses even its theoretical beneﬁts for nonconvex problems, so its use in such cases is rarer still.
Revised: Furthermore, averaging loses even its theoretical beneﬁts for nonconvex problems, so its use in such cases is rarer still.

Original: Another drawback of the analysis of Böhm et al.
Revised: Another drawback of the analysis of Böhm et al.

Original: (2020) is that, unlike in SPS, the resolvent (proximal) stepsizes also need to vanish.
Revised: (2020) is that, unlike in SPS, the resolvent (proximal) stepsizes also need to vanish.

Original: The method of Alacaoglu et al.
Revised: The method of Alacaoglu et al.

Original: (2021) applies variance reduction techniques to FRB.
Revised: (2021) applies variance reduction techniques to FRB.

Original: It only applies to ﬁnite-sum problems and requires the periodic computation of a full batch gradient, making it somewhat less ﬂexible and scalable than our method.
Revised: It only applies to ﬁnite-sum problems and requires the periodic computation of a full batch gradient, making it somewhat less ﬂexible and scalable than our method.

Original: On the other hand, it has an accelerated ergodic rate for the restricted gap function in the variational inequality setting.
Revised: On the other hand, it has an accelerated ergodic rate for the restricted gap function in the variational inequality setting.

Original: We compare the empirical performance of SPS with Alacaoglu et al.
Revised: We compare the empirical performance of SPS with Alacaoglu et al.

Original: (2021), Böhm et al.
Revised: (2021), Böhm et al.

Original: (2020), and several deterministic methods using PSR in the numerical experiments described in Section 7.
Revised: (2020), and several deterministic methods using PSR in the numerical experiments described in Section 7.

Original: Additional related work is discussed in Appendix B.
Revised: Additional related work is discussed in Appendix B.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now present some numerical results on distributionally robust supervised learning (DRSL) problems.
Revised: We now present some numerical results on distributionally robust supervised learning (DRSL) problems.

Original: We follow the approach of Yu et al.
Revised: We follow the approach of Yu et al.

Original: (2021), which introduced a min-max formulation of
Revised: (2021), which introduced a min-max formulation of

Original: Wasserstein DRSL.
Revised: Wasserstein DRSL.

Original: While other approaches reduce the problem to convex optimization, Yu et al.
Revised: While other approaches reduce the problem to convex optimization, Yu et al.

Original: (2021) reduce it to a ﬁnite-dimensional min-max problem amenable to the use of stochastic methods on large datasets.
Revised: (2021) reduce it to a ﬁnite-dimensional min-max problem amenable to the use of stochastic methods on large datasets.

Original: However, unlike our proposed SPS method, the variance-reduced extragradient method that Yu et al.
Revised: However, unlike our proposed SPS method, the variance-reduced extragradient method that Yu et al.

Original: (2021) propose cannot handle multiple nonsmooth regularizers or constraints on the model parameters.
Revised: (2021) propose cannot handle multiple nonsmooth regularizers or constraints on the model parameters.

Original: Consequently, we consider distributionally robust sparse logistic regression (DRSLR), a problem class equivalent to that considered in Yu et al.
Revised: Consequently, we consider distributionally robust sparse logistic regression (DRSLR), a problem class equivalent to that considered in Yu et al.

Original: (2021), but with an added (cid:96)regularizer, a standard tool to induce sparsity.
Revised: (2021), but with an added (cid:96)regularizer, a standard tool to induce sparsity.

Original: See the Appendix I for the full problem deﬁnition.
Revised: See the Appendix I for the full problem deﬁnition.

Original: We compared our SPS method to several methods for solving DRSLR for a collection of real datasets from the LIBSVM repository (Chang & Lin, 2011).
Revised: We compared our SPS method to several methods for solving DRSLR for a collection of real datasets from the LIBSVM repository (Chang & Lin, 2011).

Original: We implemented SPS with α k =
Revised: We implemented SPS with α k =

Original: C d k − 0 .
Revised: C d k − 0 .

Original: 51 and ρ k = C
Revised: 51 and ρ k = C

Original: d k − 0 .
Revised: d k − 0 .

Original: 25 and called it SPS-decay .
Revised: 25 and called it SPS-decay .

Original: We also implement SPS with the ﬁxed stepsize given in (15) and called it SPS-ﬁxed .
Revised: We also implement SPS with the ﬁxed stepsize given in (15) and called it SPS-ﬁxed .

Original: We compared the method to deterministic projective splitting (Johnstone &
Revised: We compared the method to deterministic projective splitting (Johnstone &

Original: Eckstein, 2020b) and the following methods based on PSR:
Revised: Eckstein, 2020b) and the following methods based on PSR:

Original: Tseng’s method (Tseng, 2000; Combettes & Pesquet, 2012), the forward-reﬂected-backward (FRB) method (Malitsky & Tam, 2020), the stochastic Tseng (S-Tseng) method of Böhm et al.
Revised: Tseng’s method (Tseng, 2000; Combettes & Pesquet, 2012), the forward-reﬂected-backward (FRB) method (Malitsky & Tam, 2020), the stochastic Tseng (S-Tseng) method of Böhm et al.

Original: (2020), and the variance-reduced stochastic FRB method (Alacaoglu et al., 2021), abbreviated FRB-VR.
Revised: (2020), and the variance-reduced stochastic FRB method (Alacaoglu et al., 2021), abbreviated FRB-VR.

Original: The S-Tseng and FRB-VR algorithms appear to be the only stochastic splitting methods other than SPS applicable to the tested problem class.
Revised: The S-Tseng and FRB-VR algorithms appear to be the only stochastic splitting methods other than SPS applicable to the tested problem class.

Original: Figure 1 show results for three LIBSVM standard datasets: epsilon 2 ( m = 4 · 10 5 , d = 2000 ),
Revised: Figure 1 show results for three LIBSVM standard datasets: epsilon 2 ( m = 4 · 10 5 , d = 2000 ),

Original: SUSY (Baldi et al., 2014; Dua & Graff, 2017) ( m = 2 · 10 6 , d = 18 ), and real-sim 3 ( m = 72 , 309 , d = 20 , 958 ).
Revised: SUSY (Baldi et al., 2014; Dua & Graff, 2017) ( m = 2 · 10 6 , d = 18 ), and real-sim 3 ( m = 72 , 309 , d = 20 , 958 ).

Original: To measure the progress of the algorithms, we used the “approximation residual” R k deﬁned in
Revised: To measure the progress of the algorithms, we used the “approximation residual” R k deﬁned in

Original: Appendix F. As with G k , having R k = 0 implies that z k solves (1).
Revised: Appendix F. As with G k , having R k = 0 implies that z k solves (1).

Original: We use R k instead of G k because it is also possible to compute essentially the same measure of convergence from the iterates of the other tested algorithms, establishing a fair comparison.
Revised: We use R k instead of G k because it is also possible to compute essentially the same measure of convergence from the iterates of the other tested algorithms, establishing a fair comparison.

Original: Appendix F provides the details of the derivation of the residual measure for each algorithm, explores the relationship between R k and G k , and provides additional implementation details.
Revised: Appendix F provides the details of the derivation of the residual measure for each algorithm, explores the relationship between R k and G k , and provides additional implementation details.

Original: Figure 1 plots the approximation residual versus running time for all seven algorithms under consideration.
Revised: Figure 1 plots the approximation residual versus running time for all seven algorithms under consideration.

Original: The computations were performed using Python 3.8.3 and numpy on a 2019 MacBook Pro with a 2.4GHz 8-core Intel I9 processor and 32GB of RAM .
Revised: The computations were performed using Python 3.8.3 and numpy on a 2019 MacBook Pro with a 2.4GHz 8-core Intel I9 processor and 32GB of RAM .

Original: Being a stochastic method, SPS-decay seems to outperform the deterministic methods at obtaining a medium-accuracy solution quickly.
Revised: Being a stochastic method, SPS-decay seems to outperform the deterministic methods at obtaining a medium-accuracy solution quickly.

Original: It also seems to outperform the stochastic PSR-based methods S-Tseng and FRB-VR.
Revised: It also seems to outperform the stochastic PSR-based methods S-Tseng and FRB-VR.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We have developed and analyzed a stochastic splitting method that can handle min-max problems with multiple regularizers and constraints.
Revised: We have developed and analyzed a stochastic splitting method that can handle min-max problems with multiple regularizers and constraints.

Original: Going forward, this development should make it possible to incorporate regularizers and constraints into adversarial formulations trained from large datasets.
Revised: Going forward, this development should make it possible to incorporate regularizers and constraints into adversarial formulations trained from large datasets.

Original: Recent versions of deterministic projective splitting (Combettes & Eckstein, 2018; Johnstone &
Revised: Recent versions of deterministic projective splitting (Combettes & Eckstein, 2018; Johnstone &

Original: Eckstein, 2020b) allow for asynchronous and incremental operation, meaning that not all operators need to be activated at every iteration, with some calculations proceeding with stale inputs.
Revised: Eckstein, 2020b) allow for asynchronous and incremental operation, meaning that not all operators need to be activated at every iteration, with some calculations proceeding with stale inputs.

Original: Such characteristics make projective splitting well-suited to distributed implementations.
Revised: Such characteristics make projective splitting well-suited to distributed implementations.

Original: Many of our SPS results may be extended to allow for these variations, but we leave those extensions to future work.
Revised: Many of our SPS results may be extended to allow for these variations, but we leave those extensions to future work.

Original: A ML A PPLICATIONS OF THE M ONOTONE I NCLUSION (1)
Revised: A ML A PPLICATIONS OF THE M ONOTONE I NCLUSION (1)

Original: There are two main classes of applications of (1) in ML: optimization problems and saddle-point games.
Revised: There are two main classes of applications of (1) in ML: optimization problems and saddle-point games.

Original: Optimization Problems
Revised: Optimization Problems

Original: In this case the monotone inclusion arises from ﬁnding the zero of a sum of subgradients of convex functions, as discussed in Section 2.
Revised: In this case the monotone inclusion arises from ﬁnding the zero of a sum of subgradients of convex functions, as discussed in Section 2.

Original: It is typical in ML to solve the empirical risk minimization problem
Revised: It is typical in ML to solve the empirical risk minimization problem

Original: [Equation]
Revised: [Equation]

Original: (16) over a size- m dataset.
Revised: over a size- m dataset.

Original: Usually, the gradient of the loss function f j for each datapoint j is Lipschitz continuous.
Revised: Usually, the gradient of the loss function f j for each datapoint j is Lipschitz continuous.

Original: The terms r i may be regularizers used to reduce overﬁtting or encourage structural properties such as sparsity or low matrix rank.
Revised: The terms r i may be regularizers used to reduce overﬁtting or encourage structural properties such as sparsity or low matrix rank.

Original: They also may represent constraints on the parameters such as nonnegativity or the being in the probability simplex.
Revised: They also may represent constraints on the parameters such as nonnegativity or the being in the probability simplex.

Original: Crucially, these regularizers are rarely differentiable.
Revised: Crucially, these regularizers are rarely differentiable.

Original: The ﬁrst-order necessary condition for the solution of (16) is
Revised: The ﬁrst-order necessary condition for the solution of (16) is

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) .
Revised: where f ( x ) .

Original: = 1 m (cid:80) mj
Revised: = 1 m (cid:80) mj

Original: =

Revised: =

Original: f j ( x ) , thus ∇ f ( x ) .
Revised: f j ( x ) , thus ∇ f ( x ) .

Original: = 1 m (cid:80) mj
Revised: = 1 m (cid:80) mj

Original: =1 ∇ f j ( x ) .
Revised: =1 ∇ f j ( x ) .

Original: The inclusion (17) is a special case of (1), and our method may use the standard stochastic oracle for ∇ f ( x ) , namely
Revised: The inclusion (17) is a special case of (1), and our method may use the standard stochastic oracle for ∇ f ( x ) , namely

Original: [Equation]
Revised: [Equation]

Original: which subsamples a randomly selected minibatch of datapoints B ∈ { 1 , .
Revised: which subsamples a randomly selected minibatch of datapoints B ∈ { 1 , .

Original: , m } .
Revised: , m } .

Original: Games Consider the following nonsmooth Nash equilibrium problem
Revised: Games Consider the following nonsmooth Nash equilibrium problem

Original: [Equation]
Revised: [Equation]

Original: The terms (cid:80) n 1 i =

Revised: The terms (cid:80) n 1 i =

Original: r
Revised: r

Original: i ( x ) and (cid:80) n 2 i =

Revised: i ( x ) and (cid:80) n 2 i =

Original: d i ( y ) once again represent regularizers and constraints on each player’s strategy.
Revised: d i ( y ) once again represent regularizers and constraints on each player’s strategy.

Original: Note that min-max (saddle-point) problems correspond to having F ( x, y )
Revised: Note that min-max (saddle-point) problems correspond to having F ( x, y )

Original: = − G ( x, y ) .
Revised: = − G ( x, y ) .

Original: Under appropriate convexity conditions and constraint qualiﬁcations, the solutions of (18) correspond to the solutions of the following monotone inclusion in the form of (1): 
Revised: Under appropriate convexity conditions and constraint qualiﬁcations, the solutions of (18) correspond to the solutions of the following monotone inclusion in the form of (1): [Equation]

Original: [Equation]
Revised: [Equation]

Original: where for i > min { n 1 , n 2 } we include “dummy functions", either r i ( x ) = 0 when n 1 < n 2 or d i ( y ) = 0 when n 1 < n 2 .
Revised: where for i > min { n 1 , n 2 } we include “dummy functions", either r i ( x ) = 0 when n 1 < n 2 or d i ( y ) = 0 when n 1 < n 2 .

Original: If the functions F and G arise as averages in the same we as f in (16), then our method may again use a stochastic oracle for them.
Revised: If the functions F and G arise as averages in the same we as f in (16), then our method may again use a stochastic oracle for them.

Original: Distributionally-Robust ML
Revised: Distributionally-Robust ML

Original: One example application of (19) is distributionally-robust ML, as demonstrated in the numerical experiment in Section 7.
Revised: One example application of (19) is distributionally-robust ML, as demonstrated in the numerical experiment in Section 7.

Original: The full problem statement is given in
Revised: The full problem statement is given in

Original: Appendix I.
Revised: Appendix I.

Original: Lagrangian Duality Another application of (19) is constrained optimization via Lagrangian duality.
Revised: Lagrangian Duality Another application of (19) is constrained optimization via Lagrangian duality.

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: As in (16), f is a loss function and the r i may represent regularizers and (“simple”) constraints; in addition, there are p functional constraints on the model parameters x .
Revised: As in (16), f is a loss function and the r i may represent regularizers and (“simple”) constraints; in addition, there are p functional constraints on the model parameters x .

Original: Introducing Lagrange multipliers γ ∈ R p , the problem can be written as
Revised: Introducing Lagrange multipliers γ ∈ R p , the problem can be written as

Original: [Equation]
Revised: [Equation]

Original: Under appropriate convexity conditions and constraint-qualiﬁcations, this reduces to the following inclusion in the form of (1):
Revised: Under appropriate convexity conditions and constraint-qualiﬁcations, this reduces to the following inclusion in the form of (1):

Original: [Equation]
Revised: [Equation]

Original: where h ( x ) =
Revised: where h ( x ) =

Original: [ h 1 ( x ) , h 2 ( x ) , . . . , h p ( x )] (cid:62) .
Revised: [ h 1 ( x ) , h 2 ( x ) , . . . , h p ( x )] (cid:62) .

Original: For certain choices of h , such as linear or quadratic functions, the ﬁrst term above is monotone and (locally)
Revised: For certain choices of h , such as linear or quadratic functions, the ﬁrst term above is monotone and (locally)

Original: Lipschitz continuous (Alacaoglu et al., 2021).
Revised: Lipschitz continuous (Alacaoglu et al., 2021).

Original: Bilinear Games with Many Constraints Finally, consider the bilinear saddlepoint problem subject to multiple constraints:
Revised: Bilinear Games with Many Constraints Finally, consider the bilinear saddlepoint problem subject to multiple constraints:

Original: [Equation]
Revised: [Equation]

Revised: s.t. x ∈ C 1 j j = 1 , . .

Original: {+. , n 1 , y ∈ C 
+}
Revised: {+. , n 1 , y ∈ C 
+}

Revised: j j = 1 , . .

Revised: . , n 2 .

Original: Under some regularity conditions, this problem reduces to the inclusion
Revised: Under some regularity conditions, this problem reduces to the inclusion

Original: [Equation]
Revised: [Equation]

Original: where we introduce additional “dummy” sets C 1 j = R d or C 

Revised: where we introduce additional “dummy” sets C 1 j = R d or C

Original: j = R d when n 1 (cid:54) =
Revised: j = R d when n 1 (cid:54) =

Original: n 2 .
Revised: n 2 .

Original: The ﬁrst term is linear and skew symmetric, and therefore can easily be shown to be Lipschitz continuous and monotone.
Revised: The ﬁrst term is linear and skew symmetric, and therefore can easily be shown to be Lipschitz continuous and monotone.

Original: If all the constraint sets are closed and convex, then the rest of the terms are maximal monotone, then the problem is of the form (1), meaning that projective splitting may be applied, possibly using a stochastic oracle for the ﬁrst term.
Revised: If all the constraint sets are closed and convex, then the rest of the terms are maximal monotone, then the problem is of the form (1), meaning that projective splitting may be applied, possibly using a stochastic oracle for the ﬁrst term.

Original: B A DDITIONAL R ELATED W ORK
Revised: B A DDITIONAL R ELATED W ORK

Original: The preprint by Bot et al.
Revised: The preprint by Bot et al.

Original: (2019) develops a stochastic version of Tseng’s method under the requirement that the noise variance converges to 0 .
Revised: (2019) develops a stochastic version of Tseng’s method under the requirement that the noise variance converges to 0 .

Original: In ML, this could be achieved with the use of perpetually increasing batch sizes, a strategy that is impractical in many scenarios.
Revised: In ML, this could be achieved with the use of perpetually increasing batch sizes, a strategy that is impractical in many scenarios.

Original: The stochastic version of
Revised: The stochastic version of

Original: FRB proposed by Van Dung & Vu (2021) has more practical noise requirements, but has stronger assumptions on the problem which are rarely satisﬁed in ML applications: either uniform/strong monotonicity or a bounded domain.
Revised: FRB proposed by Van Dung & Vu (2021) has more practical noise requirements, but has stronger assumptions on the problem which are rarely satisﬁed in ML applications: either uniform/strong monotonicity or a bounded domain.

Original: The papers by Yurtsever et al.
Revised: The papers by Yurtsever et al.

Original: (2016) and Pedregosa et al.
Revised: (2016) and Pedregosa et al.

Original: (2019) consider stochastic variants of three-operator splitting, but require B in (1) to be cocoercive, essentially restricting them to optimization problems.
Revised: (2019) consider stochastic variants of three-operator splitting, but require B in (1) to be cocoercive, essentially restricting them to optimization problems.

Original: There are several alternatives to the (stochastic) extragradient method that reduce the number of gradient evaluations per iteration from two to one (Hsieh et al., 2019; Malitsky & Tam, 2020; Gidel et al., 2019).
Revised: There are several alternatives to the (stochastic) extragradient method that reduce the number of gradient evaluations per iteration from two to one (Hsieh et al., 2019; Malitsky & Tam, 2020; Gidel et al., 2019).

Original: However, these methods have more stringent stepsize limits, making it unclear a priori whether they will outperform two-step methods.
Revised: However, these methods have more stringent stepsize limits, making it unclear a priori whether they will outperform two-step methods.

Original: DSEG is a stochastic version of EG (Hsieh et al., 2020).
Revised: DSEG is a stochastic version of EG (Hsieh et al., 2020).

Original: The primary innovation of DSEG is using different stepsizes for the extrapolation and update steps, thereby resolving some of the convergence issues affecting stochastic EG.
Revised: The primary innovation of DSEG is using different stepsizes for the extrapolation and update steps, thereby resolving some of the convergence issues affecting stochastic EG.

Original: As noted earlier, DSEG is the special case of our SPS method in which n = 0 , that is, no regularizers/constraints are present in the underlying game.
Revised: As noted earlier, DSEG is the special case of our SPS method in which n = 0 , that is, no regularizers/constraints are present in the underlying game.

Original: The analysis in (Hsieh et al., 2020) also did not consider the ﬁxed stepsize choice given in Theorem 2.
Revised: The analysis in (Hsieh et al., 2020) also did not consider the ﬁxed stepsize choice given in Theorem 2.

Original: In the context of GANs, several methods have been developed based on a variational inequality/monotone inclusion approach (Gidel et al., 2019;
Revised: In the context of GANs, several methods have been developed based on a variational inequality/monotone inclusion approach (Gidel et al., 2019;

Original: Daskalakis et al., 2018; Hsieh et al., 2019; 2020;
Revised: Daskalakis et al., 2018; Hsieh et al., 2019; 2020;

Original: Böhm et al., 2020).
Revised: Böhm et al., 2020).

Original: Many of these papers point out that variational inequalities provide a principled framework for studying the GAN training problem and correcting some of the ﬂaws in the standard method GDA.
Revised: Many of these papers point out that variational inequalities provide a principled framework for studying the GAN training problem and correcting some of the ﬂaws in the standard method GDA.

Original: C P ROOF OF T HEOREMC. S TOCHASTIC Q UASI -F EJER M ONOTONICITY
Revised: C P ROOF OF T HEOREMC. S TOCHASTIC Q UASI -F EJER M ONOTONICITY

Original: The key to the analysis is showing that the algorithm satisﬁes Stochastic Quasi-Fejer Monotonicity (Combettes & Pesquet, 2015).
Revised: The key to the analysis is showing that the algorithm satisﬁes Stochastic Quasi-Fejer Monotonicity (Combettes & Pesquet, 2015).

Original: Lemma 2 ((Combettes & Pesquet, 2015), Proposition 2.3) .
Revised: Lemma 2 ((Combettes & Pesquet, 2015), Proposition 2.3) .

Original: Suppose p k is a sequence of R d -valued random variables deﬁned on a probability space (Ω , F , P ) .
Revised: Suppose p k is a sequence of R d -valued random variables deﬁned on a probability space (Ω , F , P ) .

Original: Let F k
Revised: Let F k

Original: =
Revised: =

Original: σ
Revised: σ

Original: ( p 1 , . .
Revised: ( p 1 , . .

Original: . , p k ) .
Revised: . , p k ) .

Original: Let F be a nonempty, closed subset of R d .
Revised: Let F be a nonempty, closed subset of R d .

Original: Suppose that, for every p ∈ F , there exists χ k ( p ) ≥ 0 , η k ( p ) ≥ 0 , ν k ( p ) ≥ 0 such that (cid:80) ∞
Revised: Suppose that, for every p ∈ F , there exists χ k ( p ) ≥ 0 , η k ( p ) ≥ 0 , ν k ( p ) ≥ 0 such that (cid:80) ∞

Original: k =1 χ k ( p ) < ∞ , (cid:80) ∞
Revised: k =1 χ k ( p ) < ∞ , (cid:80) ∞

Original: k =1 η k ( p ) < ∞ and 
Revised: k =1 η k ( p ) < ∞ and ( ∀ k ∈ N ) E [ (cid:107) p

Original: [Equation]
Revised: k +1 − p (cid:107) 2 |F k ] ≤ (1 + χ k ( p )) (cid:107) p k − p (cid:107) 2 − ν k ( p ) + η k ( p ) .

Original: Then the following hold:
Revised: Then the following hold:

Original: ( ∀ p ∈ F ) : (cid:80) ∞
Revised: ( ∀ p ∈ F ) : (cid:80) ∞

Original: k
Revised: k

Original: =1 ν k ( p ) <
Revised: =1 ν k ( p ) <

Original: ∞ a.s. 2.
Revised: ∞ a.s. 2.

Original: p k is bounded a.s. 3.
Revised: p k is bounded a.s. 3.

Original: There exists ˜Ω such that P [˜Ω] = 1 and (cid:8) (cid:107) p k ( ω ) − p
Revised: There exists ˜Ω such that P [˜Ω] = 1 and (cid:8) (cid:107) p k ( ω ) − p

Original: (cid:107) (cid:9) converges for every ω ∈ ˜Ω and p ∈ F .
Revised: (cid:107) (cid:9) converges for every ω ∈ ˜Ω and p ∈ F .

Original: C. I MPORTANT R ECURSION FOR SPS
Revised: C. I MPORTANT R ECURSION FOR SPS

Original: The following lemma summarizes the key recursion satisﬁed by Algorithm 1, to which we will apply
Revised: The following lemma summarizes the key recursion satisﬁed by Algorithm 1, to which we will apply

Original: Lemma 2.
Revised: Lemma 2.

Original: Recall that L is the Lipschitz constant of B .
Revised: Recall that L is the Lipschitz constant of B .

Original: Lemma 3.
Revised: Lemma 3.

Original: For Algorithm 1, suppose (9) – (11) hold and
Revised: For Algorithm 1, suppose (9) – (11) hold and

Original: [Equation]
Revised: [Equation]

Original: L.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: then for all p ∗ ∈ S , with probability oneE
Revised: then for all p ∗ ∈ S , with probability one

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where C 1 , . .
Revised: where C 1 , . .

Original: , C 4 are nonegative constants deﬁned in (33) , (34) , (48) , and (49) below, respectively.
Revised: , C 4 are nonegative constants deﬁned in (33) , (34) , (48) , and (49) below, respectively.

Original: Note that T k is a scaled version of the approximation residual G k deﬁned in (14).
Revised: Note that T k is a scaled version of the approximation residual G k deﬁned in (14).

Original: We proceed to ﬁrst prove Lemma 3 and then exploit the implications of Lemma 2.
Revised: We proceed to ﬁrst prove Lemma 3 and then exploit the implications of Lemma 2.

Original: Referring to (10) and (11), let N .
Revised: Referring to (10) and (11), let N .

Original: = max j ∈ 1 .. 4 N j .
Revised: = max j ∈ 1 .. 4 N j .

Original: To simplify the constants, we will use N in place of N j for the noise variance bounds given in (10)-(11).
Revised: To simplify the constants, we will use N in place of N j for the noise variance bounds given in (10)-(11).

Original: C. U PPER B OUNDING THE G RADIENT
Revised: C. U PPER B OUNDING THE G RADIENT

Original: Throughout the analysis, we ﬁx some p ∗ =
Revised: Throughout the analysis, we ﬁx some p ∗ =

Original: ( z ∗ , w ∗ 1 . . .
Revised: ( z ∗ , w ∗ 1 . . .

Original: , w ∗ n +1 ) ∈ S .
Revised: , w ∗ n +1 ) ∈ S .

Original: All statements are with probability one (almost surely), but for brevity we will omit this unless it needs to be emphasized.
Revised: All statements are with probability one (almost surely), but for brevity we will omit this unless it needs to be emphasized.

Original: In this section, we derive appropriate upper bounds for (cid:107)∇ ϕ k (cid:107) 2 to use in (13).
Revised: In this section, we derive appropriate upper bounds for (cid:107)∇ ϕ k (cid:107) 2 to use in (13).

Original: We begin with ∇ z ϕ k :
Revised: We begin with ∇ z ϕ k :

Original: [Equation]
Revised: [Equation]

Original: Now next take expectations with respect to F k and E k , and use the bound on the variance of the noise in (11), obtaining
Revised: Now next take expectations with respect to F k and E k , and use the bound on the variance of the noise in (11), obtaining

Original: [Equation]
Revised: [Equation]

Original: where we have used that y ki is F k -measurable for i ∈ 1 ..n .
Revised: where we have used that y ki is F k -measurable for i ∈ 1 ..n .

Original: Thus, taking expectations over E k conditioned on F k yields
Revised: Thus, taking expectations over E k conditioned on F k yields

Original: [Equation]
Revised: [Equation]

Original: We will now bound the two terms on the right side of (22)
Revised: We will now bound the two terms on the right side of (22)

Original: [Equation]
Revised: [Equation]

Original: C

Original: .3. F 
Revised: C.3. F IRST T ERM IN (22)

Original: IRST T ERM IN (22)

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: Now, returning to theﬁrst term on theright of (22), we have
Revised: Now, returning to the term on the of 22), we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we have used (23) to obtain (24).
Revised: where we have used (23) to obtain (24).

Original: For the third term in (24), we have from the calculation on line 7 of the algorithm that
Revised: For the third term in (24) [Equation] we have from the calculation on line 7 of the algorithm that

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: We next take expectations conditioned on F k and use the noise variance bound (10) to obtain
Revised: We next take expectations conditioned on F k and use the noise variance bound (10) to obtain

Original: [Equation]
Revised: [Equation]

Original:  where in the equality uses (23) and w ∗ n +

Revised: Therefore [Equation] (25) where in the equality uses (23) and w ∗ n +

Original: = B ( z ∗ ) .
Revised: = B ( z ∗ ) .

Original: Combining (24) and (25), we arrive at
Revised: Combining (24) and (25), we arrive at

Original: [Equation]
Revised: [Equation]

Original: C.3. S ECOND TERM IN 
Revised: C.3. S ECOND TERM IN (22)

Original: For i ∈ 1 ..n , line 5 of the algorithm may be rearranged into y ki = τ − 1 ( z k − x ki ) + w ki , so [Equation]
Revised: For i ∈ 1 ..n , line 5 of the algorithm may be rearranged into y ki = τ − 1 ( z k − x ki ) + w ki , so (cid:13)(cid:13)(cid:13) n (cid:88)

Original: n
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i =

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) n(cid:88) i =
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of the solution set S in (5), w ∗ i ∈
Revised: By the deﬁnition of the solution set S in (5), w ∗ i ∈

Original: A i ( z ∗ ) , so z ∗ + τw ∗ i ∈
Revised: A i ( z ∗ ) , so z ∗ + τw ∗ i ∈

Original: ( I + τA i )( z ∗ ) , and since the resolvent is single-valued (Bauschke & Combettes, 2017, Cor.
Revised: ( I + τA i )( z ∗ ) , and since the resolvent is single-valued (Bauschke & Combettes, 2017, Cor.

Original: we therefore obtain 
Revised: we therefore obtain z

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From lines 3 and 4 of the algorithm, we also have x ki =
Revised: From lines 3 and 4 of the algorithm, we also have x ki =

Original: J τA i
Revised: J τA i

Original: ( z k + τw ki ) for i ∈ 1 ..n .
Revised: ( z k + τw ki ) for i ∈ 1 ..n .

Original: Thus, using the nonexpansiveness of the resolvent (Bauschke & Combettes, 2017, Def. 4.1 and Cor. 23.9), we haven (cid:88)
Revised: Thus, using the nonexpansiveness of the resolvent (Bauschke & Combettes, 2017, Def. 4.1 and Cor. 23.9), we have

Original: [Equation]
Revised: [Equation]

Revised: Combining (27) and (28) yields

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: Combining (26) and (29) with (22) yields

Original: [Equation]
Revised: [Equation]

Original: C.3. D UAL G RADIENT N ORM
Revised: C.3. D UAL G RADIENT N ORM

Original: Considering that ∇ ϕ k is taken with respect to the subspace P , the gradients with respect to the dual variables are — see for example Eckstein & Svaiter (2009) — for each i ∈ 1 .. ( n + 1) , (cid:107)∇w i ϕ k (cid:107) 2 = (cid:13)(cid:13)(cid:13) x ki − n +n +1 (cid:88)
Revised: Considering that ∇ ϕ k is taken with respect to the subspace P , the gradients with respect to the dual variables are — see for example Eckstein & Svaiter (2009) — for each i ∈ 1 .. ( n + 1) , (cid:107)∇

Original: j = x kj (cid:13)(cid:13)(cid:13) 2 = (cid:13)(cid:13)(cid:13) n +n +1 (cid:88) j = ( x ki − x kj ) (cid:13)(cid:13)(cid:13)≤ n +1 (cid:88)
Revised: [Equation]

Original: j = (cid:107) x
Revised: ϕ [Equation]

Original: ki − x kj (cid:107)≤n +1 (cid:88)
Revised: − n +n [Equation] (cid:88)

Original: j = (cid:0) (cid:107) x
Revised: [Equation]

Original: [-ki − z k (cid:107) 
-]
Revised: [-ki − z k (cid:107) 
-](cid:13)(cid:13)(cid:13) n

Original: + (cid:107)
Revised: [Equation]

Original: z k − x kj (cid:107) 2 (cid:1)
Revised: n [Equation] (cid:88)

Original: Summing this inequality for i ∈ 1 .. ( n + 1) and collecting terms yields n +1 (cid:88)
Revised: [Equation]

Original: i =
Revised: =

Original: (cid:107)∇ w i ϕ k (cid:107) 2 ≤ 4( n + 1) n +1 (cid:88) i = (cid:107) x ki − z k (cid:107) 2 , so taking expectations conditioned on F k produces n +1 (cid:88) i = E [ (cid:107)∇ w i ϕ k (cid:107) 2 | F k ] ≤ 4( n + 1) n +1 (cid:88) i =
Revised: [Equation]

Original: E [ (cid:107) x ki −z k (cid:107) 2 | F k ] ≤ 4( n + 1) E [ (cid:107) x kn +1 − z k (cid:107) 2 | F k ] + 4( n + 1) n (cid:88) i =
Revised: −

Original: E [ (cid:107) x ki − z k (cid:107) 2 | F k ] ≤ 4( n + 1) E
Revised: [Equation]

Original: [ (cid:107) x
Revised: [Equation]

Original: kn +1 − z k (cid:107) 2 | F k ] + 8( n + 1) n (cid:88) i = E [ (cid:107) x ki − z ∗ (cid:107) 2 | F k ] + 8( n + 1) 2 (cid:107) z k − z ∗ (cid:107)≤ 4(
Revised: Summing this inequality for i ∈ 1 .. ( n + 1) and collecting terms yields

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal inequality employs (25) and (28)
Revised: so taking expectations conditioned on F k produces

Original: [Equation]
Revised: [Equation]

Revised: where the ﬁnal inequality employs (25) and (28).

Original: All told, using (30) and (31) and simplifying the constants
Revised: All told, using (30) and (31) and simplifying the constants, one obtains

Original: [Equation]
Revised: [Equation]

Original: one obtains [Equation] [Equation] where
Revised: 32) where

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: 33) and

Original: [Equation]
Revised: [Equation]

Original: C. L OWER B OUND FOR ϕ k - GAP
Revised: C. L OWER B OUND FOR ϕ k - GAP

Original: Recalling (13) [Equation] that is,
Revised: Recalling (13), that is,

Original: [Equation]
Revised: [Equation]

Original: We may use the gradient bound from (32) to obtain
Revised: We may use the gradient bound from (32) to obtain

Revised: E

Original: [Equation]
Revised: [Equation]

Original: We now focus on ﬁnding a lower bound for the term E
Revised: We now focus on ﬁnding a lower bound for the term E

Original: [ ϕ k ( p k ) − ϕ k ( p ∗ ) | F k ] [Equation] which we call the “ ϕ k -gap”.
Revised: [ ϕ k ( p k ) − ϕ k ( p ∗ ) | F k ] , which we call the “ ϕ k -gap”.

Original: Recall that for p =
Revised: Recall that for p =

Original: ( z, w 1 , . . . , w n +1 ) , ϕ k 
Revised: ( z, w 1 , . . . , w n +1 ) , ϕ k ( p ) = n +1 (cid:88)

Original: [Equation]
Revised: i = (cid:104) z − x ki , y ki − w i (cid:105) .

Original: For each i ∈ 1 .. ( n +1) , deﬁne ϕ
Revised: For each i ∈ 1 .. ( n +1) , deﬁne ϕ

Original: i,k ( p ) .
Revised: i,k ( p ) .

Original: = (cid:104)
Revised: = (cid:104)

Original: z − x ki , y ki − w i (cid:105) .
Revised: z − x ki , y ki − w i (cid:105) .

Original: We will call E [ ϕ i,k ( p k )
Revised: We will call E [ ϕ i,k ( p k )

Original: − ϕ i,k ( p ∗ ) | F k ] the “ ϕ i,k -gap”.
Revised: − ϕ i,k ( p ∗ ) | F k ] the “ ϕ i,k -gap”.

Original: Note that ϕ k ( p )
Revised: Note that ϕ k ( p )

Original: = (cid:80)
Revised: = (cid:80)

Original: n +1 i =
Revised: n +1 i =

Original: 1 ϕ i,k ( p ) .
Revised: 1 ϕ i,k ( p ) .

Original: C. L OWER B OUND FOR ϕ i,k - GAP OVER i ∈ 1 ..n
Revised: C. L OWER B OUND FOR ϕ i,k - GAP OVER i ∈ 1 ..n

Original: For i ∈ 1 ..n , we have from line 5 of the algorithm that
Revised: For i ∈ 1 ..n , we have from line 5 of the algorithm that

Original: [Equation]
Revised: [Equation]

Original: Since ϕ i,k ( p k ) = (cid:104)
Revised: Since ϕ i,k ( p k ) = (cid:104)

Original: z k − x ki , y ki − w ki (cid:105) , one may conclude that for i ∈ 1 ..n ,
Revised: z k − x ki , y ki − w ki (cid:105) , one may conclude that for i ∈ 1 ..n ,

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for p ∗ ∈ S and i ∈ 1 ..n , one also has
Revised: On the other hand, for p ∗ ∈ S and i ∈ 1 ..n , one also has

Original: [Equation]
Revised: [Equation]

Original: (36) by the monotonicity of A i .
Revised: 36) by the monotonicity of A i .

Original: Therefore, for i ∈ 1 ..n , it holds that
Revised: Therefore, for i ∈ 1 ..n , it holds that

Original: [Equation]
Revised: [Equation]

Original: and taking expectations conditioned on F k leads to
Revised: and taking expectations conditioned on F k leads to

Original: [Equation]
Revised: [Equation]

Original: (37) where we have used that x k i and y k i are both F k -measurable for i ∈ 1 ..n .
Revised: [Equation] where we have used that x ki and y ki are both F k -measurable for i ∈ 1 ..n .

Original: C
Revised: C. L OWER B OUND FOR ϕ n +1 ,k - GAP

Original: From lines 6-7 of the algorithm, we have
Revised: From lines 6-7 of the algorithm, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where equality (a) uses line 8 of the algorithm and the inequality employs the Cauchy-Schwartz inequality followed by Lipschitz continuity of B
Revised: where equality (a) uses line 8 of the algorithm and the inequality employs the Cauchy-Schwartz inequality followed by Lipschitz continuity of B

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the second equality uses line 8 of the algorithm and the inequality follows from the monotonicity of B .
Revised: where the second equality uses line 8 of the algorithm and the inequality follows from the monotonicity of B .

Original: Combining (39) and (40) yieldsϕ n +1 ,
Revised: Combining (39) and (40) yields

Original: [Equation]
Revised: [Equation]

Original: Now [Equation] if we take expectations conditioned on F k and use (9)
Revised: Now, if we take expectations conditioned on F k and use (9), we obtain

Original: [Equation]
Revised: [Equation]

Original: we obtain

Original: [Equation]

Original: [Equation]

Original: Similarly, (9) also yields
Revised: Similarly, (9) also yields

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, using (42) and (43) and taking expectations of (41) yields 
Revised: Thus, using (42) and (43) and taking expectations of (41) yields E

Original: [Equation]
Revised: [Equation]

Original: where in the second inequality we used (12) and the noise variance bound (10).
Revised: where in the second inequality we used (12) and the noise variance bound (10).

Original: Recall from (12) that 1 − ρL > 0 .
Revised: Recall from (12) that 1 − ρL > 0 .

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we remark that
Revised: we remark that

Original: [Equation]
Revised: [Equation]

Original: Substituting this inequality into (44) yields
Revised: Substituting this inequality into (44) yields

Original: [Equation]
Revised: [Equation]

Original: Finalizing the lower bound on the ϕ k -gap Summing (37) over i ∈ 1 ..n and using (45) yields
Revised: Finalizing the lower bound on the ϕ k -gap Summing (37) over i ∈ 1 ..n and using (45) yields

Original: [Equation]
Revised: [Equation]

Original: C. E STABLISHING S TOCHASTIC Q UASI -F EJER M ONOTONICITY
Revised: C. E STABLISHING S TOCHASTIC Q UASI -F EJER M ONOTONICITY

Original: Returning to (35),
Revised: Returning to (35),

Original: [Equation]
Revised: [Equation]

Original: we may now substitute (46) for the expectation on the right-hand side.
Revised: we may now substitute (46) for the expectation on the right-hand side [Equation]

Original: First, deﬁne
Revised: First, deﬁne

Original: T

Original: [Equation]
Revised: [Equation]

Original: after which we may use (46) in (35) to yield
Revised: after which we may use (46) in (35) to yield

Original: [Equation]
Revised: [Equation]

Original: where C 1 and C 2 are deﬁned as before in (33) and (34) and
Revised: where C 1 and C 2 are deﬁned as before in (33) and (34) and

Original: [Equation]
Revised: [Equation]

Original: This completes the proof of Lemma 3.
Revised: This completes the proof of Lemma 3.

Original: C. A C ONVERGENCE L EMMA
Revised: [Equation]

Original: Before establishing almost-sure convergence, we need the following lemma to derive convergence of the iterates from convergence of T k deﬁned above.
Revised: Before establishing almost-sure convergence, we need the following lemma to derive convergence of the iterates from convergence of T k deﬁned above.

Original: Note that a more elaborate result would be needed in an inﬁnite-dimensional setting.
Revised: Note that a more elaborate result would be needed in an inﬁnite-dimensional setting.

Original: Lemma 4.
Revised: Lemma 4.

Original: For deterministic sequences z k ∈ R ( n +1)
Revised: For deterministic sequences z k ∈ R ( n +1)

Original: d , { ( w ki ) n +1 i =1 } ∈ P , and { ( x ki , y ki )
Revised: d , { ( w ki ) n +1 i =1 } ∈ P , and { ( x ki , y ki )

Original: n +1 i =1 } ∈
Revised: n +1 i =1 } ∈

Original: R 2( n +1) d , suppose
Revised: R 2( n +1) d , suppose

Original: that y ki ∈ A i ( x ki ) for i ∈ 1 ..n , (cid:80) n +1 i=1 w ki = 0 , ξn (cid:88)
Revised: that y ki ∈ A i ( x ki ) for i ∈ 1 ..n , (cid:80) n +1 i

Original: [Equation]
Revised: [Equation]

Original: +
Revised: 1 w ki = 0 ,

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  1 (cid:107) 2 →(50)
Revised: for scalars ξ 1 , ξ 2 , ξ 3 > 0 , and p k . = ( z k , w k 1 , . . . , w kn +1 )

Original: Then p 
Revised: → ˆ p . =

Original:  S 
Revised: (ˆ z, ˆ w 1 , . . . , ˆ w n +1 ) . Then ˆ p ∈ S .

Original: Proof.
Revised: Proof.

Original: Fix any i ∈ { 1 , . . . , n } .
Revised: Fix any i ∈ { 1 , . . . , n } .

Original: Since (cid:107) y ki − w ki (cid:107) → 0 by (50) and w ki → ˆ w i , we also have y ki
Revised: Since (cid:107) y ki − w ki (cid:107) → 0 by (50) and w ki → ˆ w i , we also have y ki

Original: → ˆ w i .
Revised: → ˆ w i .

Original: Similarly, (50) also implies that (cid:107) z k − x ki (cid:107) → 0 , so from z k → ˆ z we also have x ki →
Revised: Similarly, (50) also implies that (cid:107) z k − x ki (cid:107) → 0 , so from z k → ˆ z we also have x ki →

Original: ˆ z .
Revised: ˆ z .

Original: Since y ki ∈ A i ( x ki ) and ( x ki , y ki ) →
Revised: Since y ki ∈ A i ( x ki ) and ( x ki , y ki ) →

Original: (ˆ z, ˆ w i ) , (Bauschke & Combettes, 2017, Prop. 20.37) implies ˆ w i ∈
Revised: (ˆ z, ˆ w i ) , (Bauschke & Combettes, 2017, Prop. 20.37) implies ˆ w i ∈

Original: A i (ˆ z ) .
Revised: A i (ˆ z ) .

Original: Since i was arbitrary, the preceding conclusions hold for i ∈ 1 ..n .
Revised: Since i was arbitrary, the preceding conclusions hold for i ∈ 1 ..n .

Original: Now, (50) also implies that (cid:107) B ( z k ) − w kn +1 (cid:107) → 0 .
Revised: Now, (50) also implies that (cid:107) B ( z k ) − w kn +1 (cid:107) → 0 .

Original: Therefore, since w kn +1 → ˆ w n +1 , we also have
Revised: Therefore, since w kn +1 → ˆ w n +1 , we also have

Original: B ( z k ) →
Revised: B ( z k ) →

Original: ˆ w n +1 .
Revised: ˆ w n +1 .

Original: Much as before, since ( z k , B ( z k ))
Revised: Much as before, since ( z k , B ( z k ))

Original: → (ˆ z, ˆ w n +1 ) , we may apply (Bauschke &
Revised: → (ˆ z, ˆ w n +1 ) , we may apply (Bauschke &

Original: Combettes, 2017, Prop. 20.37) to conclude that that ˆ w n +

Revised: Combettes, 2017, Prop. 20.37) to conclude that that ˆ w n +

Original: =
Revised: =

Original: B
Revised: B

Original: (ˆ z ) .
Revised: (ˆ z ) .

Original: Since the linear subspace P deﬁned in (6) must be closed, the limit (ˆ z, ˆ w 1 , . . .
Revised: Since the linear subspace P deﬁned in (6) must be closed, the limit (ˆ z, ˆ w 1 , . . .

Original: , ˆ w n +1 ) of { ( z k , w k 1 , . . .
Revised: , ˆ w n +1 ) of { ( z k , w k 1 , . . .

Original: , w kn +1 ) } ⊂ P must be in P , hence (cid:80) n +1 i
Revised: , w kn +1 ) } ⊂ P must be in P , hence (cid:80) n +1 i

Original: =

Revised: =

Original: ˆ w i = 0 .
Revised: ˆ w i = 0 .

Original: Thus, the point ˆ p =
Revised: Thus, the point ˆ p =

Original: (ˆ z, ˆ w 1 , . . .
Revised: (ˆ z, ˆ w 1 , . . .

Original: , ˆ w n +1 ) satisﬁes ˆ w i ∈
Revised: , ˆ w n +1 ) satisﬁes ˆ w i ∈

Original: A i (ˆ z ) for i ∈ 1 ..n , ˆ w n +1 =
Revised: A i (ˆ z ) for i ∈ 1 ..n , ˆ w n +1 =

Original: B (ˆ z ) , and (cid:80) n + i =

Revised: B (ˆ z ) , and (cid:80) n + i =

Original: ˆ w i = 0 .
Revised: ˆ w i = 0 .

Original: These are the three conditions deﬁning membership in S from (5), so ˆ p ∈ S .
Revised: These are the three conditions deﬁning membership in S from (5), so ˆ p ∈ S .

Original: C. F INISHING THE P ROOF OF T HEOREM
Revised: C. F INISHING THE P ROOF OF T HEOREM

Original: Given (cid:80) k
Revised: Given (cid:80) k

Original: α 2 k < ∞ , and (cid:80) α k ρ 2 k < ∞ , (47) satisﬁes the conditions of Stochastic Quasi-Fejer
Revised: α 2 k < ∞ , and (cid:80) α k ρ 2 k < ∞ , (47) satisﬁes the conditions of Stochastic Quasi-Fejer

Original: Monotonicity as given in Lemma 2.
Revised: Monotonicity as given in Lemma 2.

Original: By applying Lemma 2, we conclude that there exist Ω 1 , Ω 2 , Ωsuch that P
Revised: By applying Lemma 2, we conclude that there exist Ω 1 , Ω 2 , Ωsuch that P

Original: [Ω i ] = 1 for i = 1 , 2 , 3 and 
Revised: [Ω i ] = 1 for i = 1 , 2 , 3 and 1.

Original: for all v ∈ Ω
Revised: for all v ∈ Ω

Original: [Equation]
Revised: [Equation]

Original: for all v ∈ Ω 2 , and p ∗ ∈ S , (cid:107) p k ( v ) − p ∗ (cid:107) converges to a ﬁnite nonnegative random-variable, 3.
Revised: for all v ∈ Ω 2 , and p ∗ ∈ S , (cid:107) p k ( v ) − p ∗ (cid:107) converges to a ﬁnite nonnegative random-variable, 3.

Original: for all v ∈ Ω 3 , p k ( v ) remains bounded.
Revised: for all v ∈ Ω 3 , p k ( v ) remains bounded.

Original: Since (cid:80) ∞ k =1 α k ρ k = ∞ , (51) implies that for all v ∈ Ω 1 there exists a subsequence q k ( v ) such that 
Revised: Since (cid:80) ∞ k =1 α k ρ k = ∞ , (51) implies that for all v ∈ Ω 1 there exists a subsequence q k ( v ) such that [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let Ω (cid:48)
Revised: Let Ω (cid:48)

Original: =
Revised: =

Original: Ω 1 ∩ Ω 2 ∩ Ω 3 and note that P [Ω (cid:48) ] = 1 .
Revised: Ω 1 ∩ Ω 2 ∩ Ω 3 and note that P [Ω (cid:48) ] = 1 .

Original: Choose v ∈ Ω (cid:48) .
Revised: Choose v ∈ Ω (cid:48) .

Original: Since p k ( v ) remains bounded, so does p q k ( v ) ( v ) for q k ( v ) deﬁned above in (52).
Revised: Since p k ( v ) remains bounded, so does p q k ( v ) ( v ) for q k ( v ) deﬁned above in (52).

Original: Thus there exists a subsequence r k ( v ) ⊆
Revised: Thus there exists a subsequence r k ( v ) ⊆

Original: q k ( v ) and ˆ p ( v ) ∈ R ( n +2)
Revised: q k ( v ) and ˆ p ( v ) ∈ R ( n +2)

Original: d such that p r k ( v ) ( v ) → ˆ p ( v ) .
Revised: d such that p r k ( v ) ( v ) → ˆ p ( v ) .

Original: But since T q k ( v ) → 0 , it also follows that T r k ( v ) → 0 , that is,
Revised: But since T q k ( v ) → 0 , it also follows that T r k ( v ) → 0 , that is,

Original: [Equation]
Revised: [Equation]

Original: We then have from Lemma 4 that ˆ p ( v ) ∈ S .
Revised: We then have from Lemma 4 that ˆ p ( v ) ∈ S .

Original: Since p r k ( v ) ( v ) → ˆ p ( v ) , it follows that (cid:107) p r k ( v ) ( v ) − ˆ p ( v ) (cid:107) → 0 .
Revised: Since p r k ( v ) ( v ) → ˆ p ( v ) , it follows that (cid:107) p r k ( v ) ( v ) − ˆ p ( v ) (cid:107) → 0 .

Original: But since ˆ p ( v ) ∈ S , (cid:107) p k ( v ) − ˆ p ( v ) (cid:107) converges by point 2 above.
Revised: But since ˆ p ( v ) ∈ S , (cid:107) p k ( v ) − ˆ p ( v ) (cid:107) converges by point 2 above.

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Therefore p k ( v ) → ˆ p ( v ) ∈ S .
Revised: Therefore p k ( v ) → ˆ p ( v ) ∈ S .

Original: Thus there exists ˆ p ∈ S such that p k
Revised: Thus there exists ˆ p ∈ S such that p k

Original: → ˆ p a.s., which completes the proof of Theorem 1.
Revised: → ˆ p a.s., which completes the proof of Theorem 1.

Revised: Finally, note that it also holds that x ki → ˆ z (a.s.) for i = 1 , . . .

Revised: , n where ˆ p =

Revised: (ˆ z, ˆ w 1 , . . .

Revised: , ˆ w n +1 ) .

Revised: To see this, note that

Original: D P ROOF OF L EMMA
Revised: [Equation] and since z k and w ki convergence a.s., so does x ki . Consider the subsequence q k ( v ) such that (52) holds. Then z

Revised: [Equation]

Revised: thus x q k ( v )

Revised: [Equation]

Revised: → ˆ z.

Revised: Since x ki converges to some limit (a.s.), that limit must be ˆ z .

Revised: D P ROOF OF L EMMA

Original: If G k = 0 , then ∀ i = 1 , . . .
Revised: If G k = 0 , then ∀ i = 1 , . . .

Original: , n : y k i = w k i and z k = x k i .
Revised: , n : y ki = w ki and z k = x ki .

Original: Since y ki ∈ A i ( x ki ) for i = 1 , . . . , n , (53) implies that that ∀ i ∈ 1 ..n : w ki ∈
Revised: Since y ki ∈ A i ( x ki ) for i = 1 , . . . , n , (53) implies that that ∀ i ∈ 1 ..n : w [Equation]

Original: A i ( z k ) .
Revised: A i ( z k ) .

Original: Furthermore G k = 0 also implies that w kn +

Revised: Furthermore G k = 0 also implies that w kn +

Original: =
Revised: =

Original: B ( z k ) .
Revised: B ( z k ) .

Original: Finally, since (cid:80) n +1 i =1 w ki = 0 , we have that ( z k , w k 1 , . . .
Revised: Finally, since (cid:80) n +1 i =1 w ki = 0 , we have that ( z k , w k 1 , . . .

Original: , w kn +1 ) ∈ S .
Revised: , w kn +1 ) ∈ S .

Original: Conversely, suppose ( z k , w k 1 , . . .
Revised: Conversely, suppose ( z k , w k 1 , . . .

Original: , w kn +1 ) ∈ S .
Revised: , w kn +1 ) ∈ S .

Original: The deﬁnition of S implies that B ( z k ) =
Revised: The deﬁnition of S implies that B ( z k ) =

Original: w kn +1 and
Revised: w kn +1 and

Original: furthermore that w ki ∈
Revised: furthermore that w ki ∈

Original: A i ( z k ) for i ∈ 1 ..n .
Revised: A i ( z k ) for i ∈ 1 ..n .

Original: For any i ∈ 1 ..n , considering line 3 of Algorithm 1, we may write t k i =
Revised: For any i ∈ 1 ..n , considering line 3 of Algorithm 1, we may write t ki =

Original: z k + τw ki ∈
Revised: z k + τw ki ∈

Original: ( I + τA i )( z k ) , implying z k ∈ ( I + τA i ) − 1 ( t k i ) .
Revised: ( I + τA i )( z k ) , implying z k ∈ ( I + τA i ) − 1 ( t ki ) .

Original: But since the resolvent J τA i =
Revised: But since the resolvent J τA i =

Original: ( I + τA i ) − 1 is single-valued (Bauschke & Combettes, 2017, Prop. 23.8), we must have z k
Revised: ( I + τA i ) − 1 is single-valued (Bauschke & Combettes, 2017, Prop. 23.8), we must have z k

Original: = ( I + τA i ) − 1 ( t ki ) .
Revised: = ( I + τA i ) − 1 ( t ki ) .

Original: Thus, by line 4, we have x ki =
Revised: Thus, by line 4, we have x ki =

Original: z k .
Revised: z k .

Original: We may also derive from line 5 that
Revised: We may also derive from line 5 that

Original: [Equation]
Revised: [Equation]

Original: Thus, since x k i = z k and y k i = w k i for i = 1 , . . . , n and w  k[-n +
-] = 
Revised: Thus, since x ki = z k and y ki = w ki for i = 1 , . . . , n and w kn +1 = B ( z k ) , we have that G k[-n +
-] = 0 .

Original: B

Original: ( z k ) , we have that G k = 0 .

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: In addition to the proof [Equation] we provide a more detailed statement of the theorem:
Revised: In addition to the proof, we provide a more detailed statement of the theorem:

Original: Theorem 3.
Revised: Theorem 3.

Original: Fix the total iterations K ≥ 1 of Algorithm 1 and set
Revised: Fix the total iterations K ≥ 1 of Algorithm 1 and set

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: for some C f > 0 .
Revised: for some C f > 0 .

Original: Suppose (9) - (11) hold.
Revised: Suppose (9) - (11) hold.

Original: Then for any p ∗ ∈ S , 
Revised: Then for any p ∗ ∈ S , K

Original: [Equation]
Revised: [Equation]

Original: where G k is the approximation residual deﬁned in (14)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  and C 1 , C 2 , C 3 , C 4 are the nonegative constants deﬁned in (33) , (34) , (48) [Equation] and (49) , respectively.
Revised: where G k is the approximation residual deﬁned in (14) , and C 1 , C 2 , C 3 , C 4 are the nonegative constants deﬁned in (33) , (34) , (48) , and (49) , respectively.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix α k
Revised: Fix α k

Original: = α and ρ k
Revised: = α and ρ k

Original: = ρ , where α and ρ are the respective right-hand sides of (55)-(56).
Revised: = ρ , where α and ρ are the respective right-hand sides of (55)-(56).

Original: Lemma 3 implies that (21) so long as (9)-(11) hold and the stepsize ρ satisﬁes
Revised: Lemma 3 implies that (21) so long as (9)-(11) hold and the stepsize ρ satisﬁes

Original: ρ < L − 1 .
Revised: ρ < L − 1 .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we conclude that (21) applies.
Revised: we conclude that (21) applies.

Original: Rewriting (21) with α k = α and ρ k = ρ , we have
Revised: Rewriting (21) with α k = α and ρ k = ρ , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, taking expectations over F k [Equation] we have
Revised: Therefore, taking expectations over F k , we have

Original: [Equation]
Revised: [Equation]

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: where for the ﬁrst two terms we have simply set ρ = ρ because the stepsize is constant.
Revised: where for the ﬁrst two terms we have simply set ρ = ρ because the stepsize is constant.

Original: However, for the ﬁnal term [Equation] we will still use an upper bound, ρ , on ρ .
Revised: However, for the ﬁnal term, we will still use an upper bound, ρ , on ρ .

Original: In the current setting [Equation] we know that ρ ≤ (1 / 2) L − 1 and therefore we may set ρ = 
Revised: In the current setting, we know that ρ ≤ (1 / 2) L − 1 and therefore we may set ρ = (1 / 2)

Original: L − 1 .
Revised: L − 1 .

Original: Thus 1 − ρL = 1 / 2 , leading to
Revised: Thus 1 − ρL = 1 / 2 , leading to

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:107) 2 .

Original: Let
Revised: Let

Original: U k .

Original: =

Original: E (cid:107) B

Original: [Equation]
Revised: [Equation]

Original: z k ) − w kn +

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: τ n [Equation] i = E
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: ki [Equation] [Equation] τ
Revised: and also let

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i = E [Equation] z k − x ki (cid:107)
Revised: Using these deﬁnitions in (59) we write

Original: [Equation]
Revised: [Equation]

Original: , so that [Equation]
Revised: Therefore,

Original: , and also let
Revised: V

Original: [Equation]
Revised: [Equation]

Original: Using these deﬁnitions in (59) we write
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Therefore,{+⇐⇒ V k +
+}
Revised: {+⇐⇒ V k +
+}

Original: [Equation]
Revised: + αρ k (cid:88) j = U j + α k (cid:88) j = W j ≤ (1 + C 1 α 2 + C 3 αρ 2 )

Revised: V k + αρ k − 1 (cid:88)

Revised: j =

Original: where we have used that U  k {+− 1 (cid:88) j = W j + C 
 α 2 + C 4 αρ+}
Revised: U j + α k {+− 1 (cid:88) j = W j + C 
 α 2 + C 4 αρ+}

Original: [Equation]
Revised: [Equation]

Original:  W k ≥ 0 .
Revised: where we have used that U k , W k ≥ 0 .

Original: Letting
Revised: Letting

Original: R k

Original: [Equation]
Revised: [Equation]

Original: we then have
Revised: we then have

Original: [Equation]
Revised: [Equation]

Original: , which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Now,
Revised: Now,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Fix the number of iterations K ≥ 
Revised: Fix the number of iterations K ≥ 1 .

Revised: Now

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: min

Original: [Equation]
Revised: [Equation]

Original: Therefore, 
Revised: Therefore, αρ

Original: [Equation]
Revised: [Equation]

Original: Dividing through by αρK , we obtain
Revised: Dividing through by αρK , we obtain

Original: [Equation]
Revised: [Equation]

Original: and since α = C f ρ 2 , we also have
Revised: and since α = C f ρ 2 , we also have

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: Substituting these into (60) yields

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where we have used that for any t ≥ 0 , 1 + t/K ≤ e t/K , so therefore (1 + t/K ) K ≤ e t .
Revised: where we have used that for any t ≥ 0 , 1 + t/K ≤ e t/K , so therefore (1 + t/K ) K ≤ e t .

Original: The worst-case rates in terms of K occur when ρ = K − 1 / 4 and α = C f K − 1 / 2 .
Revised: The worst-case rates in terms of K occur when ρ = K − 1 / 4 and α = C f K − 1 / 2 .

Original: This is the case when K ≥ (2 L ) 4 .
Revised: This is the case when K ≥ (2 L ) 4 .

Original: Substituting these into the denominator yields, for K ≥ (2 L ) 4 , that
Revised: Substituting these into the denominator yields, for K ≥ (2 L ) 4 , that

Original: [Equation]
Revised: [Equation]

Original: Thus, since G k ≤ max { τ, τ − 1 } ( U k + W k ) , we obtain 
Revised: Thus, since G k ≤ max { τ, τ − 1 } ( U k + W k ) , we obtain K

Original: [Equation]
Revised: [Equation]

Original: which is (58).
Revised: which is (58).

Original: When K < (2 L ) 4 , (57) can similarly be obtained by substituting ρ = (2 L ) − 1 and α = C f (2 L ) −into (61).
Revised: When K < (2 L ) 4 , (57) can similarly be obtained by substituting ρ = (2 L ) − 1 and α = C f (2 L ) −into (61).

Original: F A PPROXIMATION R ESIDUALS
Revised: F A PPROXIMATION R ESIDUALS

Original: In this section we derive the approximation residual used to assess the performance of the algorithms in the numerical experiments.
Revised: In this section we derive the approximation residual used to assess the performance of the algorithms in the numerical experiments.

Original: This residual relies on the following product-space reformulation of (1).
Revised: This residual relies on the following product-space reformulation of (1).

Original: F. P RODUCT -S PACE R EFORMULATION AND R ESIDUAL P RINCIPLE
Revised: F. P RODUCT -S PACE R EFORMULATION AND R ESIDUAL P RINCIPLE

Original: Recall (1), the monotone inclusion we are solving:
Revised: Recall (1), the monotone inclusion we are solving:

Original: [Equation]
Revised: [Equation]

Original: In this section we demonstrate a “product-space" reformulation of (1) which allows us to rewrite it in a standard form involving just two operators, one maximal monotone and the other monotone and Lipschitz.
Revised: In this section we demonstrate a “product-space" reformulation of (1) which allows us to rewrite it in a standard form involving just two operators, one maximal monotone and the other monotone and Lipschitz.

Original: This approach was pioneered in (Briceño-Arias & Combettes, 2011; Combettes &
Revised: This approach was pioneered in (Briceño-Arias & Combettes, 2011; Combettes &

Original: Pesquet, 2012).
Revised: Pesquet, 2012).

Original: Along with allowing for a simple deﬁnition of an approximation residual as a measure of approximation error in solving (1), it allows one to apply operator splitting methods originally formulated for two operators to problems such as (1) for any ﬁnite n .
Revised: Along with allowing for a simple deﬁnition of an approximation residual as a measure of approximation error in solving (1), it allows one to apply operator splitting methods originally formulated for two operators to problems such as (1) for any ﬁnite n .

Original: Observe that solving (1) is equivalent to
Revised: Observe that solving (1) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: This formulation resembles that of the extended solution set S used in projective spitting, as given in (5), except that it combines the ﬁnal two conditions in the deﬁnition of S , and thus does not need the ﬁnal dual variable w n +1 .
Revised: This formulation resembles that of the extended solution set S used in projective spitting, as given in (5), except that it combines the ﬁnal two conditions in the deﬁnition of S , and thus does not need the ﬁnal dual variable w n +1 .

Original: From the deﬁnition of the inverse of an operator, the above formulation is equivalent to
Revised: From the deﬁnition of the inverse of an operator, the above formulation is equivalent to

Original: [Equation]
Revised: [Equation]

Original: These conditions are in turn equivalent to ﬁnding ( w 1 , . . .
Revised: These conditions are in turn equivalent to ﬁnding ( w 1 , . . .

Original: , w n , z ) ∈ R ( n +1) d such that
Revised: , w n , z ) ∈ R ( n +1) d such that

Original: [Equation]
Revised: [Equation]

Original: where A is the set-valued map
Revised: where A is the set-valued map

Original: [Equation]
Revised: [Equation]

Original: and B is the single-valued operator
Revised: and B is the single-valued operator

Original: [Equation]
Revised: [Equation]

Original: It is easily established that B is maximal monotone and Lipschitz continuous, while A is maximal monotone.
Revised: It is easily established that B is maximal monotone and Lipschitz continuous, while A is maximal monotone.

Original: Letting T .
Revised: Letting T .

Original: =
Revised: =

Original: A + B , it follows from (Bauschke & Combettes, 2017, Proposition 20.23) that T is maximal monotone.
Revised: A + B , it follows from (Bauschke & Combettes, 2017, Proposition 20.23) that T is maximal monotone.

Original: Thus, we have reformulated (1) as the monotone inclusion 0 ∈ T ( q ) for q in the product space R ( n +1) d .
Revised: Thus, we have reformulated (1) as the monotone inclusion 0 ∈ T ( q ) for q in the product space R ( n +1) d .

Original: A vector z ∈ R d solves (1) if and only if there exists ( w 1 , . . . , w n ) ∈ R nd such that 0 ∈ T ( q ) , where q = ( w 1 , . . . , w n , z ) .
Revised: A vector z ∈ R d solves (1) if and only if there exists ( w 1 , . . . , w n ) ∈ R nd such that 0 ∈ T ( q ) , where q = ( w 1 , . . . , w n , z ) .

Original: For any pair ( q, v ) such that v ∈ T ( q ) , (cid:107) v (cid:107) 2 represents an approximation residual for q in the sense that v = 0 implies q is a solution to (62).
Revised: For any pair ( q, v ) such that v ∈ T ( q ) , (cid:107) v (cid:107) 2 represents an approximation residual for q in the sense that v = 0 implies q is a solution to (62).

Original: One may take (cid:107) v (cid:107) 2 as a measure of the error of q as an approximate solution to (62), and it can only be 0 if q is a solution.
Revised: One may take (cid:107) v (cid:107) 2 as a measure of the error of q as an approximate solution to (62), and it can only be 0 if q is a solution.

Original: Given two approximate solutions q 1 and q 2 with certiﬁcates v 1 ∈ T ( q 1 ) and v 2 ∈ T ( q 2 ) , we will treat q 1 as a “better” approximate solution than q 2 if (cid:107) v 1 (cid:107) 2 < (cid:107) v 2 (cid:107) 2 .
Revised: Given two approximate solutions q 1 and q 2 with certiﬁcates v 1 ∈ T ( q 1 ) and v 2 ∈ T ( q 2 ) , we will treat q 1 as a “better” approximate solution than q 2 if (cid:107) v 1 (cid:107) 2 < (cid:107) v 2 (cid:107) 2 .

Original: Doing so is somewhat analogous to the practice, common in optimization, of using the gradient (cid:107)∇ f ( x ) (cid:107) 2 as a measure of quality of an approximate minimizer of some differentiable function f .
Revised: Doing so is somewhat analogous to the practice, common in optimization, of using the gradient (cid:107)∇ f ( x ) (cid:107) 2 as a measure of quality of an approximate minimizer of some differentiable function f .

Original: However, note that since T ( q 1 ) is a set, there may exist elements of T ( q 1 ) with smaller norm than v 1 .
Revised: However, note that since T ( q 1 ) is a set, there may exist elements of T ( q 1 ) with smaller norm than v 1 .

Original: Thus any given certiﬁcate v 1 only corresponds to an upper bound on dist 2 (0 , T ( q 1 )) .
Revised: Thus any given certiﬁcate v 1 only corresponds to an upper bound on dist 2 (0 , T ( q 1 )) .

Original: F. A PPROXIMATION R ESIDUAL FOR P ROJECTIVE S PLITTING
Revised: F. A PPROXIMATION R ESIDUAL FOR P ROJECTIVE S PLITTING

Original: In SPS (Algorithm 1), for i ∈ 1 ..n , the pairs ( x ki , y ki ) are chosen so that y ki ∈
Revised: In SPS (Algorithm 1), for i ∈ 1 ..n , the pairs ( x ki , y ki ) are chosen so that y ki ∈

Original: A i ( x ki ) .
Revised: A i ( x ki ) .

Original: This can be seen from the deﬁnition of the resolvent.
Revised: This can be seen from the deﬁnition of the resolvent.

Original: Thus x ki ∈ A − 1 i ( y ki ) .
Revised: Thus x ki ∈ A − 1 i ( y ki ) .

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: The approximation residual for SPS is thus
Revised: The approximation residual for SPS is thus

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: which is an approximation residual for ( y k 1 , . .
Revised: which is an approximation residual for ( y k 1 , . .

Original: , y kn , z k ) in the sense deﬁned above.
Revised: , y kn , z k ) in the sense deﬁned above.

Original: We may relate
Revised: We may relate

Original: R k to the approximation residual G k for SPS from Section 5 as follows:
Revised: R k to the approximation residual G k for SPS from Section 5 as follows:

Original: [Equation]
Revised: [Equation]

Original: nG k where in the second equality we have used the fact that (cid:80) n +1 i =1 w ki = 0 .
Revised: where in the second equality we have used the fact that (cid:80) n +1 i =1 w ki = 0 .

Original: Thus, R k has the same convergence rate as Gk given in Theorem 2.
Revised: Thus, R k has the same convergence rate as G

Original: F. T SENG ’ S METHOD
Revised: F. T SENG ’ S METHOD

Original: Tseng’s method (Tseng, 2000) can be applied to (62), resulting in the following recursion with iterates q k , 
Revised: Tseng’s method (Tseng, 2000) can be applied to (62), resulting in the following recursion with iterates q k , ¯ q k ∈ R

Original: [Equation]
Revised: [Equation]

Original: n +
Revised: n +1)

Revised: d :

Original: [Equation]
Revised: [Equation]

Original: where A and B are deﬁned in (63) and (64).
Revised: where A and B are deﬁned in (63) and (64).

Original: The resolvent of A may be readily computed from the resolvents of the A i using Moreau’s identity (Bauschke & Combettes, 2017, Proposition 23.20).
Revised: The resolvent of A may be readily computed from the resolvents of the A i using Moreau’s identity (Bauschke & Combettes, 2017, Proposition 23.20).

Original: Analogous to SPS, Tseng’s method has an approximation residual, which in this case is an element of T (¯ q k ) .
Revised: Analogous to SPS, Tseng’s method has an approximation residual, which in this case is an element of T (¯ q k ) .

Original: In particular, using the general properties of resolvent operators as applied to J α A , we have
Revised: In particular, using the general properties of resolvent operators as applied to J α A , we have

Original: [Equation]
Revised: [Equation]

Original: Also, rearranging (67) produces
Revised: Also, rearranging (67) produces

Original: [Equation]
Revised: [Equation]

Original: Adding these two relations produces
Revised: Adding these two relations produces

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: represents a measure of the approximation error for Tseng’s method equivalent to R k deﬁned in (65) for SPS.
Revised: represents a measure of the approximation error for Tseng’s method equivalent to R k deﬁned in (65) for SPS.

Original: F. FRB
Revised: F. FRB

Original: The forward-reﬂected-backward method (FRB) (Malitsky & Tam, 2020) is another method that may be applied to the splitting T =
Revised: The forward-reﬂected-backward method (FRB) (Malitsky & Tam, 2020) is another method that may be applied to the splitting T =

Original: A + B for A and B as deﬁned in (63) and (64).
Revised: A + B for A and B as deﬁned in (63) and (64).

Original: Doing so yields recursion
Revised: Doing so yields recursion

Original: [Equation]
Revised: [Equation]

Original: Following similar arguments to those for Tseng’s method, it can be shown that
Revised: Following similar arguments to those for Tseng’s method, it can be shown that

Original: [Equation]
Revised: [Equation]

Original: Thus, FRB admits the following approximation residual equivalent to R k for SPS:
Revised: Thus, FRB admits the following approximation residual equivalent to R k for SPS:

Original: [Equation]
Revised: [Equation]

Original: Finally, we remark that the stepsizes used in both the Tseng and FRB methods can be chosen via a linesearch procedure that we do not detail here.
Revised: Finally, we remark that the stepsizes used in both the Tseng and FRB methods can be chosen via a linesearch procedure that we do not detail here.

Original: F. S TOCHASTIC T SENG M ETHOD
Revised: F. S TOCHASTIC T SENG M ETHOD

Original: The stochastic version of Tseng’s method of (Böhm et al., 2020) (S-Tseng) may be applied to the inclusion 0 ∈ A ( q ) + B ( q ) , since the operator A may be written as a subdifferential.
Revised: The stochastic version of Tseng’s method of (Böhm et al., 2020) (S-Tseng) may be applied to the inclusion 0 ∈ A ( q ) + B ( q ) , since the operator A may be written as a subdifferential.

Original: However, unlike the deterministic Tseng method, it does not produce a valid residual.
Revised: However, unlike the deterministic Tseng method, it does not produce a valid residual.

Original: Note also that S-Tseng outputs an ergodic sequence q k erg .
Revised: Note also that S-Tseng outputs an ergodic sequence q k erg .

Original: To construct a residual for the ergodic sequence, we compute a deterministic step of Tseng’s method according to (66)-(67), starting at q k erg .
Revised: To construct a residual for the ergodic sequence, we compute a deterministic step of Tseng’s method according to (66)-(67), starting at q k erg .

Original: That is, letting
Revised: That is, letting

Original: [Equation]
Revised: [Equation]

Original: we can then compute essentially the same residual as in Section F.3,
Revised: we can then compute essentially the same residual as in Section F.3,

Original: [Equation]
Revised: [Equation]

Original:  To construct the stochastic oracle for S-Tseng, we assumed B ( z ) = 1 m (cid:80) mi =1 B i ( z ) .
Revised: S-Tseng [Equation] To construct the stochastic oracle for S-Tseng, we assumed B ( z ) = 1 m (cid:80) mi =1 B i ( z ) .

Original: Then we used ˜ B ( w 1 , . . .
Revised: Then we used ˜ B ( w 1 , . . .

Original: , w n , z ) (cid:55)→
Revised: , w n , z ) (cid:55)→

Original:  · · · − I ... · · · − I I · · · I     w 1 .
Revised: [Equation]

Original: w n z   +   ...

Original: 0| B | (cid:80) j ∈ B B j ( z )

Original:   .

Original: (68) for some minibatch B ∈ { 1 , . . .
Revised: for some minibatch B ∈ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: F. V ARIANCE -R EDUCED FRB
Revised: F. V ARIANCE -R EDUCED FRB

Original: The FRB-VR method of Alacaoglu et al.
Revised: The FRB-VR method of Alacaoglu et al.

Original: (2021) can also be applied to 0 ∈ A ( q ) + B ( q ) , using the same stochastic oracle ˜ B deﬁned in (68).
Revised: (2021) can also be applied to 0 ∈ A ( q ) + B ( q ) , using the same stochastic oracle ˜ B deﬁned in (68).

Original: if we let the iterates of FRB-VR be ( q k , p k ) , then line 4 of
Revised: if we let the iterates of FRB-VR be ( q k , p k ) , then line 4 of

Original: Algorithm 1 of Alacaoglu et al.
Revised: Algorithm 1 of Alacaoglu et al.

Original: (2021) can be written as
Revised: (2021) can be written as

Original: [Equation]
Revised: [Equation]

Original: Once again, the method does not directly produce a residual, but one can be developed from the algorithm deﬁnition as follows: (70) yields τ − 1 (ˆ q k − q k +1 )
Revised: Once again, the method does not directly produce a residual, but one can be developed from the algorithm deﬁnition as follows: (70) yields τ − 1 (ˆ q k − q k +1 )

Original: ∈
Revised: ∈

Original: A ( q k +1 ) and hence
Revised: A ( q k +1 ) and hence

Original: [Equation]
Revised: [Equation]

Original: Therefore we use the residual
Revised: Therefore we use the residual

Original: [Equation]
Revised: [Equation]

Original: Figure 1 plots R k for SPS, R Tseng k for Tseng’s method, R FRB k for FRB, R S-Tseng k for S-Tseng, and

Original: R FRB-VR k for FRB-VR.

Original: F. B ENEFITS AND D RAWBACKS OF THE P RODUCT S PACE R EFORMULATION

Original: The main beneﬁt of the product space reformulation (PSR) is that it allows one to use familiar 2-operator splitting schemes for solving 0 ∈ A ( q ) + B ( q ) to solve the more complicated recursion (1).
Revised: The main beneﬁt of the product space reformulation (PSR) is that it allows one to use familiar 2-operator splitting schemes for solving 0 ∈ A ( q ) + B ( q ) to solve the more complicated recursion (1).

Original: However, one drawback of this approach is that the operator B , deﬁned in (64), combines a skew-symmetric consensus matrix with the Lipschitz operator B .
Revised: However, one drawback of this approach is that the operator B , deﬁned in (64), combines a skew-symmetric consensus matrix with the Lipschitz operator B .

Original: Treating B as a single operator necessitates using a single stepsize for both of its constituent operators, but the B component will generally have a much larger Lipschitz constant than the skew part, necessitating a smaller stepsize than is ideal for the skew operator.
Revised: Treating B as a single operator necessitates using a single stepsize for both of its constituent operators, but the B component will generally have a much larger Lipschitz constant than the skew part, necessitating a smaller stepsize than is ideal for the skew operator.

Original: This difﬁculty can be countered by using different stepsizes for the primal and dual components, but that strategy introduces additional tuning parameters.
Revised: This difﬁculty can be countered by using different stepsizes for the primal and dual components, but that strategy introduces additional tuning parameters.

Original: G V ARIATIONAL I NEQUALITIES
Revised: G V ARIATIONAL I NEQUALITIES

Original: For a mapping B : R d → R d and a closed and convex set C , the variational inequality problem (Harker & Pang, 1990) is to ﬁnd z ∗ ∈ C such that 
Revised: For a mapping B : R d → R d and a closed and convex set C , the variational inequality problem (Harker & Pang, 1990) is to ﬁnd z ∗ ∈ C such that [Equation]

Original: [Equation]
Revised: [Equation]

Original: Consider the normal cone mapping discussed in Section 2 and deﬁned as
Revised: Consider the normal cone mapping discussed in Section 2 and deﬁned as

Original: [Equation]
Revised: [Equation]

Original: It is easily seen that (71) is equivalent to ﬁnding z ∗ such that − B ( z ∗ ) ∈ N C ( z ∗ ) .
Revised: It is easily seen that (71) is equivalent to ﬁnding z ∗ such that − B ( z ∗ ) ∈ N C ( z ∗ ) .

Original: Hence, if B is monotone, (71) is equivalent to the monotone inclusion
Revised: Hence, if B is monotone, (71) is equivalent to the monotone inclusion

Original: [Equation]
Revised: [Equation]

Original:  Thus, monotone variational inequalities are a special case of monotone inclusions with two operators, one of which is single-valued and the other is the normal cone map of the constraint set C .
Revised: C ( z ∗ ) . Thus, monotone variational inequalities are a special case of monotone inclusions with two operators, one of which is single-valued and the other is the normal cone map of the constraint set C .

Original: As a consequence, methods for monotone inclusions can be used to solve monotone variational inequality problems.
Revised: As a consequence, methods for monotone inclusions can be used to solve monotone variational inequality problems.

Original: The reverse, however, may not be true.
Revised: The reverse, however, may not be true.

Original: For example, the analysis of the extragradient method (Korpelevich, 1977) relies on the second operator N C in (72) being a normal cone, as opposed to a more general monotone operator.
Revised: For example, the analysis of the extragradient method (Korpelevich, 1977) relies on the second operator N C in (72) being a normal cone, as opposed to a more general monotone operator.

Original: We are not aware of any direct extension of the extragradient method’s analysis allowing a more general resolvent to be used in place of the projection map corresponding to N C .
Revised: We are not aware of any direct extension of the extragradient method’s analysis allowing a more general resolvent to be used in place of the projection map corresponding to N C .

Original: The Restricted Gap Function There is a disadvantage to pursuing convergence rates based on variational inequalities (as in Böhm et al.
Revised: The Restricted Gap Function There is a disadvantage to pursuing convergence rates based on variational inequalities (as in Böhm et al.

Original: (2020) and Alacaoglu et al.
Revised: (2020) and Alacaoglu et al.

Original: (2021)) rather than monotone inclusions.
Revised: (2021)) rather than monotone inclusions.

Original: Convergence rate analyses for variational inequalities focus on the gap function :
Revised: Convergence rate analyses for variational inequalities focus on the gap function :

Original: [Equation]
Revised: [Equation]

Original: It can be shown that G C ( z ) ≥ 0 and G C ( z ) = 

Revised: It can be shown that G C ( z ) ≥ 0 and G C ( z ) =

Original: if and only if z solves (71).
Revised: if and only if z solves (71).

Original: However, (73) is meaningless for most problems, since unless C is compact, G C ( z ) is typically equal to + ∞ for any nonsolution (Diakonikolas, 2020).
Revised: However, (73) is meaningless for most problems, since unless C is compact, G C ( z ) is typically equal to + ∞ for any nonsolution (Diakonikolas, 2020).

Original: Thus researchers instead focus on the restricted gap function (Nesterov, 2007)
Revised: Thus researchers instead focus on the restricted gap function (Nesterov, 2007)

Original: [Equation]
Revised: [Equation]

Original: (74) where C 2 is an arbitrary compact set.
Revised: 74) where C 2 is an arbitrary compact set.

Original: However, now the results are only meaningful over the set C 2 .
Revised: However, now the results are only meaningful over the set C 2 .

Original: Thus, C 2 must be chosen large enough so that the iterates of the algorithm remain in the interior of
Revised: Thus, C 2 must be chosen large enough so that the iterates of the algorithm remain in the interior of

Original: C 2 (Böhm et al., 2020).
Revised: C 2 (Böhm et al., 2020).

Original: Further, the convergence rate bound depends on the diameter of C 2 .
Revised: Further, the convergence rate bound depends on the diameter of C 2 .

Original: For some algorithms (Mokhtari et al., 2020) a valid set is provided which bounds the iterates.
Revised: For some algorithms (Mokhtari et al., 2020) a valid set is provided which bounds the iterates.

Original: However
Revised: However

Original: Böhm et al.
Revised: Böhm et al.

Original: (2020) and Alacaoglu et al.
Revised: (2020) and Alacaoglu et al.

Original: (2021) do not provide one, although in principle it could be done so long as the ergodic sequence can be bounded almost-surely.
Revised: (2021) do not provide one, although in principle it could be done so long as the ergodic sequence can be bounded almost-surely.

Original: Thus, the convergence rates depending on (74) in Böhm et al.
Revised: Thus, the convergence rates depending on (74) in Böhm et al.

Original: (2020) and Alacaoglu et al.
Revised: (2020) and Alacaoglu et al.

Original: (2021) are somewhat incomplete in that they depend on unknown constants.
Revised: (2021) are somewhat incomplete in that they depend on unknown constants.

Original: In contrast, rates based on the approximation residual in the monotone inclusion setting, including ours given in (57)–(58), completely avoid this pitfall.
Revised: In contrast, rates based on the approximation residual in the monotone inclusion setting, including ours given in (57)–(58), completely avoid this pitfall.

Original: There is no need to select a compact set containing the algorithm’s iterates and the constants in our rates are all explicit or depend on standard quantities such as the initial distance to a solution.
Revised: There is no need to select a compact set containing the algorithm’s iterates and the constants in our rates are all explicit or depend on standard quantities such as the initial distance to a solution.

Original: H M EMORY -S AVING T ECHNIQUE FOR SPS
Revised: H M EMORY -S AVING T ECHNIQUE FOR SPS

Original: The variables t ki , x ki , and y ki on lines 3-5 of SPS are stored in variables t, x and y .
Revised: The variables t ki , x ki , and y ki on lines 3-5 of SPS are stored in variables t, x and y .

Original: Another two variables ¯ x and ¯ y keep track of (cid:80) ni
Revised: Another two variables ¯ x and ¯ y keep track of (cid:80) ni

Original: =1 x ki and (cid:80) ni
Revised: =1 x ki and (cid:80) ni

Original: =1 y ki .
Revised: =1 y ki .

Original: The dual variables are stored as w i for i ∈ 1 ..n and the primal variable as z .
Revised: The dual variables are stored as w i for i ∈ 1 ..n and the primal variable as z .

Original: Once x = x ki is computed, the i th dual variable w i can be partially updated as w i ← w i − α k x .
Revised: Once x = x k i is computed, the i th dual variable w i can be partially updated as w i ← w i − α k x .

Original: Once all the operators have been processed, the update for each dual variable may be completed via w i ← w i + α k ( n + 1) − 1 ¯ x .
Revised: Once all the operators have been processed, the update for each dual variable may be completed via w i ← w i + α k ( n + 1) − 1 ¯ x .

Original: Also, the primal update is computed as z ← z − α k ¯ y .
Revised: Also, the primal update is computed as z ← z − α k ¯ y .

Original: During the calculation loop for the x ki , y ki , the terms in approximation residual R k may also be accumulated one by one.
Revised: During the calculation loop for the x ki , y ki , the terms in approximation residual R k may also be accumulated one by one.

Original: The total total number of vector elements that must be stored is ( n + 7) d .
Revised: The total total number of vector elements that must be stored is ( n + 7) d .

Original: I A DDITIONAL I NFORMATION A BOUT THE N UMERICAL E XPERIMENTS
Revised: I A DDITIONAL I NFORMATION A BOUT THE N UMERICAL E XPERIMENTS

Original: We solve the following convex-concave min-max problem:
Revised: We solve the following convex-concave min-max problem:

Original: [Equation]
Revised: [Equation]

Original:  This model is identical to that of (Yu et al., 2021, Thm. 4.3) except for the addition of the (cid:96)regularization term c (cid:107) β (cid:107) 1 , where c ≥ 0 is a given constant.
Revised: [Equation] This model is identical to that of (Yu et al., 2021, Thm. 4.3) except for the addition of the (cid:96)regularization term c (cid:107) β (cid:107) 1 , where c ≥ 0 is a given constant.

Original: The goal is to learn the model weights β from a training dataset of m feature vectors ˆ x i and corresponding labels ˆ y i .
Revised: The goal is to learn the model weights β from a training dataset of m feature vectors ˆ x i and corresponding labels ˆ y i .

Original: Rather than computing the expected loss over the training set, the formulation uses, for each β , the worst possible distribution within a Wasserstein-metric ball around the empirical distribution of the { (ˆ x i , ˆ y i ) } , with the parameter δ ≥ 0 giving the diameter of the ball and the parameter κ ≥ 0 specifying the relative weighting of features and labels.
Revised: Rather than computing the expected loss over the training set, the formulation uses, for each β , the worst possible distribution within a Wasserstein-metric ball around the empirical distribution of the { (ˆ x i , ˆ y i ) } , with the parameter δ ≥ 0 giving the diameter of the ball and the parameter κ ≥ 0 specifying the relative weighting of features and labels.

Original: The variables γ and λ parameterize the selection of this worst-case distribution in response to the model weights β .
Revised: The variables γ and λ parameterize the selection of this worst-case distribution in response to the model weights β .

Original: Finally, Ψ is the logistic loss kernel t (cid:55)→ log( e t + e − t ) and L Ψ =is the corresponding Lipschitz constant.
Revised: Finally, Ψ is the logistic loss kernel t (cid:55)→ log( e t + e − t ) and L Ψ =is the corresponding Lipschitz constant.

Original: In all the experiments, we set δ = κ = 1 and c = 10 − 3 .
Revised: In all the experiments, we set δ = κ = 1 and c = 10 − 3 .

Original: We now show how we converted this problem to the form (1) for our experiments.
Revised: We now show how we converted this problem to the form (1) for our experiments.

Original: Let z be a shorthand for ( λ, β, γ ) , and deﬁne
Revised: Let z be a shorthand for ( λ, β, γ ) , and deﬁne

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst-order necessary and sufﬁcient conditions for the convex-concave saddlepoint problem in (75) are
Revised: The ﬁrst-order necessary and sufﬁcient conditions for the convex-concave saddlepoint problem in (75) are

Original: [Equation]
Revised: [Equation]

Original: where the vector ﬁeld B ( z ) is deﬁned as
Revised: where the vector ﬁeld B ( z ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: It is readily conﬁrmed that B deﬁned in this manner is Lipschitz.
Revised: It is readily conﬁrmed that B deﬁned in this manner is Lipschitz.

Original: The monotonicity of B follows from its being the generalized gradient of a convex-concave saddle function (Rockafellar, 1970).
Revised: The monotonicity of B follows from its being the generalized gradient of a convex-concave saddle function (Rockafellar, 1970).

Original: For the set-valued operators, A 1 ( z ) corresponds to the constraints and A 2 ( z ) to the nonsmooth (cid:96)regularizer, and are deﬁned as
Revised: For the set-valued operators, A 1 ( z ) corresponds to the constraints and A 2 ( z ) to the nonsmooth (cid:96)regularizer, and are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here, the notation 

Revised: Here, the notation

Original: p × 1 denotes the p -dimensional vector of all zeros.
Revised: p × 1 denotes the p -dimensional vector of all zeros.

Original: C 1 is a scaled version of the second-order cone, well known to be a closed convex set, while C 2 is the unit ball of the (cid:96) ∞ norm, also closed and convex.
Revised: C 1 is a scaled version of the second-order cone, well known to be a closed convex set, while C 2 is the unit ball of the (cid:96) ∞ norm, also closed and convex.

Original: Since A 1 is a normal cone map of a closed convex set and A 2 is the subgradient map of a closed proper convex function (the scaled 1 -norm), both of these operators are maximal monotone and problem (76) is a special case of (1) for n = 2 .
Revised: Since A 1 is a normal cone map of a closed convex set and A 2 is the subgradient map of a closed proper convex function (the scaled 1 -norm), both of these operators are maximal monotone and problem (76) is a special case of (1) for n = 2 .

Original: Stochastic oracle implementation The operator B : R m + d +1 (cid:55)→ R m + d +1 , deﬁned in (77), can be written as 
Revised: Stochastic oracle implementation The operator B : R m + d +1 (cid:55)→ R m + d +1 , deﬁned in (77), can be written as [Equation] i = B i

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In our SPS experiments, the stochastic oracle for B is simply ˜ B ( z )
Revised: In our SPS experiments, the stochastic oracle for B is simply ˜ B ( z )

Original: = 1 | B | (cid:80) i ∈
Revised: = 1 | B | (cid:80) i ∈

Original: B B i ( z ) for some minibatch B ⊆ { 1 , . . .
Revised: B B i ( z ) for some minibatch B ⊆ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: We used a batchsize of 100 .
Revised: We used a batchsize of 100 .

Original: Resolvent computations
Revised: Resolvent computations

Original: The resolvent of A 1 is readily constructed from the projection maps of the simple sets C 1 and C 2 , while the resolvent A 2 involves the proximal operator of the (cid:96) 1 norm.
Revised: The resolvent of A 1 is readily constructed from the projection maps of the simple sets C 1 and C 2 , while the resolvent A 2 involves the proximal operator of the (cid:96) 1 norm.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: The constraint C 1 is a scaled second-order cone and C 2 is the (cid:96) ∞ ball, both of which have closed-form projections.
Revised: The constraint C 1 is a scaled second-order cone and C 2 is the (cid:96) ∞ ball, both of which have closed-form projections.

Original: The proximal operator of the (cid:96) 1 norm is the well-known soft-thresholding operator (Parikh & Boyd, 2013, Section 6.5.2).
Revised: The proximal operator of the (cid:96) 1 norm is the well-known soft-thresholding operator (Parikh & Boyd, 2013, Section 6.5.2).

Original: Therefore all resolvents in the formulation may be computed quickly and accurately.
Revised: Therefore all resolvents in the formulation may be computed quickly and accurately.

Original: SPS stepsize choices For the stepsize in SPS, we ordinarily require ρ k ≤ ρ < 1 /L for the global
Revised: SPS stepsize choices For the stepsize in SPS, we ordinarily require ρ k ≤ ρ < 1 /L for the global

Original: Lipschitz constant L of B .
Revised: Lipschitz constant L of B .

Original: However, since the global Lipschitz constant may be pessimistic, better performance can often be achieved by experimenting with larger stepsizes.
Revised: However, since the global Lipschitz constant may be pessimistic, better performance can often be achieved by experimenting with larger stepsizes.

Original: If divergence is observed, then the stepsize can be decreased.
Revised: If divergence is observed, then the stepsize can be decreased.

Original: This type of strategy is common for SGD and similar stochastic methods.
Revised: This type of strategy is common for SGD and similar stochastic methods.

Original: Thus, for SPS-decay we set α k =
Revised: Thus, for SPS-decay we set α k =

Original: C d k − 0 .
Revised: C d k − 0 .

Original: 51 and ρ k = C
Revised: 51 and ρ k = C

Original: d k − 0 .
Revised: d k − 0 .

Original: 25 , and performed a grid search to select the best C d from { 0 . 1 , 0 . 5 , 1 , 5 , 10 } , arriving at C d = 1 for epsilon and SUSY, and
Revised: 25 , and performed a grid search to select the best C d from { 0 . 1 , 0 . 5 , 1 , 5 , 10 } , arriving at C d = 1 for epsilon and SUSY, and

Original: C d = 0 .
Revised: C d = 0 .

Original: 5 for real-sim.
Revised: 5 for real-sim.

Original: For SPS-ﬁxed we used ρ =
Revised: For SPS-ﬁxed we used ρ =

Original: K − 1 / 4 and α = C f ρ 2 , and performed a grid search to select C f over { 0 . 
Revised: K − 1 / 4 and α = C f ρ 2 , and performed a grid search to select C f over { 0 . 1 , 0 . 5 , 1 , 5 , 10 } , arriving at C f = 1 for epsilon and real-sim, and C f =for SUSY.

Revised: The total number of iterations for SPS-ﬁxed was chosen as follows: For the epsilon dataset, we used K = 5000 , for SUSY we used K = 200 , and for real-sim we used K = 1000 .

Original: Parameter choices for the other algorithms All methods are initialized at the same random point.
Revised: Parameter choices for the other algorithms All methods are initialized at the same random point.

Original: For Tseng’s method, we used the backtracking linesearch variant with an initial stepsize of 1 , θ = 0 . 8 , and a stepsize reduction factor of 0 .
Revised: For Tseng’s method, we used the backtracking linesearch variant with an initial stepsize of 1 , θ = 0 . 8 , and a stepsize reduction factor of 0 .

Original: For FRB, we used the backtracking linesearch variant with the same settings as for Tseng’s method.
Revised: For FRB, we used the backtracking linesearch variant with the same settings as for Tseng’s method.

Original: For deterministic PS, we used a ﬁxed stepsize of 0 . 9 /L .
Revised: For deterministic PS, we used a ﬁxed stepsize of 0 . 9 /L .

Original: For the stochastic Tseng’s method of Böhm et al.
Revised: For the stochastic Tseng’s method of Böhm et al.

Original: (2020), the stepsize α k must satisfy: (cid:80) ∞ k
Revised: (2020), the stepsize α k must satisfy: (cid:80) ∞ k

Original: =1 α k = ∞ and (cid:80) ∞
Revised: =1 α k = ∞ and (cid:80) ∞

Original: k =1 α 2 k < ∞ .
Revised: k =1 α 2 k < ∞ .

Original: So we set α k
Revised: So we set α k

Original: =
Revised: =

Original: Ck − d and perform a grid search over { C, d } in the range [10 − 4 , 10] ×
Revised: Ck − d and perform a grid search over { C, d } in the range [10 − 4 , 10] ×

Original: [0 . 51 , 1] , checking 5 × 5 values to ﬁnd the best setting for each of the three problems.
Revised: [0 . 51 , 1] , checking 5 × 5 values to ﬁnd the best setting for each of the three problems.

Original: The selected values are in Table 1.
Revised: The selected values are in Table 1.

Revised: epsilon

Revised: SUSY real-sim

Revised: C 0 .

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: d

Revised: [Table]

Revised: [Table]

Original: The work of Böhm et al.
Revised: The work of Böhm et al.

Original: (2020) also introduced FBFp, a stochastic version of Tseng’s method that reuses a previously-computed gradient and therefore only needs one additional gradient calculation per iteration.
Revised: (2020) also introduced FBFp, a stochastic version of Tseng’s method that reuses a previously-computed gradient and therefore only needs one additional gradient calculation per iteration.

Original: In our experiments, the performance of the two methods was about the same, so we only report the performance of stoch.
Revised: In our experiments, the performance of the two methods was about the same, so we only report the performance of stoch.

Original: Tseng’s method.
Revised: Tseng’s method.

Original: For variance-reduced FRB, the main parameter is the probability p .
Revised: For variance-reduced FRB, the main parameter is the probability p .

Original: We hand-tuned p ,arriving at p = 0 .
Revised: We hand-tuned p ,arriving at p = 0 .

Original: 01 for all problems.
Revised: 01 for all problems.

Original: We set the stepsize to its maximum allowed value of
Revised: We set the stepsize to its maximum allowed value of

Original: [Equation]
Revised: [Equation]

Revised: Plots versus Epoch Figure 2 plots the performance of each method versus epoch (i.e. data pass).

Revised: This shows an even more dramatic beneﬁt for the stochastic methods than the plots versus time, since at each iteration the stochastic methods only need to process small amounts of data, whereas deterministic methods must process all of it.

Revised: We believe these beneﬁts do not fully manifest themselves in the plots versus time due to overheads in each iteration of the stochastic methods, multithreading providing a boost for the deterministic methods, memory access patterns, and other practical considerations.

Original: J L OCAL C ONVERGENCE ON N ON -M ONOTONE P ROBLEMS
Revised: J L OCAL C ONVERGENCE ON N ON -M ONOTONE P ROBLEMS

Original: The work by Hsieh et al.
Revised: The work by Hsieh et al.

Original: (2020) provides a local convergence analysis for DSEG applied to locally monotone problems.
Revised: (2020) provides a local convergence analysis for DSEG applied to locally monotone problems.

Original: Recall that DSEG is equivalent to the special case of SPS for which n = 0 .
Revised: Recall that DSEG is equivalent to the special case of SPS for which n = 0 .

Original: While extending this result to the more general setting of SPS is beyond the scope of this manuscript, we next provide a preliminary sketch of how the analysis of Hsieh et al.
Revised: While extending this result to the more general setting of SPS is beyond the scope of this manuscript, we next provide a preliminary sketch of how the analysis of Hsieh et al.

Original: (2020) might be generalized to our setting.
Revised: (2020) might be generalized to our setting.

Original: We leave a formal proof to future work.
Revised: We leave a formal proof to future work.

Original: Sketch of assumptions and main result The ﬁrst assumption needed is the existence of an isolated solution p ∗ =
Revised: Sketch of assumptions and main result The ﬁrst assumption needed is the existence of an isolated solution p ∗ =

Original: ( z ∗ , w ∗ 1 , . . .
Revised: ( z ∗ , w ∗ 1 , . . .

Original: , w ∗ n +1 ) ∈ S .
Revised: , w ∗ n +1 ) ∈ S .

Original: We then require that there exists a ball B r ( z ∗ ) , centered at z ∗ , throughout which the operator B is “well-behaved”, meaning that it satisﬁes monotonicity and
Revised: We then require that there exists a ball B r ( z ∗ ) , centered at z ∗ , throughout which the operator B is “well-behaved”, meaning that it satisﬁes monotonicity and

Original: Lipschitz continuity.
Revised: Lipschitz continuity.

Original: In addition, we need each A i , for i ∈ 1 ..n , to be maximal monotone within this ball.
Revised: In addition, we need each A i , for i ∈ 1 ..n , to be maximal monotone within this ball.

Original: Outside of the ball, the operators do not need to be monotone or Lipschitz.
Revised: Outside of the ball, the operators do not need to be monotone or Lipschitz.

Original: Following (Hsieh et al., 2020, Assumption 2 (cid:48) ), the noise variance assumptions are slightly stronger than in the monotone case.
Revised: Following (Hsieh et al., 2020, Assumption 2 (cid:48) ), the noise variance assumptions are slightly stronger than in the monotone case.

Original: In particular, we require that E [ (cid:107) (cid:15)
Revised: In particular, we require that E [ (cid:107) (cid:15)

Original: k (cid:107) q |F k ] ≤ N q and E [ (cid:107) e k (cid:107) q |F k ] ≤ N q for some q > 2 .
Revised: k (cid:107) q |F k ] ≤ N q and E [ (cid:107) e k (cid:107) q |F k ] ≤ N q for some q > 2 .

Original: As before, the noise must be zero-mean.
Revised: As before, the noise must be zero-mean.

Original: Finally, the stepsize requirements are also slightly stronger than (12), having the added assumption that (cid:80) ∞ k
Revised: Finally, the stepsize requirements are also slightly stronger than (12), having the added assumption that (cid:80) ∞ k

Original: =1 ρ qk < ∞ .
Revised: =1 ρ qk < ∞ .

Original: With these assumptions, the goal is to show that, so long as the initial point p 1 is sufﬁciently close to p ∗ , then with high probability p k converges to p ∗ .
Revised: With these assumptions, the goal is to show that, so long as the initial point p 1 is sufﬁciently close to p ∗ , then with high probability p k converges to p ∗ .

Original: Proof strategy The initial strategy is to develop the following recursion, satisﬁed by SPS, that does not (yet) utilize local monotonicity or Lipschitz continuity:
Revised: Proof strategy The initial strategy is to develop the following recursion, satisﬁed by SPS, that does not (yet) utilize local monotonicity or Lipschitz continuity:

Original: [Equation]
Revised: [Equation]

Original: for appropriate constants c 1 . . .
Revised: for appropriate constants c 1 . . .

Original: c 5 ≥ 0 .
Revised: c 5 ≥ 0 .

Original: In this inequality, we use
Revised: In this inequality, we use

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: with L being the local Lipschitz constant of B on B r ( z ∗ ) .
Revised: with L being the local Lipschitz constant of B on B r ( z ∗ ) .

Original: The iterate ˜ x k is the analog of the iterate ˜ X t +1 / 2 used in Hsieh et al.
Revised: The iterate ˜ x k is the analog of the iterate ˜ X t +1 / 2 used in Hsieh et al.

Original: The recursion (78) is derived by once again starting from (13) and following the arguments leading to (35), but this time not taking conditional expectations.
Revised: The recursion (78) is derived by once again starting from (13) and following the arguments leading to (35), but this time not taking conditional expectations.

Original: In particular, the upper bounds on (cid:107)∇ z ϕ k (cid:107)and (cid:107)∇ w i ϕ k (cid:107) 2 contribute the terms c 1 α 2 k
Revised: In particular, the upper bounds on (cid:107)∇ z ϕ k (cid:107)and (cid:107)∇ w i ϕ k (cid:107) 2 contribute the terms c 1 α 2 k

Original: [Equation]
Revised: [Equation]

Original: k (cid:107) 2 + c 4 ) and c 1 α 2 k (cid:107) p k − p ∗ (cid:107) 2 .
Revised: k (cid:107) 2 + c 4 ) and c 1 α 2 k (cid:107) p k − p ∗ (cid:107) 2 .

Original: For i ∈ 1 ..n , the “ ϕ i,k -gap" term, ϕ i,k ( p k ) − ϕ i,k ( p ∗ ) , is dealt with in a similar manner to Section C.5, but this time not using monotonicity as in (36).
Revised: For i ∈ 1 ..n , the “ ϕ i,k -gap" term, ϕ i,k ( p k ) − ϕ i,k ( p ∗ ) , is dealt with in a similar manner to Section C.5, but this time not using monotonicity as in (36).

Original: This contributes T (cid:48) k and the ﬁrst term in l k .
Revised: This contributes T (cid:48) k and the ﬁrst term in l k .

Original: Finally, as we sketch below, the “ ϕ n +1 ,k -gap" term contributes r k , r (cid:48) k , q k , q (cid:48) k , and the last term in l k .
Revised: Finally, as we sketch below, the “ ϕ n +1 ,k -gap" term contributes r k , r (cid:48) k , q k , q (cid:48) k , and the last term in l k .

Original: For the “ ϕ n +1 ,k -gap”, that is, ϕ n +1 ,k ( p k ) − ϕ n +1 ,k ( p ∗ ) , we have to depart from the analysis in
Revised: For the “ ϕ n +1 ,k -gap”, that is, ϕ n +1 ,k ( p k ) − ϕ n +1 ,k ( p ∗ ) , we have to depart from the analysis in

Original: Section C.6 and use an alternative argument involving ˜ x k .
Revised: Section C.6 and use an alternative argument involving ˜ x k .

Original: We now provide some details of this argument: in the following, we use Bz as shorthand for B ( z ) for any vector z ∈ R d .
Revised: We now provide some details of this argument: in the following, we use Bz as shorthand for B ( z ) for any vector z ∈ R d .

Original: We begin the analysis with
Revised: We begin the analysis with

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal term will combine with the term (cid:104) x kn +1 − z ∗ , e k (cid:105) coming from
Revised: The ﬁnal term will combine with the term (cid:104) x kn +1 − z ∗ , e k (cid:105) coming from

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: to yield r (cid:48)
Revised: to yield r (cid:48)

Original: k above.
Revised: k above.

Original: Equation (81) also yields the second term in l k .
Revised: Equation (81) also yields the second term in l k .

Original: Using that ˜ x
Revised: Using that ˜ x

Original: k − x kn +

Revised: k − x kn +

Original: = ρ k (cid:15) k , we rewrite the ﬁrst term in (80) as
Revised: = ρ k (cid:15) k , we rewrite the ﬁrst term in (80) as

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: the terms in (82) admit the lower bound
Revised: the terms in (82) admit the lower bound

Original: [Equation]
Revised: [Equation]

Original: ﬁrst part of q
Revised: ﬁrst part of q

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Considering the ﬁrst term on right-hand side of this bound, we also have
Revised: Considering the ﬁrst term on right-hand side of this bound, we also have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: for any d > 0 , using Young’s inequality.
Revised: for any d > 0 , using Young’s inequality.

Original: Finally, for the ﬁrst two terms of the right-hand side of the above relation, we may write
Revised: Finally, for the ﬁrst two terms of the right-hand side of the above relation, we may write

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁnal inequality we use the Cauchy-Schwartz inequality and substitute Bz k − w kn
Revised: where in the ﬁnal inequality we use the Cauchy-Schwartz inequality and substitute Bz k − w kn

Original: +

Revised: +

Original: =
Revised: =

Original: ρ − 1 k ( z k − ˜ x k ) , from the deﬁnition of ˜ x k in (79).
Revised: ρ − 1 k ( z k − ˜ x k ) , from the deﬁnition of ˜ x k in (79).

Original: We have now accounted for all the terms appearing in (78).
Revised: We have now accounted for all the terms appearing in (78).

Original: The recursion (78) is analogous to equation (F.7) on page 24 of Hsieh et al.
Revised: The recursion (78) is analogous to equation (F.7) on page 24 of Hsieh et al.

Original: and provides the starting point for the local convergence analysis.
Revised: and provides the starting point for the local convergence analysis.

Original: The next step would be to derive an analog of
Revised: The next step would be to derive an analog of

Original: Theorem F.1.
Revised: Theorem F.1.

Original: of Hsieh et al.
Revised: of Hsieh et al.

Original: (2020) using (78).
Revised: (2020) using (78).

Original: The following translation to the notation of Theorem
Revised: The following translation to the notation of Theorem

Original: F.1.
Revised: F.1.

Original: could be used (note that Hsieh et al.
Revised: could be used (note that Hsieh et al.

Original: (2020) uses t for iteration counter):
Revised: (2020) uses t for iteration counter):

Original: [Equation]
Revised: [Equation]

Original: and the event E ρ ∞ is translated to
Revised: and the event E ρ ∞ is translated to

Original: [Equation]
Revised: [Equation]

Original: An analog of Theorem 2 of Hsieh et al.
Revised: An analog of Theorem 2 of Hsieh et al.

Original: (2020) could then be developed based on this result.
Revised: (2020) could then be developed based on this result.