Original: Stochastic Gradient Methods with Compressed Communication for Decentralized Saddle Point Problems
Revised: Stochastic Gradient Methods with Compressed Communication for Decentralized Saddle Point Problems

Original: Abstract
Revised: Abstract

Original: We develop two compression based stochastic gradient algorithms to solve a class of non-smooth strongly convex-strongly concave saddle-point problems in a decentralized setting (without a central server).
Revised: We develop two compression based stochastic gradient algorithms to solve a class of non-smooth strongly convex-strongly concave saddle-point problems in a decentralized setting (without a central server).

Original: Our ﬁrst algorithm is a
Revised: Our ﬁrst algorithm is a

Original: Restart-based Decentralized Proximal Stochastic Gradient method with Compression (C-RDPSG) for general stochastic settings.
Revised: Restart-based Decentralized Proximal Stochastic Gradient method with Compression (C-RDPSG) for general stochastic settings.

Original: We provide rigorous theoretical guarantees of C-RDPSG with gradient computation complexity and communication complexity of order O
Revised: We provide rigorous theoretical guarantees of C-RDPSG with gradient computation complexity and communication complexity of order O

Original: ((1 + δ ) 4 1 L 2 κ 2 f κ 2 g 1 (cid:15) ) , to achieve an (cid:15) -accurate saddle-point solution, where δ denotes the compression factor, κ f and κ g denote respectively the condition numbers of objective function and communication graph, and L denotes the smoothness parameter of the smooth part of the objective function.
Revised: ((1 + δ ) 4 1 L 2 κ 2 f κ 2 g 1 (cid:15) ) , to achieve an (cid:15) -accurate saddle-point solution, where δ denotes the compression factor, κ f and κ g denote respectively the condition numbers of objective function and communication graph, and L denotes the smoothness parameter of the smooth part of the objective function.

Original: Next, we present a Decentralized Proximal Stochastic Variance Reduced Gradient algorithm with Compression (C-DPSVRG) for ﬁnite sum setting which exhibits gradient computation complexity and communication complexity of order
Revised: Next, we present a Decentralized Proximal Stochastic Variance Reduced Gradient algorithm with Compression (C-DPSVRG) for ﬁnite sum setting which exhibits gradient computation complexity and communication complexity of order

Original: [Equation]
Revised: [Equation]

Original: Extensive numerical experiments show competitive performance of the proposed algorithms and provide support to the theoretical results obtained.
Revised: Extensive numerical experiments show competitive performance of the proposed algorithms and provide support to the theoretical results obtained.

Original:  Introduction
Revised:  Introduction

Original: We focus on solving the following saddle point (or mini-max) problem in a fully decentralized setting without a central server:
Revised: We focus on solving the following saddle point (or mini-max) problem in a fully decentralized setting without a central server:

Original: [Equation]
Revised: [Equation]

Original: where X ⊂ R d x , Y ⊂ R d y are convex and compact sets, f i : X×Y (cid:55)→ R for every node i ∈ [ m ] is smooth, strongly-convex in x and strongly-concave in y , and g :
Revised: where X ⊂ R d x , Y ⊂ R d y are convex and compact sets, f i : X×Y (cid:55)→ R for every node i ∈ [ m ] is smooth, strongly-convex in x and strongly-concave in y , and g :

Original: X(cid:55)→ R and r : Y(cid:55)→ R are proper, continuous, convex functions which might be non-smooth.
Revised: X(cid:55)→ R and r : Y(cid:55)→ R are proper, continuous, convex functions which might be non-smooth.

Original: This class of saddle point problems ﬁnds its use in robust classiﬁcation and regression applications and AUC maximization problems [30, 40].
Revised: This class of saddle point problems ﬁnds its use in robust classiﬁcation and regression applications and AUC maximization problems [30, 40].

Original: We assume that the static topology of the decentralized environment is represented using an undirected, connected, simple graph G =
Revised: We assume that the static topology of the decentralized environment is represented using an undirected, connected, simple graph G =

Original: ( V , E ) , where V = { 1 , 2 , . . .
Revised: ( V , E ) , where V = { 1 , 2 , . . .

Original: , m }
Revised: , m }

Original: =: [ m ] denotes the set of m computing nodes (with similar computing capabilities and memory) and an edge e ij ∈ E denotes that nodes i, j ∈ V are connected.
Revised: =: [ m ] denotes the set of m computing nodes (with similar computing capabilities and memory) and an edge e ij ∈ E denotes that nodes i, j ∈ V are connected.

Original: Also, we assume that the communication is synchronous and at every synchronization step, node i communicates only with its neighbors N ( i ) =
Revised: Also, we assume that the communication is synchronous and at every synchronization step, node i communicates only with its neighbors N ( i ) =

Original: { j ∈ V : e ij ∈ E} .
Revised: { j ∈ V : e ij ∈ E} .

Original: In this work, we design algorithms which can achieve sublinear/linear convergence rates to solve problem (SPP), by using stochastic oracles for efﬁcient gradient computation, and compressed information exchange for efﬁcient communication.
Revised: In this work, we design algorithms which can achieve sublinear/linear convergence rates to solve problem (SPP), by using stochastic oracles for efﬁcient gradient computation, and compressed information exchange for efﬁcient communication.

Original: In particular, we focus on the following two settings of problem (SPP): (i) general stochastic setting (ii) ﬁnite sum setting.
Revised: In particular, we focus on the following two settings of problem (SPP): (i) general stochastic setting (ii) ﬁnite sum setting.

Original: In general stochastic setting, we assume the following form for the local function: f i ( x, y ) =
Revised: In general stochastic setting, we assume the following form for the local function: f i ( x, y ) =

Original: E ξ i ∼D i (cid:2)
Revised: E ξ i ∼D i (cid:2)

Original: f i ( x, y ; ξ i ) (cid:3) , where
Revised: f i ( x, y ; ξ i ) (cid:3) , where

Original: D i is the local sample distribution in node i , allowing for general heterogeneous data distributions across different nodes.
Revised: D i is the local sample distribution in node i , allowing for general heterogeneous data distributions across different nodes.

Original: In ﬁnite sum setting, we assume that each f i ( x, y ) is represented as the average over local batches, hence f i ( x, y ) = 1 n (cid:80) nj =1 f ij ( x, y ) , where f ij ( x, y ) represents the loss function at j th batch of samples at node i .
Revised: In ﬁnite sum setting, we assume that each f i ( x, y ) is represented as the average over local batches, hence f i ( x, y ) = 1 n (cid:80) nj =1 f ij ( x, y ) , where f ij ( x, y ) represents the loss function at j th batch of samples at node i .

Original: Our Contributions : Inspired by algorithms developed for decentralized minimization problems [24, 20], we design two algorithms for the general stochastic setting and ﬁnite sum setting.
Revised: Our Contributions : Inspired by algorithms developed for decentralized minimization problems [24, 20], we design two algorithms for the general stochastic setting and ﬁnite sum setting.

Original: For general stochastic setting, we propose Restart-based Decentralized Proximal Stochastic Gradient method with Compression (C-RDPSG) which achieves gradient computation complexity (measured in terms of number of calls to stochastic gradient oracles (SGO)) and communication complexity (measured in terms of number of communication rounds) of order O ((1 + δ ) 4 1 L 2 κ 2 f κ 2 g 1 (cid:15) ) , to obtain an (cid:15) -accurate saddle-point solution, where δ denotes the compression factor, κ f and κ g denote respectively the condition numbers of objective function f and communication graph G , and L denotes the smoothness parameter of smooth part 1 m (cid:80) i f i .
Revised: For general stochastic setting, we propose Restart-based Decentralized Proximal Stochastic Gradient method with Compression (C-RDPSG) which achieves gradient computation complexity (measured in terms of number of calls to stochastic gradient oracles (SGO)) and communication complexity (measured in terms of number of communication rounds) of order O ((1 + δ ) 4 1 L 2 κ 2 f κ 2 g 1 (cid:15) ) , to obtain an (cid:15) -accurate saddle-point solution, where δ denotes the compression factor, κ f and κ g denote respectively the condition numbers of objective function f and communication graph G , and L denotes the smoothness parameter of smooth part 1 m (cid:80) i f i .

Original: In ﬁnite sum setting common in machine learning, we propose
Revised: In ﬁnite sum setting common in machine learning, we propose

Original: Decentralized Proximal Stochastic Variance Reduced Gradient algorithm with Compression (CDPSVRG) which is shown to have gradient computation complexity and communication complexity of order O
Revised: Decentralized Proximal Stochastic Variance Reduced Gradient algorithm with Compression (CDPSVRG) which is shown to have gradient computation complexity and communication complexity of order O

Original: ((1+ δ ) max { κ 2 f , √ δκ 2 f κ g , κ g } log( 1 (cid:15) )) .
Revised: ((1+ δ ) max { κ 2 f , √ δκ 2 f κ g , κ g } log( 1 (cid:15) )) .

Original: Using extensive experiments, we show empirical evidence supporting the theoretical guarantees obtained in our analysis.
Revised: Using extensive experiments, we show empirical evidence supporting the theoretical guarantees obtained in our analysis.

Original: Notations: The following notations will be used in the paper.
Revised: Notations: The following notations will be used in the paper.

Original: The k -tuple ( u 1 , u 2 , . . . , u k ) of k vectors u 1 , u 2 , . . .
Revised: The k -tuple ( u 1 , u 2 , . . . , u k ) of k vectors u 1 , u 2 , . . .

Original: , u k , each of size d × 1 denotes the vector (cid:2) ( u 1 ) (cid:62) ( u 2 ) (cid:62) . . .
Revised: , u k , each of size d × 1 denotes the vector (cid:2) ( u 1 ) (cid:62) ( u 2 ) (cid:62) . . .

Original: ( u k ) (cid:62) (cid:3) (cid:62) of size kd × 1 .
Revised: ( u k ) (cid:62) (cid:3) (cid:62) of size kd × 1 .

Original: The notation z =( x, y ) ∈
Revised: The notation z =( x, y ) ∈

Original: R d x + d y denotes the pair of primal variable x and dual variable y , and z (cid:63) =(
Revised: R d x + d y denotes the pair of primal variable x and dual variable y , and z (cid:63) =(

Original: x (cid:63) , y (cid:63) ) denotes a saddle point of problem (SPP).
Revised: x (cid:63) , y (cid:63) ) denotes a saddle point of problem (SPP).

Original: The communication link between a pair of nodes ( i, j ) ∈
Revised: The communication link between a pair of nodes ( i, j ) ∈

Original: V × V is assumed to have associated weight W ij ∈ [0 , 1] .
Revised: V × V is assumed to have associated weight W ij ∈ [0 , 1] .

Original: Weights W ij are collected into a weight matrix W of size m × m .
Revised: Weights W ij are collected into a weight matrix W of size m × m .

Original: I p denotes a p × p identity matrix, 1 denotes a m × 1 column vector of ones and J = 1 m 11 (cid:62) is a m × m matrix of uniform weights equal to 1 m .
Revised: I p denotes a p × p identity matrix, 1 denotes a m × 1 column vector of ones and J = 1 m 11 (cid:62) is a m × m matrix of uniform weights equal to 1 m .

Original: We deﬁne f ( z ) :
Revised: We deﬁne f ( z ) :

Original: = f ( x, y ) :
Revised: = f ( x, y ) :

Original: = (cid:80) mi =

Revised: = (cid:80) mi =

Original: f i ( x, y ) , L = max { L xx , L yy , L xy , L yx } , µ = min { µ x , µ y } where
Revised: f i ( x, y ) , L = max { L xx , L yy , L xy , L yx } , µ = min { µ x , µ y } where

Original: L xx , L yy , L xy , L yx are the smoothness parameters of f i ( x, y )
Revised: L xx , L yy , L xy , L yx are the smoothness parameters of f i ( x, y )

Original: (see Appendices E, G) and µ x , µ y are the strong convexity, strong concavity parameters of f i ( x, y )
Revised: (see Appendices E, G) and µ x , µ y are the strong convexity, strong concavity parameters of f i ( x, y )

Original: (see Assumptions 3.1-3.2).
Revised: (see Assumptions 3.1-3.2).

Original: The condition number κ f of f is deﬁned as L/µ .
Revised: The condition number κ f of f is deﬁned as L/µ .

Original: The condition number κ g of communication graph G is deﬁned as the ratio of largest eigenvalue and second smallest eigenvalue of I − W .
Revised: The condition number κ g of communication graph G is deﬁned as the ratio of largest eigenvalue and second smallest eigenvalue of I − W .

Original: The notation (cid:104) u, v (cid:105) denotes the inner product between two d × 1 vectors u and v .
Revised: The notation (cid:104) u, v (cid:105) denotes the inner product between two d × 1 vectors u and v .

Original: For a d × 1 vector u and for some d × d symmetric positive semi-deﬁnite (p.s.d) matrix A , we deﬁne (cid:107) u (cid:107) 

Revised: For a d × 1 vector u and for some d × d symmetric positive semi-deﬁnite (p.s.d) matrix A , we deﬁne (cid:107) u (cid:107)

Original: A = u (cid:62) Au .
Revised: A = u (cid:62) Au .

Original: (cid:107) u (cid:107) 2 denotes (cid:96)norm of vector u .
Revised: (cid:107) u (cid:107) 2 denotes (cid:96)norm of vector u .

Original: A ⊗ B denotes the Kronecker product of two matrices A and B .
Revised: A ⊗ B denotes the Kronecker product of two matrices A and B .

Original: Paper Organization: We illustrate an inexact primal-dual hybrid algorithm for solving (SPP) in
Revised: Paper Organization: We illustrate an inexact primal-dual hybrid algorithm for solving (SPP) in

Original: Section 2, followed by a discussion of assumptions used for problem setup (Section 3).
Revised: Section 2, followed by a discussion of assumptions used for problem setup (Section 3).

Original: Details about
Revised: Details about

Original: C-RDPSG for general stochastic setting and C-DPSVRG for ﬁnite sum setting are given respectively in Sections 4 and 5.
Revised: C-RDPSG for general stochastic setting and C-DPSVRG for ﬁnite sum setting are given respectively in Sections 4 and 5.

Original: Due to space constraints, related work is given in Appendix A and experiment details are in Appendix I.
Revised: Due to space constraints, related work is given in Appendix A and experiment details are in Appendix I.

Original:  An Inexact Primal-Dual Hybrid Algorithm to Solve Problem (SPP)
Revised:  An Inexact Primal-Dual Hybrid Algorithm to Solve Problem (SPP)

Original: Assuming the local copy of ( x, y ) in i -th node as ( x i , y i ) , we collect local primal and dual variables into x = (cid:0) x 1 , x 2 , . . .
Revised: Assuming the local copy of ( x, y ) in i -th node as ( x i , y i ) , we collect local primal and dual variables into x = (cid:0) x 1 , x 2 , . . .

Original: x m (cid:1) ∈
Revised: x m (cid:1) ∈

Original: R md x and y = (cid:0) y 1 , y 2 , . . .
Revised: R md x and y = (cid:0) y 1 , y 2 , . . .

Original: y m (cid:1)
Revised: y m (cid:1)

Original: ∈ R
Revised: ∈ R

Original: md y .
Revised: md y .

Original: Using this notation, the problem (SPP) can be formulated as an optimization problem with consensus constraints on x and y :
Revised: Using this notation, the problem (SPP) can be formulated as an optimization problem with consensus constraints on x and y :

Original: [Equation]
Revised: [Equation]

Original: where F ( x , y ) = (cid:80) mi =

Revised: where F ( x , y ) = (cid:80) mi =

Original: f i ( x i , y i ) , G ( x ) =
Revised: f i ( x i , y i ) , G ( x ) =

Original: (cid:80) mi
Revised: (cid:80) mi

Original: =1 ( g ( x i )+ δ X ( x i )) , R ( y ) =
Revised: =1 ( g ( x i )+ δ X ( x i )) , R ( y ) =

Original: (cid:80) mi
Revised: (cid:80) mi

Original: =1 ( r ( y i )
Revised: =1 ( r ( y i )

Original: + δ Y
Revised: + δ Y

Original: ( y i ))
Revised: ( y i ))

Original: and U = √
Revised: and U = √

Original: I m − W .
Revised: I m − W .

Original: Formulating a decentralized optimization problem as an equivalent problem with a consensus constraint on the local variables is well-known [26].
Revised: Formulating a decentralized optimization problem as an equivalent problem with a consensus constraint on the local variables is well-known [26].

Original: The assumptions on W (to be made later) would imply I m − W to be symmetric p.s.d.
Revised: The assumptions on W (to be made later) would imply I m − W to be symmetric p.s.d.

Original: and hence the existence of √ I m − W .
Revised: and hence the existence of √ I m − W .

Original: δ C ( u ) denotes the indicator function of set C which is 0 when u ∈ C and + ∞ otherwise.
Revised: δ C ( u ) denotes the indicator function of set C which is 0 when u ∈ C and + ∞ otherwise.

Original: We have the following Lagrangian function of problem (1):
Revised: We have the following Lagrangian function of problem (1):

Original: [Equation]
Revised: [Equation]

Original: where S x ∈ R md x and S y ∈ R md
Revised: where S x ∈ R md x and S y ∈ R md

Original: y denote the Lagrange multipliers associated with consensus constraints on variables x and y respectively.
Revised: y denote the Lagrange multipliers associated with consensus constraints on variables x and y respectively.

Original: We prove that solving constrained problem (1) is equivalent to solving the following problem (see Theorem C.1 in Appendix C):
Revised: We prove that solving constrained problem (1) is equivalent to solving the following problem (see Theorem C.1 in Appendix C):

Original: [Equation]
Revised: [Equation]

Original: To solve problem (3), we propose inexact Primal Dual Hybrid Gradient (PDHG) parallel updates for the primal-dual variable pair x , S x and dual-primal pair y , S y , illustrated in eq.
Revised: To solve problem (3), we propose inexact Primal Dual Hybrid Gradient (PDHG) parallel updates for the primal-dual variable pair x , S x and dual-primal pair y , S y , illustrated in eq.

Original: (P1) and (D1).
Revised: (P1) and (D1).

Original: [Equation]
Revised: [Equation]

Original: Note that in eq.
Revised: Note that in eq.

Original: (P1), ν x t +1 is found using a prox-linear step involving linearization of F ( x , y ) with respect to x and a penalized cost-to-move term 12 s (cid:107) x − x t (cid:107) 2 , followed by an ascent step to update the Lagrange dual variable S x .
Revised: (P1), ν x t +1 is found using a prox-linear step involving linearization of F ( x , y ) with respect to x and a penalized cost-to-move term 12 s (cid:107) x − x t (cid:107) 2 , followed by an ascent step to update the Lagrange dual variable S x .

Original: Then ˆ x t +1 is found using prox-linear step similar to the ﬁrst step but using the recent S x t +1 .
Revised: Then ˆ x t +1 is found using prox-linear step similar to the ﬁrst step but using the recent S x t +1 .

Original: Finally x t +1 is found by a prox step where prox sG ( x ) =
Revised: Finally x t +1 is found by a prox step where prox sG ( x ) =

Original: arg min u ∈ R mdx G ( u ) + 12 s (cid:107) u − x (cid:107) 2 .
Revised: arg min u ∈ R mdx G ( u ) + 12 s (cid:107) u − x (cid:107) 2 .

Original: Letting D x t
Revised: Letting D x t

Original: = ( U ⊗ I d x ) S x t , and pre-multiplying by U ⊗ I d x in the update step of S x , equations (P1) reduce to those in equations (P2).
Revised: = ( U ⊗ I d x ) S x t , and pre-multiplying by U ⊗ I d x in the update step of S x , equations (P1) reduce to those in equations (P2).

Original: Similarly, the updates to y , S y can be done using appropriate gradient ascent-descent steps which lead to corresponding equations (D1) and (D2).
Revised: Similarly, the updates to y , S y can be done using appropriate gradient ascent-descent steps which lead to corresponding equations (D1) and (D2).

Original: Updates to primal-dual pair x , S x :
Revised: Updates to primal-dual pair x , S x :

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: prox
Revised: prox

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: prox
Revised: prox

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: Inexact PDHG type updates are known for solving a Lagrangian function of an underlying convex minimization problem in single machine setting (e.g. proximal alternating predictor-corrector (PAPC) algorithm [7, 25], primal-dual ﬁxed point (PDFP) algorithm [8]), and in decentralized setting [20].
Revised: Inexact PDHG type updates are known for solving a Lagrangian function of an underlying convex minimization problem in single machine setting (e.g. proximal alternating predictor-corrector (PAPC) algorithm [7, 25], primal-dual ﬁxed point (PDFP) algorithm [8]), and in decentralized setting [20].

Original: In contrast to PAPC, PDFP, and the algorithm in [20], the type of inexact PDHG updates proposed in our work addresses a Lagrangian function corresponding to an underlying saddle point problem (SPP).
Revised: In contrast to PAPC, PDFP, and the algorithm in [20], the type of inexact PDHG updates proposed in our work addresses a Lagrangian function corresponding to an underlying saddle point problem (SPP).

Original: To our knowledge, our work is the ﬁrst to extend inexact PDHG type updates to solve Lagrangian of saddle point problem of form (SPP).
Revised: To our knowledge, our work is the ﬁrst to extend inexact PDHG type updates to solve Lagrangian of saddle point problem of form (SPP).

Original: For other perspectives and generalizations of inexact PDHG, see [6, 12, 9, 35, 38, 20].
Revised: For other perspectives and generalizations of inexact PDHG, see [6, 12, 9, 35, 38, 20].

Original: Observe that the term (( I m − W ) ⊗ I d x ) ν x
Revised: Observe that the term (( I m − W ) ⊗ I d x ) ν x

Original: t +1 in eq.
Revised: t +1 in eq.

Original: (P2) and (( I m − W ) ⊗ I d y )
Revised: (P2) and (( I m − W ) ⊗ I d y )

Original: ν
Revised: ν

Original: y t +1 in eq.
Revised: y t +1 in eq.

Original: (D2) denote the communication of ν x t +1 and ν y t +1 across the nodes.
Revised: (D2) denote the communication of ν x t +1 and ν y t +1 across the nodes.

Original: Further note that ν x t +1 and ν y t
Revised: Further note that ν x t +1 and ν y t

Original: +1 need to be communicated only once for updating D x t +1 , ˆ x t +1 and D y t
Revised: +1 need to be communicated only once for updating D x t +1 , ˆ x t +1 and D y t

Original: +1 , ˆ y t +1 in the inexact PDHG updates for x and y respectively.
Revised: +1 , ˆ y t +1 in the inexact PDHG updates for x and y respectively.

Original: This results in cheaper communication in every iteration when compared to the multiple communications that happen in a single iteration of the algorithms in [5, 36, 23].
Revised: This results in cheaper communication in every iteration when compared to the multiple communications that happen in a single iteration of the algorithms in [5, 36, 23].

Original: To improve the communication efﬁciency further, we propose to compress ν x t +1 and ν y t +1 .
Revised: To improve the communication efﬁciency further, we propose to compress ν x t +1 and ν y t +1 .

Original: We recall that compression has not yet been used in existing algorithms to solve (SPP) in decentalized setting without a central server.
Revised: We recall that compression has not yet been used in existing algorithms to solve (SPP) in decentalized setting without a central server.

Original: Compression based algorithms to solve smooth variational inequalities (related to problem (SPP)) are available only for decentralized settings with a central server [3] .
Revised: Compression based algorithms to solve smooth variational inequalities (related to problem (SPP)) are available only for decentralized settings with a central server [3] .

Original: We follow [27, 24] to compress a related difference vector instead of directly compressing ν x t + and ν y t +1 .
Revised: We follow [27, 24] to compress a related difference vector instead of directly compressing ν x t + and ν y t +1 .

Original: Each node i is assumed to maintain a local vector H i, x and a stochastic compression operator Q is applied on the difference vector
Revised: Each node i is assumed to maintain a local vector H i, x and a stochastic compression operator Q is applied on the difference vector

Original: ν
Revised: ν

Original: i, x t +1 − H
Revised: i, x t +1 − H

Original: i, x t .
Revised: i, x t .

Original: The concise form of these updates is illustrated in Algorithm 4 (COMM procedure) in Appendix B. Algorithm 1 illustrates the proposed inexact PDHG updates with compression.
Revised: The concise form of these updates is illustrated in Algorithm 4 (COMM procedure) in Appendix B. Algorithm 1 illustrates the proposed inexact PDHG updates with compression.

Original: Thus the inexact PDHG update steps to obtain ν x t +1 and ν y t
Revised: Thus the inexact PDHG update steps to obtain ν x t +1 and ν y t

Original: +1 involve computing the gradients ∇ x F ( x t , y t )
Revised: +1 involve computing the gradients ∇ x F ( x t , y t )

Original: =
Revised: =

Original: (cid:0) ∇ x f 1 ( x 1 , y 1 ) , . .
Revised: (cid:0) ∇ x f 1 ( x 1 , y 1 ) , . .

Original: , ∇ x f m ( x m , y m ) (cid:1) and ∇ y F ( x t , y t ) = (cid:0) ∇ y f 1 ( x 1 , y 1 ) , . .
Revised: , ∇ x f m ( x m , y m ) (cid:1) and ∇ y F ( x t , y t ) = (cid:0) ∇ y f 1 ( x 1 , y 1 ) , . .

Original: . , ∇ y f m ( x m , y m ) (cid:1) using a gradient computation oracle G .
Revised: . , ∇ y f m ( x m , y m ) (cid:1) using a gradient computation oracle G .

Original: We now discuss methods based on two different stochastic gradient oracles to compute the gradients.
Revised: We now discuss methods based on two different stochastic gradient oracles to compute the gradients.

Original: Before discussing the methods, we state technical assumptions common to both the methods.
Revised: Before discussing the methods, we state technical assumptions common to both the methods.

Original:  Assumptions
Revised:  Assumptions

Original: We list below the assumptions to be used throughout this work.
Revised: We list below the assumptions to be used throughout this work.

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: Each f i ( · , y ) is µ x -strongly convex for every y ∈ Y ; hence for any x 1 , x 2 ∈ X and ﬁxed y ∈ Y , it holds: f i ( x 1 , y ) ≥ f i ( x 2 , y )
Revised: Each f i ( · , y ) is µ x -strongly convex for every y ∈ Y ; hence for any x 1 , x 2 ∈ X and ﬁxed y ∈ Y , it holds: f i ( x 1 , y ) ≥ f i ( x 2 , y )

Original: + (cid:104)∇ x f i ( x 2 , y ) , x 1 − x 2 (cid:105) + µ x 2 (cid:107) x 1 − x 2 (cid:107) 2 .
Revised: + (cid:104)∇ x f i ( x 2 , y ) , x 1 − x 2 (cid:105) + µ x 2 (cid:107) x 1 − x 2 (cid:107) 2 .

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: Each f i ( x, · ) is µ y -strongly concave for every x ∈ X ; hence for any y 1 , y 2 ∈ Y and ﬁxed x ∈ X , it holds: f i ( x, y 1 ) ≤
Revised: Each f i ( x, · ) is µ y -strongly concave for every x ∈ X ; hence for any y 1 , y 2 ∈ Y and ﬁxed x ∈ X , it holds: f i ( x, y 1 ) ≤

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.3.
Revised: Assumption 3.3.

Original: g ( x ) and r ( y ) are proper, convex, continuous and possibly non-smooth functions.
Revised: g ( x ) and r ( y ) are proper, convex, continuous and possibly non-smooth functions.

Original: Assumption 3.4.
Revised: Assumption 3.4.

Original: The compression operator Q satisﬁes the following for every u ∈ R d : (i) Q ( u ) is an unbiased estimate of u : E [ Q ( u )]
Revised: The compression operator Q satisﬁes the following for every u ∈ R d : (i) Q ( u ) is an unbiased estimate of u : E [ Q ( u )]

Original: = u (ii) E
Revised: = u (ii) E

Original: [ (cid:107) Q ( u ) − u (cid:107) 2 ] ≤ δ (cid:107) u (cid:107) 2 , where the constant δ ≥denotes the amount of compression induced by operator Q and is called a compression factor.
Revised: [ (cid:107) Q ( u ) − u (cid:107) 2 ] ≤ δ (cid:107) u (cid:107) 2 , where the constant δ ≥denotes the amount of compression induced by operator Q and is called a compression factor.

Original: When δ = 0 , Q achieves no compression.
Revised: When δ = 0 , Q achieves no compression.

Original: Assumption 3.5.
Revised: Assumption 3.5.

Original: The weight matrix W satisﬁes the following conditions: W is symmetric and row stochastic, W ij > 0 if and only if ( i, j
Revised: The weight matrix W satisﬁes the following conditions: W is symmetric and row stochastic, W ij > 0 if and only if ( i, j

Original: ) ∈ E and W ii > 0 for all i ∈ [ m ] .
Revised: ) ∈ E and W ii > 0 for all i ∈ [ m ] .

Original: The eigenvalues of W denoted by λ 1 , . . .
Revised: The eigenvalues of W denoted by λ 1 , . . .

Original: , λ m satisfy: − 1 < λ m ≤ λ m − 1 ≤ . . .
Revised: , λ m satisfy: − 1 < λ m ≤ λ m − 1 ≤ . . .

Original: ≤ λ 2 < λ 1 = 1 .
Revised: ≤ λ 2 < λ 1 = 1 .

Original: Note that Assumptions 3.1-3.3 and Assumption 3.5 are standard in the study of saddle point problems (e.g. [4, 5, 29, 23]).
Revised: Note that Assumptions 3.1-3.3 and Assumption 3.5 are standard in the study of saddle point problems (e.g. [4, 5, 29, 23]).

Original: We also note that Assumption 3.4 is standard in existing works (e.g. [1, 20, 24]).
Revised: We also note that Assumption 3.4 is standard in existing works (e.g. [1, 20, 24]).

Original: For example, b -bits quantization operator (see Appendix I ) satisﬁes Assumption 3.4.
Revised: For example, b -bits quantization operator (see Appendix I ) satisﬁes Assumption 3.4.

Original:  General Stochastic Setting
Revised:  General Stochastic Setting

Original: In the general stochastic setting, we allow for availability of heterogeneous data distributions in each node and assume that the gradients ∇ x f i ( x, y ) and ∇ y f i ( x, y ) are computed using an oracle G described below.
Revised: In the general stochastic setting, we allow for availability of heterogeneous data distributions in each node and assume that the gradients ∇ x f i ( x, y ) and ∇ y f i ( x, y ) are computed using an oracle G described below.

Original: General Stochastic Gradient Oracle (GSGO):
Revised: General Stochastic Gradient Oracle (GSGO):

Original: Sample a mini-batch of samples ξ i ∼ D
Revised: Sample a mini-batch of samples ξ i ∼ D

Original: i in each node i , where D i is the data distribution local to node i .
Revised: i in each node i , where D i is the data distribution local to node i .

Original: Compute stochastic gradients: G i,x = ∇ x f i ( x i , y i ; ξ i ) and G i,y = ∇ y f i ( x i , y i ; ξ i ) .
Revised: Compute stochastic gradients: G i,x = ∇ x f i ( x i , y i ; ξ i ) and G i,y = ∇ y f i ( x i , y i ; ξ i ) .

Original: Inspired by restart based schemes in single machine setting [39, 40], we design in this work, a restart based stochastic gradient method illustrated in Algorithm 2, which invokes in every iteration k , a sequence of t k primal and dual variable updates using inexact PDHG with compression (Algorithm 1).
Revised: Inspired by restart based schemes in single machine setting [39, 40], we design in this work, a restart based stochastic gradient method illustrated in Algorithm 2, which invokes in every iteration k , a sequence of t k primal and dual variable updates using inexact PDHG with compression (Algorithm 1).

Original: However, the restart scheme proposed in Algorithm 2 is simpler than that in [40], where the restart scheme in single machine setting requires the computation of Fenchel conjugate at every restart incurring additional O ( d y ) operations.
Revised: However, the restart scheme proposed in Algorithm 2 is simpler than that in [40], where the restart scheme in single machine setting requires the computation of Fenchel conjugate at every restart incurring additional O ( d y ) operations.

Original: Our scheme is also different in design compared to other restart based schemes studied for saddle point problems in single machine setting [42, 13, 22].
Revised: Our scheme is also different in design compared to other restart based schemes studied for saddle point problems in single machine setting [42, 13, 22].

Original: In
Revised: In

Original: Algorithm 2, step length s k is chosen to decrease geometrically only at every restart step k , resulting in better convergence.
Revised: Algorithm 2, step length s k is chosen to decrease geometrically only at every restart step k , resulting in better convergence.

Original: The details of derivations of parameter ρ , step lengths γ xk , γ yk and parameters α x,k , α y,k used in COMM procedure are provided in Appendix E.4.
Revised: The details of derivations of parameter ρ , step lengths γ xk , γ yk and parameters α x,k , α y,k used in COMM procedure are provided in Appendix E.4.

Original: Under appropriate assumptions on unbiasedness and smoothness of the stochastic gradients (see
Revised: Under appropriate assumptions on unbiasedness and smoothness of the stochastic gradients (see

Original: Appendix E), we have the following convergence result of Algorithm 2.
Revised: Appendix E), we have the following convergence result of Algorithm 2.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Suppose { x k, 0 } k and { y k, 0 } k are the sequences generated by Algorithm 2.
Revised: Suppose { x k, 0 } k and { y k, 0 } k are the sequences generated by Algorithm 2.

Original: Then with at most
Revised: Then with at most

Revised: K

Original: [Equation]
Revised: [Equation]

Original: iterations, E [
Revised: iterations, E [

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: 2 +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:63) (cid:107) 2 ] ≤ (cid:15) , where σ 2 is the local variance bound in stochastic gradients (see Assumption E.1 in Appendix E).
Revised: 2 ] ≤ (cid:15) , where σ 2 is the local variance bound in stochastic gradients (see Assumption E.1 in Appendix E).

Original: Moreover the total gradient computation complexity and communication complexity to achieve (cid:15) -accurate saddle point solution in expectation are
Revised: Moreover the total gradient computation complexity and communication complexity to achieve (cid:15) -accurate saddle point solution in expectation are

Original: [Equation]
Revised: [Equation]

Original: and T 
Revised: and T comm

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: + K 
Revised: + K ( (cid:15) ) respectively.

Original: [Equation]
Revised: [Equation]

Original: ) respectively.

Original: [Equation]
Revised: [Equation]

Original: C-RDPSG)

Original: [Equation]
Revised: [Equation]

Original: using ,  number  iterations K [Equation] [Equation]
Revised: , G obtained using GSGO, number of iterations K , ρ depending on κ f , κ g and δ .

Original: : for k = 0 to K − 1 do
Revised: 2: for k = 0 to K − 1 do

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: : for t = 0 to t k − 1 do 10:
Revised: 9: for t = 0 to t k − 1 do 10:

Original: [Equation]
Revised: [Equation]

Original: 1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t + = IPDHG ( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s k , γ xk , γ yk , α x,k , α y,k , G ) 11: end for 12: x k +1 , 

Revised: 1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t + = IPDHG ( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s k , γ xk , γ yk , α x,k , α y,k , G ) 11: end for 12: x k +1 ,

Original: = x k,t k , y k +1 , 0 = y k,t k 13: end for 14: RETURN: x K, 0 , y K, 0 .
Revised: = x k,t k , y k +1 , 0 = y k,t k 13: end for 14: RETURN: x K, 0 , y K, 0 .

Original: Due to space considerations, we discuss proof details in Appendix F.
Revised: Due to space considerations, we discuss proof details in Appendix F.

Original: We note that the number of outer iterates K ( (cid:15) ) in Theorem 4.1 depend logarithmically on κ g , δ and total variance bound mσ 2 .
Revised: We note that the number of outer iterates K ( (cid:15) ) in Theorem 4.1 depend logarithmically on κ g , δ and total variance bound mσ 2 .

Original: Larger values of these parameters contribute to the accumulation of consensus, compression and gradient approximation errors.
Revised: Larger values of these parameters contribute to the accumulation of consensus, compression and gradient approximation errors.

Original: Hence more restarts might be required to reduce these errors accumulated during IPDHG updates with GSGO (see Figures 6, 4, 8 and 10 in Appendix I for empirical evidence of this fact).
Revised: Hence more restarts might be required to reduce these errors accumulated during IPDHG updates with GSGO (see Figures 6, 4, 8 and 10 in Appendix I for empirical evidence of this fact).

Revised: The computation complexity in Theorem 4.1 depends on compression factor and graph condition number as O ( δ 4 ) and O ( κ 2 g ) .

Revised: The dependence on graph condition number reduces to O ( κ g ) when σ = 0 .

Revised: Therefore, in the deterministic gradients regime, C-RDPSG is less sensitive to change in network topology compared to stochastic gradients regime.

Revised: We note that [17] develops non-compression based optimal algorithms with complexity O

Original:   Finite Sum Setting
Revised: ( κ f √ κ g log(1 /(cid:15) )) for solving decentralized ﬁnite-sum variational inequalities. However, Theorem 4.1 explains the complexity results of C-RDPSG in the general stochastic setting.  Finite Sum Setting

Original: In the ﬁnite sum setting, we assume that each local function f i ( x, y ) is of the form 1 n (cid:80) nj =1 f ij ( x, y ) .
Revised: In the ﬁnite sum setting, we assume that each local function f i ( x, y ) is of the form 1 n (cid:80) n j =1 f ij ( x, y ) .

Original: For simplicity, we assume that each node i has same number of batches n .
Revised: For simplicity, we assume that each node i has same number of batches n .

Original: However, our analysis easily extends to different number of batches n i .
Revised: However, our analysis easily extends to different number of batches n i .

Original: Let N denote the total number of samples.
Revised: Let N denote the total number of samples.

Original: Let
Revised: Let

Original: B and N (cid:96) respectively denote the batch size and number of local samples at each node i .
Revised: B and N (cid:96) respectively denote the batch size and number of local samples at each node i .

Original: The number of samples in the function component f ij is determined by the batch size B = N (cid:96) /n .
Revised: The number of samples in the function component f ij is determined by the batch size B = N (cid:96) /n .

Original: Let
Revised: Let

Original: P i = { p il : l ∈ { 1 , 2 , . . . , n }} denote a probability distribution where p il is the probability with which batch l is sampled at node i .
Revised: P i = { p il : l ∈ { 1 , 2 , . . . , n }} denote a probability distribution where p il is the probability with which batch l is sampled at node i .

Original: Let p min :
Revised: Let p min :

Original: =
Revised: =

Original: min i,l p il .
Revised: min i,l p il .

Original: Without loss of generality we assume that p min > 0 , hence each batch is chosen with a positive probability.
Revised: Without loss of generality we assume that p min > 0 , hence each batch is chosen with a positive probability.

Original: Note that GSGO in Algorithmshows sublinear convergence for solving (SPP).
Revised: Note that GSGO in Algorithmshows sublinear convergence for solving (SPP).

Original: Stochastic variance reduction techniques [14, 18] are known to accelerate convergence of GSGO based methods for decentralized convex minimization problems [20, 37].
Revised: Stochastic variance reduction techniques [14, 18] are known to accelerate convergence of GSGO based methods for decentralized convex minimization problems [20, 37].

Original: Inspired by this success, we propose a Stochastic Variance Reduced Gradient (SVRG) oracle comprising the following steps.
Revised: Inspired by this success, we propose a Stochastic Variance Reduced Gradient (SVRG) oracle comprising the following steps.

Original: Stochastic Variance Reduced Gradient Oracle (SVRGO): (1).
Revised: Stochastic Variance Reduced Gradient Oracle (SVRGO): (1).

Original: Index sampling: Sample l  { 1 , 2 , . . . , n } ∼ P i for every node i .
Revised: Index sampling: Sample l ∈ { 1 , 2 , . . . , n } ∼ P i for every node i .

Original: Stochastic gradient computation with variance reduction: For a reference point ˜
Revised: Stochastic gradient computation with variance reduction: For a reference point ˜

Original: z i =
Revised: z i =

Original: (˜ x i , ˜ y i ) , compute stochastic gradients at z i =
Revised: (˜ x i , ˜ y i ) , compute stochastic gradients at z i =

Original: ( x i , y i ) with respect to x and y as follows:
Revised: ( x i , y i ) with respect to x and y as follows:

Original: [Equation]
Revised: [Equation]

Original: Reference point update: Sample ω ∈ { 0 , 1 } ∼ Bernoulli ( p ) and update the reference points as follows:
Revised: Reference point update: Sample ω ∈ { 0 , 1 } ∼ Bernoulli ( p ) and update the reference points as follows:

Original: [Equation]
Revised: [Equation]

Original: The SVRGO setup above requires computation of full-batch gradient at the reference point periodically (equations (4)-(5)), but is memory-efﬁcient compared to other variance reduction schemes (e.g.
Revised: The SVRGO setup above requires computation of full-batch gradient at the reference point periodically (equations (4)-(5)), but is memory-efﬁcient compared to other variance reduction schemes (e.g.

Original: SAGA [10]).
Revised: SAGA [10]).

Original: Note also that the SVRGO based algorithm for single machine setting in [30] is for a differently structured problem than (SPP).
Revised: Note also that the SVRGO based algorithm for single machine setting in [30] is for a differently structured problem than (SPP).

Original: The C-DPSVRG methodology using SVRGO is illustrated in Algorithm 3 with step sizes deﬁned in Appendix G.
Revised: The C-DPSVRG methodology using SVRGO is illustrated in Algorithm 3 with step sizes deﬁned in Appendix G.

Original: Algorithm 3 Decentralized P roximal S tochastic V ariance R eduction method with Compression (C-DPSVRG) 1:
Revised: Algorithm 3 D

Original: [Equation]
Revised: [Equation]

Original: = µnp min 24 L 2 , α x , α y , γ x , γ y , G SVRGO. parallel all i do 3: x t +1 , y t
Revised: [Equation]

Original: +1 , D xt +1 ,
Revised: [Equation]

Original: D yt +1 , H xt +1 , H yt +1 , H w,xt +1 , H w,yt + = IPDHG ( x t , y t , D xt , D yt , H xt , H yt , H w,xt , H w,yt , s, γ x , γ y , α x , α y , G )
Revised: [Equation]

Original: Under suitable assumptions on smoothness of mini-batch gradients (see Appendix G), the convergence behavior of Algorithm 3 is given in the following result.
Revised: Under suitable assumptions on smoothness of mini-batch gradients (see Appendix G), the convergence behavior of Algorithm 3 is given in the following result.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Let { x t } t , { y t } t be the sequences generated by Algorithm 3.
Revised: Let { x t } t , { y t } t be the sequences generated by Algorithm 3.

Original: Suppose Assumptions 3.1-3.5 and Assumptions G.1-G.4 hold.
Revised: Suppose Assumptions 3.1-3.5 and Assumptions G.1-G.4 hold.

Original: Then computational and communication complexity of algorithm 3 for achieving (cid:15) -accurate saddle point solution in expectation are
Revised: Then computational and communication complexity of algorithm 3 for achieving (cid:15) -accurate saddle point solution in expectation are

Original: [Equation]
Revised: [Equation]

Original: where ˜Φ 0 denotes the distance of the initial values x 0 , y 0 , D x 0 , D y 0 , H x 0 , H y 0 from their respective limit points (described in equation (215) in Appendix G).
Revised: where ˜Φ 0 denotes the distance of the initial values x 0 , y 0 , D x 0 , D y 0 , H x 0 , H y 0 from their respective limit points (described in equation (215) in Appendix G).

Original: Due to space considerations, we have given proof details of Theorem 5.1 in Appendix H.
Revised: Due to space considerations, we have given proof details of Theorem 5.1 in Appendix H.

Revised: The complexity of C-DPSVRG depends on κ f , κ g and δ as O ((1 + δ ) max {√ δκ 2 f κ g , κ g , κ 2 f } ) .

Revised: However, the optimal complexity for solving decentralized saddle point problems without compression is shown to be of order O

Revised: ( κ f √ κ g ) in [17].

Revised: Such optimal dependence on κ g is obtained at the cost of multiple communication rounds per iterate unlike one communication round per iterate in CDPSVRG.

Original:  The following corollary of Theorem 5.1 gives particular settings of number of batches n and reference probability p yielding factors of the form √ N (cid:96) + N (cid:96) for total gradient computations per node, which resemble the factors in the corresponding complexity results for optimal algorithms to solve decentralized variational inequalities without compression [17].
Revised: When there is no compression ( δ = 0 ) in C-DPSVRG, the complexity reduces to O (cid:16) max { κ g , κ 2 f np min , 2 p } log (cid:16) ˜Φ 0 (cid:15) (cid:17) (cid:17) . We note that the complexity results in Theorem 5.1 are not optimal, however we observe convergence speedup for C-DPSVRG over baseline methods in our empirical study. The following corollary of Theorem 5.1 gives particular settings of number of batches n and reference probability p yielding factors of the form √ N (cid:96) + N (cid:96) for total gradient computations per node, which resemble the factors in the corresponding complexity results for optimal algorithms to solve decentralized variational inequalities without compression [17].

Original: Corollary 5.2.
Revised: Corollary 5.2.

Original: Under the setting of Theorem 5.1, choose n > √ N (cid:96) , p = B/ √ N (cid:96) and p min = √
Revised: Under the setting of Theorem 5.1, choose n > √ N (cid:96) , p = B/ √ N (cid:96) and p min = √

Original: N (cid:96) / 2 n 2 .
Revised: N (cid:96) / 2 n 2 .

Original: Then the total number of gradient computations per node to achieve (cid:15) -accurate saddle point solution is of order O (( √ N(cid:96) + N (cid:96) ) max {√ δ (1 + δ )
Revised: Then the total number of gradient computations per node to achieve (cid:15) -accurate saddle point solution is of order O (( √ N

Original: κ g κ 2 f , (1+ δ ) κ g, (1 + δ ) κ 2 f } log ( ˜Φ 0 (cid:15) )) .
Revised: [Equation]

Original:  Conclusion
Revised:  Conclusion

Original: We have proposed two stochastic gradient algorithms for decentralized optimization for saddle point problems, with compression.
Revised: We have proposed two stochastic gradient algorithms for decentralized optimization for saddle point problems, with compression.

Original: Both the algorithms offer practical advantages and are shown to have rigorous theoretical guarantees.
Revised: Both the algorithms offer practical advantages and are shown to have rigorous theoretical guarantees.

Original: It would be interesting to adapt both C-RDPSG and C-DPSVRG to cases where some of the constants are unknown in the problem setup.
Revised: It would be interesting to adapt both C-RDPSG and C-DPSVRG to cases where some of the constants are unknown in the problem setup.

Original: A Related Work
Revised: A Related Work

Original: Decentralized minimization: We provide here only a very brief survey of literature on decentralized minimization (see e.g. [20] for more comprehensive survey).
Revised: Decentralized minimization: We provide here only a very brief survey of literature on decentralized minimization (see e.g. [20] for more comprehensive survey).

Original: A simple distributed deterministic gradient method with sublinear convergence rate guarantees was proposed in [32].
Revised: A simple distributed deterministic gradient method with sublinear convergence rate guarantees was proposed in [32].

Original: Distributed gradient methods using gradient tracking [34, 31] are known to converge to an exact optimal value using constant step size.
Revised: Distributed gradient methods using gradient tracking [34, 31] are known to converge to an exact optimal value using constant step size.

Original: Stochastic gradient oracles for decentralized minimization problems proposed in [28], have recently been extended to design efﬁcient compression based decentralized methods for solving convex minimization problems with provable theoretical guarantees [15, 24, 20].
Revised: Stochastic gradient oracles for decentralized minimization problems proposed in [28], have recently been extended to design efﬁcient compression based decentralized methods for solving convex minimization problems with provable theoretical guarantees [15, 24, 20].

Original: More recent works are proposed for non-convex optimization in decentralized settings [41]
Revised: More recent works are proposed for non-convex optimization in decentralized settings [41]

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: re
Revised: re

Original: [Table]
Revised: [Table]

Original: κ 4 / 3 g log (cid:0) 1 (cid:15) (cid:1) (cid:17)
Revised: κ 4 / 3 g log (cid:0) 1 (cid:15) (cid:1) (cid:17)

Original: DSPAwLA[26] (cid:55) (cid:51)
Revised: DSPAwLA[26] (cid:55) (cid:51)

Original: C-C
Revised: C-C

Original: O (1 /(cid:15) 2 )
Revised: O (1 /(cid:15) 2 )

Original: O (1 /(cid:15) 2 )
Revised: O (1 /(cid:15) 2 )

Original: DMHSGD [36] (cid:51) (cid:55)
Revised: DMHSGD [36] (cid:51) (cid:55)

Original: NC-SC
Revised: NC-SC

Original: O
Revised: O

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 − λ 2 ( W ))
Revised: 1 − λ 2 ( W ))

Original: [Equation]
Revised: [Equation]

Original: (cid:15) 3 )
Revised: (cid:15) 3 )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: λ 2 ( W ))
Revised: λ 2 ( W ))

Original: [Equation]
Revised: [Equation]

Original: C-RDPSG (cid:51) (cid:51) (Theorem 4.1)
Revised: C-RDPSG (cid:51) (cid:51) (Theorem 4.1)

Original: [Equation]
Revised: [Equation]

Original: GSS)
Revised: GSS)

Original: C-
Revised: C-

Original: [Equation]
Revised: [Equation]

Original: } (Theorem 5.1)
Revised: } (Theorem 5.1)

Original: [Equation]
Revised: [Equation]

Original: )) log
Revised: )) log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (FSS)
Revised: (FSS)

Original: C o m p re ss i o n
Revised: C o m p re ss i o n

Original: N o
Revised: N o

Original: G o ss i p
Revised: G o ss i p

Original: C-RDPSG (cid:51) (cid:51) (Theorem 4.1)
Revised: C-RDPSG (cid:51) (cid:51) (Theorem 4.1)

Original: [Equation]
Revised: [Equation]

Original: ) (GSS)
Revised: ) (GSS)

Original: C-
Revised: C-

Original: [Equation]
Revised: [Equation]

Original: (Theorem 5.1)
Revised: (Theorem 5.1)

Original: [Equation]
Revised: [Equation]

Original: ( FSS )
Revised: ( FSS )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: log(
Revised: log(

Original: [Equation]
Revised: [Equation]

Original: Decentralized saddle-point problems: A distributed saddle-point algorithm with Laplacian averaging (DSPAwLA) in [26], based on gradient descent ascent updates to solve non-smooth convexconcave saddle point problems achieves O (1 /(cid:15) 2 ) convergence rate.
Revised: Decentralized saddle-point problems: A distributed saddle-point algorithm with Laplacian averaging (DSPAwLA) in [26], based on gradient descent ascent updates to solve non-smooth convexconcave saddle point problems achieves O (1 /(cid:15) 2 ) convergence rate.

Original: DSPAwLA uses consensus constrained formulation of saddle point problem.
Revised: DSPAwLA uses consensus constrained formulation of saddle point problem.

Original: DSPAwLA is obtained by incorporating (cid:96) 2 norm based penalty of consensus constraints into the objective function and employing gradient descent ascent scheme to the resultant penalized objective.
Revised: DSPAwLA is obtained by incorporating (cid:96) 2 norm based penalty of consensus constraints into the objective function and employing gradient descent ascent scheme to the resultant penalized objective.

Original: However, our work proposes an equivalent
Revised: However, our work proposes an equivalent

Original: Lagrangian formulation of consensus constrained saddle point problem (1) and updates primal-dual variables using a variant of primal dual hybrid method.
Revised: Lagrangian formulation of consensus constrained saddle point problem (1) and updates primal-dual variables using a variant of primal dual hybrid method.

Original: An extragradient method with gradient tracking (GT-EG) proposed in [29] is shown to have linear convergence rates for solving decentralized strongly convex strongly concave problems, under a positive lower bound assumption on the gradient difference norm.
Revised: An extragradient method with gradient tracking (GT-EG) proposed in [29] is shown to have linear convergence rates for solving decentralized strongly convex strongly concave problems, under a positive lower bound assumption on the gradient difference norm.

Original: However such assumptions might not hold for problems without bilinear structure.
Revised: However such assumptions might not hold for problems without bilinear structure.

Original: Both [29] and [26] are based on non-compression based communications and full batch gradient computations which limit their applicability to large scale machine learning problems.
Revised: Both [29] and [26] are based on non-compression based communications and full batch gradient computations which limit their applicability to large scale machine learning problems.

Original: Recently, multiple works [23, 36, 4] have proposed using minibatch gradients for solving decentralized saddle point problems.
Revised: Recently, multiple works [23, 36, 4] have proposed using minibatch gradients for solving decentralized saddle point problems.

Original: Decentralized extra step (DES) [4] shows linear communication complexity with dependence on the graph condition number as √ κ g , obtained at the cost of incorporating multiple rounds of communication of primal and dual updates.
Revised: Decentralized extra step (DES) [4] shows linear communication complexity with dependence on the graph condition number as √ κ g , obtained at the cost of incorporating multiple rounds of communication of primal and dual updates.

Original: A near optimal distributed Min-Max data similarity (MMDS) algorithm is proposed in [5] for saddle point problems with a suitable data similarity assumption.
Revised: A near optimal distributed Min-Max data similarity (MMDS) algorithm is proposed in [5] for saddle point problems with a suitable data similarity assumption.

Original: MMDS is based on full batch gradient computations and requires solving an inner saddle point problem at every iteration.
Revised: MMDS is based on full batch gradient computations and requires solving an inner saddle point problem at every iteration.

Original: MMDS allows for communication efﬁciency by choosing only one node uniformly at random to update the iterates.
Revised: MMDS allows for communication efﬁciency by choosing only one node uniformly at random to update the iterates.

Original: However, every node computes the full batch gradient before heading to next gradient based updates.
Revised: However, every node computes the full batch gradient before heading to next gradient based updates.

Original: Moreover, this scheme employs accelerated gossip [21] multiple times to propagate the gradients and model updates to the entire network.
Revised: Moreover, this scheme employs accelerated gossip [21] multiple times to propagate the gradients and model updates to the entire network.

Original: Communication complexity of MMDS is shown to depend on eigengap of weight matrix
Revised: Communication complexity of MMDS is shown to depend on eigengap of weight matrix

Original: W , while gradient computation complexity is not investigated.
Revised: W , while gradient computation complexity is not investigated.

Original: Decentralized parallel optimistic stochastic gradient method (DPOSG) was proposed in [23] for nonconvex-nonconcave saddle point problems.
Revised: Decentralized parallel optimistic stochastic gradient method (DPOSG) was proposed in [23] for nonconvex-nonconcave saddle point problems.

Original: This method involves local model averaging step (multiple communication rounds) to reduce the effect of consensus error.
Revised: This method involves local model averaging step (multiple communication rounds) to reduce the effect of consensus error.

Original: A gradient tracking based algorithm called DMHSGD for solving nonconvex-strongly concave saddle point problems proposed in [36], uses a large mini-batch at the ﬁrst iteration and requires the nodes to communicate both model and gradient updates, to achieve better aggregates of quantities.
Revised: A gradient tracking based algorithm called DMHSGD for solving nonconvex-strongly concave saddle point problems proposed in [36], uses a large mini-batch at the ﬁrst iteration and requires the nodes to communicate both model and gradient updates, to achieve better aggregates of quantities.

Original: Variance reduction based optimal methods to solve strongly convex-strongly concave nonsmooth ﬁnite sum variational inequalities are developed in [17].
Revised: Variance reduction based optimal methods to solve strongly convex-strongly concave nonsmooth ﬁnite sum variational inequalities are developed in [17].

Original: The improvement of complexity on graph condition number κ g in [17] is achieved using an accelerated gossip scheme.
Revised: The improvement of complexity on graph condition number κ g in [17] is achieved using an accelerated gossip scheme.

Original: However, C-DPSVRG does not involve any gossip scheme and hence yields cheaper communication per iterate.
Revised: However, C-DPSVRG does not involve any gossip scheme and hence yields cheaper communication per iterate.

Original: The complexity of C-DPSVRG and C-RDPSG does not have optimal dependence on κ f , κ g
Revised: The complexity of C-DPSVRG and C-RDPSG does not have optimal dependence on κ f , κ g

Original: and we leave it for future work.
Revised: and we leave it for future work.

Original: Table 1 positions our work in the context of existing methods.
Revised: Table 1 positions our work in the context of existing methods.

Original: B Compression Algorithm of [24]
Revised: B Compression Algorithm of [24]

Original: We follow [24, 27] to compress a related difference vector instead of directly compressing ν x t +1 and ν y t +1 .
Revised: We follow [24, 27] to compress a related difference vector instead of directly compressing ν x t +1 and ν y t +1 .

Original: We now describe the compression related updates for ν x t +1 .
Revised: We now describe the compression related updates for ν x t +1 .

Original: Each node i is assumed to maintain a local vector H i, x and a stochastic compression operator Q is applied on the difference vector
Revised: Each node i is assumed to maintain a local vector H i, x and a stochastic compression operator Q is applied on the difference vector

Original: ν
Revised: ν

Original: i, x t +1 − H
Revised: i, x t +1 − H

Original: i, x t .
Revised: i, x t .

Original: Hence the compressed estimate ˆ ν i, x t +1 of ν
Revised: Hence the compressed estimate ˆ ν i, x t +1 of ν

Original: i, x t +1 is obtained by adding the local vector and the compressed difference vector using ˆ ν i, x t
Revised: i, x t +1 is obtained by adding the local vector and the compressed difference vector using ˆ ν i, x t

Original: +

Revised: +

Original: =
Revised: =

Original: H
Revised: H

Original: i, x t + Q
Revised: i, x t + Q

Original: ( ν i, x t +1 − H
Revised: ( ν i, x t +1 − H

Original: i, x t ) .
Revised: i, x t ) .

Original: The local vector H i, x t is then updated using a convex combination of the previous local vector information and the new estimate
Revised: The local vector H i, x t is then updated using a convex combination of the previous local vector information and the new estimate

Original: ˆ ν i, x t +1 using H i, x t +1 =
Revised: ˆ ν i, x t +1 using H i, x t +1 =

Original: (1 − α )
Revised: (1 − α )

Original: H i, x t + α ˆ ν
Revised: H i, x t + α ˆ ν

Original: i, x t +1 for a suitable α ∈ [0 , 1] .
Revised: i, x t +1 for a suitable α ∈ [0 , 1] .

Original: Collecting the quantities in individual nodes into H x t +

Revised: Collecting the quantities in individual nodes into H x t +

Original: = ( H 1 , x t +1 , . . .
Revised: = ( H 1 , x t +1 , . . .

Original: , H m, x t +1 ) and ν x t +1 =
Revised: , H m, x t +1 ) and ν x t +1 =

Original: ( ν 1 , x t +1 , . . .
Revised: ( ν 1 , x t +1 , . . .

Original: , ν m, x t +1 ) , the update step can be written as H x t +1 =
Revised: , ν m, x t +1 ) , the update step can be written as H x t +1 =

Original: (1 − α )
Revised: (1 − α )

Original: H x t + α ˆ ν x t +1 .
Revised: H x t + α ˆ ν x t +1 .

Original: Pre-multiplying both sides of H x t +1 update by W ⊗ I , and denoting ( W ⊗ I )
Revised: Pre-multiplying both sides of H x t +1 update by W ⊗ I , and denoting ( W ⊗ I )

Original: H x t by H w, x t , and ( W ⊗ I )
Revised: H x t by H w, x t , and ( W ⊗ I )

Original: ˆ ν x t by ˆ ν w, x t we have: H w, x t +

Revised: ˆ ν x t by ˆ ν w, x t we have: H w, x t +

Original: =
Revised: =

Original: (1 − α )
Revised: (1 − α )

Original: H w, x t + α ˆ ν w, x t +1 .
Revised: H w, x t + α ˆ ν w, x t +1 .

Original: ˆ ν w, x t +1 can be further simpliﬁed as ˆ ν w, x t +1 =
Revised: ˆ ν w, x t +1 can be further simpliﬁed as ˆ ν w, x t +1 =

Original: ( W ⊗ I )
Revised: ( W ⊗ I )

Original: ˆ ν x
Revised: ˆ ν x

Original: t +1 =
Revised: t +1 =

Original: ( W ⊗ I )
Revised: ( W ⊗ I )

Original: ( H x t + Q ( ν x t +1 − H x t ))
Revised: ( H x t + Q ( ν x t +1 − H x t ))

Original: =
Revised: =

Original: H w, x t + ( W ⊗ I ) Q
Revised: H w, x t + ( W ⊗ I ) Q

Original: ( ν x t +1 − H x t ) .
Revised: ( ν x t +1 − H x t ) .

Original: A similar update scheme is used for compressing ν y t +1 .
Revised: A similar update scheme is used for compressing ν y t +1 .

Original: The entire procedure is illustrated in Algorithm 4, where we have used ν t +

Revised: The entire procedure is illustrated in Algorithm 4, where we have used ν t +

Original: = ( ν x t +1 , ν y t +1 ) , H t
Revised: = ( ν x t +1 , ν y t +1 ) , H t

Original: =
Revised: =

Original: ( H x t , H y t ) ,
Revised: ( H x t , H y t ) ,

Original: H wt =
Revised: H wt =

Original: ( H w, x t , H w, y t ) .
Revised: ( H w, x t , H w, y t ) .

Original: Further recall that ν x t +1 =
Revised: Further recall that ν x t +1 =

Original: ( ν 1 , x t +1 , . . .
Revised: ( ν 1 , x t +1 , . . .

Original: , ν m, x t +1 ) denotes the collection of the local variables at m nodes.
Revised: , ν m, x t +1 ) denotes the collection of the local variables at m nodes.

Original: Similar is the case for the other variables ν y t +1 , H x t , H y t , H w, x t , H w, y t .
Revised: Similar is the case for the other variables ν y t +1 , H x t , H y t , H w, x t , H w, y t .

Original: C Basic Results and Inequalities
Revised: C Basic Results and Inequalities

Original: Equivalence between problems (1) and (3) .
Revised: Equivalence between problems (1) and (3) .

Original: Theorem C.1.
Revised: Theorem C.1.

Original: Under assumptions of compactness of feasible sets, (sub)gradient boundedness and continuity of f i ( x, y ) , g ( x ) and r ( y ) over X and Y , problem (1) is equivalent to problem (3) in the sense that for any solution ( x (cid:63) , y (cid:63) , ˜ S x , ˜ S y ) of (3) , the point ( x (cid:63) , y (cid:63) ) is a solution to (1) .
Revised: Under assumptions of compactness of feasible sets, (sub)gradient boundedness and continuity of f i ( x, y ) , g ( x ) and r ( y ) over X and Y , problem (1) is equivalent to problem (3) in the sense that for any solution ( x (cid:63) , y (cid:63) , ˜ S x , ˜ S y ) of (3) , the point ( x (cid:63) , y (cid:63) ) is a solution to (1) .

Original: Proof.
Revised: Proof.

Original: The proof is based on the analysis of a similar result in [33].
Revised: The proof is based on the analysis of a similar result in [33].

Original: Let ˜Ψ( x , y ) =
Revised: Let ˜Ψ( x , y ) =

Original: F ( x , y )
Revised: F ( x , y )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: mi =1 ( g ( x i ) − r ( y i )) .
Revised: mi =1 ( g ( x i ) − r ( y i )) .

Original: For simplicity of notation, we assume U ⊗ I d x = U x
Revised: For simplicity of notation, we assume U ⊗ I d x = U x

Original: and U ⊗ I d y
Revised: and U ⊗ I d y

Original: =
Revised: =

Original: U y .
Revised: U y .

Original: The objective function ˜Ψ( x , y ) is convex-concave and feasible sets X m , Y m are compact.
Revised: The objective function ˜Ψ( x , y ) is convex-concave and feasible sets X m , Y m are compact.

Original: Then, using
Revised: Then, using

Original: Sion-Kakutani Theorem [2],
Revised: Sion-Kakutani Theorem [2],

Original: [Equation]
Revised: [Equation]

Original: Consider the following problem:
Revised: Consider the following problem:

Original: [Equation]
Revised: [Equation]

Original: We assume that ( x (cid:63) , y (cid:63) ) is the saddle point solution of problem (1).
Revised: We assume that ( x (cid:63) , y (cid:63) ) is the saddle point solution of problem (1).

Original: Therefore, constraint qualiﬁcation ( U x x (cid:63) = 0 ) holds.
Revised: Therefore, constraint qualiﬁcation ( U x x (cid:63) = 0 ) holds.

Original: Then using Lagrange strong duality,
Revised: Then using Lagrange strong duality,

Original: [Equation]
Revised: [Equation]

Original: Since X and Y are compact sets, the gradients and subgradients respectively of f i ( x, y ) , g ( x ) and r ( y ) with respect to x will be bounded by a suitable constant B 1 .
Revised: Since X and Y are compact sets, the gradients and subgradients respectively of f i ( x, y ) , g ( x ) and r ( y ) with respect to x will be bounded by a suitable constant B 1 .

Original: Therefore using Theorem 2 in [19], there exists an optimal dual multiplier ˜ S
Revised: Therefore using Theorem 2 in [19], there exists an optimal dual multiplier ˜ S

Original: x for r.h.s in (10) such that (cid:107) ˜
Revised: x for r.h.s in (10) such that (cid:107) ˜

Original: S x (cid:107) 2 ≤ √ mB 1 λ +min ( U x ) =: R 1 , where λ +min ( U x ) denotes the smallest nonzero eigenvalue of U x .
Revised: S x (cid:107) 2 ≤ √ mB 1 λ +min ( U x ) =: R 1 , where λ +min ( U x ) denotes the smallest nonzero eigenvalue of U x .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: This implies that (10) can be rewritten as
Revised: This implies that (10) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Plugging above equality into (8) and by repeated application of Sion-Kakutani theorem [2]
Revised: Plugging above equality into (8) and by repeated application of Sion-Kakutani theorem [2]

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the previous equality due to separability of the objective function in (13) in y and S x .
Revised: where the last equality follows from the previous equality due to separability of the objective function in (13) in y and S x .

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we consider a maximization problem associated with the consensus constraint U y y
Revised: we consider a maximization problem associated with the consensus constraint U y y

Original: = 0 to write (13) in the form of (3).
Revised: = 0 to write (13) in the form of (3).

Original: Towards this end [Equation] consider
Revised: Towards this end, consider

Original: [Equation]
Revised: [Equation]

Original: that U y y
Revised: that U y y

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The dual formulation of (14) is given by
Revised: The dual formulation of (14) is given by

Original: [Equation]
Revised: [Equation]

Original: Again using Lagrange strong duality
Revised: Again using Lagrange strong duality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By following arguments similar to the primal problem with constraint U x x =
Revised: By following arguments similar to the primal problem with constraint U x x =

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: where R 2 := √ mB 2 λ +min ( U y ) .
Revised: where R 2 := √ mB 2 λ +min ( U y ) .

Original: By substituting (15) in (13), we obtain
Revised: By substituting (15) in (13), we obtain

Original: [Equation]
Revised: [Equation]

Original: Since optimal Lagrange dual variables ˜ S x and ˜ S y always lie in (cid:96) 2 balls of radius R 1 and Rrespectively, equation (16) can be equivalently written as
Revised: Since optimal Lagrange dual variables ˜ S x and ˜ S y always lie in (cid:96) 2 balls of radius R 1 and Rrespectively, equation (16) can be equivalently written as

Original: [Equation]
Revised: [Equation]

Original: This completes the proof of Theorem C.1.
Revised: This completes the proof of Theorem C.1.

Original: We now provide the optimality conditions for the optimization problem (SPP).
Revised: We now provide the optimality conditions for the optimization problem (SPP).

Original: Optimality Conditions of problem (SPP) .
Revised: Optimality Conditions of problem (SPP) .

Original: Let f ( x, y ) :
Revised: Let f ( x, y ) :

Original: = (cid:80) mi =

Revised: = (cid:80) mi =

Original: f i ( x, y ) .
Revised: f i ( x, y ) .

Original: Since ( x (cid:63) , y (cid:63) ) is the saddle point solution to (SPP), we have x (cid:63) =
Revised: Since ( x (cid:63) , y (cid:63) ) is the saddle point solution to (SPP), we have x (cid:63) =

Original: arg min x ∈X Ψ( x, y (cid:63) ) and y (cid:63) =
Revised: arg min x ∈X Ψ( x, y (cid:63) ) and y (cid:63) =

Original: arg max y ∈Y Ψ( x (cid:63) , y ) .
Revised: arg max y ∈Y Ψ( x (cid:63) , y ) .

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: We also have y (cid:63) =
Revised: We also have y (cid:63) =

Original: arg min y ∈Y − Ψ
Revised: arg min y ∈Y − Ψ

Original: ( x (cid:63) , y ) =
Revised: ( x (cid:63) , y ) =

Original: arg min y ∈ R dy ( − Ψ( x (cid:63) , y )
Revised: arg min y ∈ R dy ( − Ψ( x (cid:63) , y )

Original: + I Y ( y )) .
Revised: + I Y ( y )) .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore,y
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Notations useful for further analysis:
Revised: Notations useful for further analysis:

Original: In the subsequent analysis, we assume d x = d y = 1 for simplicity of representation.
Revised: In the subsequent analysis, we assume d x = d y = 1 for simplicity of representation.

Original: Our analysis still holds for d x > 1 and d y > 1 by incorporating Kronecker product.
Revised: Our analysis still holds for d x > 1 and d y > 1 by incorporating Kronecker product.

Original: We deﬁne Bregman distance with respect to each function f i ( · , y ) and − f i ( x, · ) as
Revised: We deﬁne Bregman distance with respect to each function f i ( · , y ) and − f i ( x, · ) as

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Let L = max { L xx , L yy , L xy , L yx } and µ = min { µ x , µ y } .
Revised: Let L = max { L xx , L yy , L xy , L yx } and µ = min { µ x , µ y } .

Original: Suppose λ max ( I − W ) , λ m − 1 ( I − W ) and ( I − W ) † denote the largest eigenvalue, second smallest eigenvalue and pseudo inverse of I − W respectively.
Revised: Suppose λ max ( I − W ) , λ m − 1 ( I − W ) and ( I − W ) † denote the largest eigenvalue, second smallest eigenvalue and pseudo inverse of I − W respectively.

Original: Let κ f = L/µ and κ g = λ max
Revised: Let κ f = L/µ and κ g = λ max

Original: ( I − W ) /λ
Revised: ( I − W ) /λ

Original: m − 1 ( I − W ) denote the condition number of function f and graph G respectively.
Revised: m − 1 ( I − W ) denote the condition number of function f and graph G respectively.

Original: We further deﬁne D (cid:63)x := − ( I −
Revised: We further deﬁne D (cid:63)x := − ( I −

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Range
Revised: Range

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: W )
Revised: W )

Original: z = 0 } .
Revised: z = 0 } .

Original: We now state a few preliminary results that will be used in later sections.
Revised: We now state a few preliminary results that will be used in later sections.

Original: These results may be of independent interest as well.
Revised: These results may be of independent interest as well.

Original: Proposition C.2.
Revised: Proposition C.2.

Original: Let W be a weight matrix satisfying assumption 3.5.
Revised: Let W be a weight matrix satisfying assumption 3.5.

Original: Then Null ( I − W ) =
Revised: Then Null ( I − W ) =

Original: Range ( 1 ) .
Revised: Range ( 1 ) .

Original: Proof.
Revised: Proof.

Original: We prove this result in two parts.
Revised: We prove this result in two parts.

Original: We ﬁrst show that Null ( I − W ) ⊆
Revised: We ﬁrst show that Null ( I − W ) ⊆

Original: Range ( 1 ) and then show that Range ( 1 ) ⊆
Revised: Range ( 1 ) and then show that Range ( 1 ) ⊆

Original: Null ( I − W ) .
Revised: Null ( I − W ) .

Original: In this regard, let
Revised: In this regard, let

Original: y ∈ Null
Revised: y ∈ Null

Original: ( I − W ) .
Revised: ( I − W ) .

Original: Then we have ( I − W ) y
Revised: Then we have ( I − W ) y

Original: = 0 which implies that
Revised: = 0 which implies that

Original: Wy = y .
Revised: Wy = y .

Original: Hence y is an eigen vector of W with eigen value 1 .
Revised: Hence y is an eigen vector of W with eigen value 1 .

Original: We know that algebraic multiplicity of eigenvalue 1 is one using assumption 3.5.
Revised: We know that algebraic multiplicity of eigenvalue 1 is one using assumption 3.5.

Original: Therefore, there is only one linearly independent eigenvector associated with eigenvalue 1 .
Revised: Therefore, there is only one linearly independent eigenvector associated with eigenvalue 1 .

Original: We also know that 1 is an eigenvector associated with eigenvalue 1 because W 1 = 1 .
Revised: We also know that 1 is an eigenvector associated with eigenvalue 1 because W 1 = 1 .

Original: Therefore, y must belong to
Revised: Therefore, y must belong to

Original: Range ( 1 ) .
Revised: Range ( 1 ) .

Original: This completes the ﬁrst part of the proof.
Revised: This completes the ﬁrst part of the proof.

Original: To prove the other part, let
Revised: To prove the other part, let

Original: y ∈ Range
Revised: y ∈ Range

Original: Then ( I − W )
Revised: Then ( I − W )

Original: y
Revised: y

Original: = ( I − W ) η 1 = 0 .
Revised: = ( I − W ) η 1 = 0 .

Original: This shows that y ∈ Null ( I − W ) .
Revised: This shows that y ∈ Null ( I − W ) .

Original: By combining both the parts, we get the desired result.
Revised: By combining both the parts, we get the desired result.

Original: Proposition C.3.
Revised: Proposition C.3.

Original: Let W satisfy Assumption 3.5 and let D (cid:63)x and D (cid:63)y be as deﬁned in above paragraph.
Revised: Let W satisfy Assumption 3.5 and let D (cid:63)x and D (cid:63)y be as deﬁned in above paragraph.

Original: Then, D (cid:63)x ∈ Range ( I − W ) and D (cid:63)y ∈ Range ( I − W ) .
Revised: Then, D (cid:63)x ∈ Range ( I − W ) and D (cid:63)y ∈ Range ( I − W ) .

Original: Proof.
Revised: Proof.

Original: To prove this result, we ﬁrst show that Range ( I − W ) =
Revised: To prove this result, we ﬁrst show that Range ( I − W ) =

Original: ( Range ( 1 ))
Revised: ( Range ( 1 ))

Original: ⊥ using Assumption 3.5.
Revised: ⊥ using Assumption 3.5.

Original: Then we prove that both D (cid:63)x and D (cid:63)y lie in ( Range ( 1 ))
Revised: Then we prove that both D (cid:63)x and D (cid:63)y lie in ( Range ( 1 ))

Original: ⊥ .
Revised: ⊥ .

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst show that Range ( I − W ) ⊆
Revised: We ﬁrst show that Range ( I − W ) ⊆

Original: (cid:0) Range ( 1 (cid:62) ) (cid:1) ⊥ .
Revised: (cid:0) Range ( 1 (cid:62) ) (cid:1) ⊥ .

Original: Towards that end, let
Revised: Towards that end, let

Original: y ∈ Range ( I − W ) .
Revised: y ∈ Range ( I − W ) .

Original: This implies that there exists a z ∈ R m such that ( I − W ) z
Revised: This implies that there exists a z ∈ R m such that ( I − W ) z

Original: = y .
Revised: = y .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: The last step follows from W 1 = 1 .
Revised: The last step follows from W 1 = 1 .

Original: This implies that y ∈ ( Range ( 1 )) ⊥ .
Revised: This implies that y ∈ ( Range ( 1 )) ⊥ .

Original: Therefore,
Revised: Therefore,

Original: Range ( I − W ) ⊆
Revised: Range ( I − W ) ⊆

Original: ( Range ( 1 )) ⊥ .
Revised: ( Range ( 1 )) ⊥ .

Original: Next we show that dim(Range ( I − W ) )
Revised: Next we show that dim(Range ( I − W ) )

Original: =
Revised: =

Original: dim( ( Range ( 1 )) ⊥ ).
Revised: dim( ( Range ( 1 )) ⊥ ).

Original: Using Proposition C.2, we have Null ( I − W ) =
Revised: Using Proposition C.2, we have Null ( I − W ) =

Original: Range ( 1 ) .
Revised: Range ( 1 ) .

Original: This implies that dim ( Null ( I − W ))
Revised: This implies that dim ( Null ( I − W ))

Original: = 1 .
Revised: = 1 .

Original: Using Rank-Nullity Theorem [11], we get dim ( Range ( I − W ))
Revised: Using Rank-Nullity Theorem [11], we get dim ( Range ( I − W ))

Original: = m − 1 .
Revised: = m − 1 .

Original: Further Range ( 1 ) is a onedimensional subspace of R m and hence dim (( Range ( 1 ))
Revised: Further Range ( 1 ) is a onedimensional subspace of R m and hence dim (( Range ( 1 ))

Original: ⊥ )
Revised: ⊥ )

Original: = m − 1 .
Revised: = m − 1 .

Original: Therefore, dim(Range ( I − W ) )
Revised: Therefore, dim(Range ( I − W ) )

Original: =
Revised: =

Original: dim( ( Range ( 1 )) ⊥ ).
Revised: dim( ( Range ( 1 )) ⊥ ).

Original: Using Theorem 1.11 in [11], we get Range ( I − W ) =
Revised: Using Theorem 1.11 in [11], we get Range ( I − W ) =

Original: ( Range ( 1 )) ⊥ .
Revised: ( Range ( 1 )) ⊥ .

Original: This completes the ﬁrst part of the proof.
Revised: This completes the ﬁrst part of the proof.

Original: Recall
Revised: Recall

Original: [Equation]
Revised: [Equation]

Original: Therefore, η 1 (cid:62) D (cid:63)x = − η 1 (cid:62) ( I − J ) ∇ x F
Revised: Therefore, η 1 (cid:62) D (cid:63)x = − η 1 (cid:62) ( I − J ) ∇ x F

Original: ( 1 z (cid:63) ) = 0 because 1 (cid:62) J = 1 (cid:62) .
Revised: ( 1 z (cid:63) ) = 0 because 1 (cid:62) J = 1 (cid:62) .

Original: Similarly, η 1 (cid:62) D (cid:63)y = 1 (cid:62) ( I − J ) ∇ y F ( 1 z (cid:63) )
Revised: Similarly, η 1 (cid:62) D (cid:63)y = 1 (cid:62) ( I − J ) ∇ y F ( 1 z (cid:63) )

Original: = 0 .
Revised: = 0 .

Original: Hence, D (cid:63)x
Revised: Hence, D (cid:63)x

Original: Proposition C.4.
Revised: Proposition C.4.

Original: ( Smoothness in x ) Assume that f ( x, y ) is convex and L xx -smooth in x for any ﬁxed y .
Revised: ( Smoothness in x ) Assume that f ( x, y ) is convex and L xx -smooth in x for any ﬁxed y .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Using the smoothness of f ( · , y ) , we have
Revised: Using the smoothness of f ( · , y ) , we have

Original: [Equation]
Revised: [Equation]

Original: This completes the proof of second inequality.
Revised: This completes the proof of second inequality.

Original: Let h ( x 1 ) :
Revised: Let h ( x 1 ) :

Original: = V f,y ( x 1 , x 2 ) for a given y and x 2 .
Revised: = V f,y ( x 1 , x 2 ) for a given y and x 2 .

Original: Notice that h ( x 1 ) = 0 at x 1 = x 2 .
Revised: Notice that h ( x 1 ) = 0 at x 1 = x 2 .

Original: Using convexity of f ( x, y ) in x , h ( x 1 ) ≥ 0 .
Revised: Using convexity of f ( x, y ) in x , h ( x 1 ) ≥ 0 .

Original: Therefore, h ( x 1 ) achieves its minimum value at x 2 and the minimum value is 0 .
Revised: Therefore, h ( x 1 ) achieves its minimum value at x 2 and the minimum value is 0 .

Original: [Equation]
Revised: [Equation]

Original: This implies that h ( x 1 ) is also L xx -smooth.
Revised: This implies that h ( x 1 ) is also L xx -smooth.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Take minimization over ¯ x on both sides.
Revised: Take minimization over ¯ x on both sides.

Original: [Equation]
Revised: [Equation]

Original: Let δ (¯ x ) =
Revised: Let δ (¯ x ) =

Original: (cid:104)∇ x h ( x 1 ) , ¯ x − x 1 (cid:105) + L xx 2 (cid:107) ¯
Revised: (cid:104)∇ x h ( x 1 ) , ¯ x − x 1 (cid:105) + L xx 2 (cid:107) ¯

Original: [Equation]
Revised: [Equation]

Original: − x
Revised: − x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, min
Revised: Therefore, min

Original: [Equation]
Revised: [Equation]

Original: Plug in above minimum value into (42).
Revised: Plug in above minimum value into (42).

Original: [Equation]
Revised: [Equation]

Original: This gives
Revised: This gives

Original: [Equation]
Revised: [Equation]

Original: Proposition C.5.
Revised: Proposition C.5.

Original: ( Smoothness in y )
Revised: ( Smoothness in y )

Original: Assume that − f ( x, y ) is convex and L yy -smooth in y for any ﬁxed x .
Revised: Assume that − f ( x, y ) is convex and L yy -smooth in y for any ﬁxed x .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition C.5 is similar to that of Proposition C.4, and is omitted.
Revised: The proof of Proposition C.5 is similar to that of Proposition C.4, and is omitted.

Original: D A recursion relationship useful for further analysis
Revised: D A recursion relationship useful for further analysis

Original: In the following discussion, we use the shorthand notation 1 u to represent ( 1 ⊗ I d ) u , for any u ∈ R d , and the notation A † denotes the pseudo-inverse of a square matrix A .
Revised: In the following discussion, we use the shorthand notation 1 u to represent ( 1 ⊗ I d ) u , for any u ∈ R d , and the notation A † denotes the pseudo-inverse of a square matrix A .

Original: Lemma D.1.
Revised: Lemma D.1.

Original: Let x k,t +1 , y k,t +1 , D xk,t +1 , D yk,t +1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t +1 be obtained from
Revised: Let x k,t +1 , y k,t +1 , D xk,t +1 , D yk,t +1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t +1 be obtained from

Original: Algorithm 1 using IPDHG( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s , γ x , γ y , α x , α y , G ).
Revised: Algorithm 1 using IPDHG( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s , γ x , γ y , α x , α y , G ).

Original: Let
Revised: Let

Original: Assumption 3.4 and Assumption 3.5 hold.
Revised: Assumption 3.4 and Assumption 3.5 hold.

Original: Suppose α y ∈ (cid:0) 0 , (1 + δ ) − 1 (cid:1) and
Revised: Suppose α y ∈ (cid:0) 0 , (1 + δ ) − 1 (cid:1) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then the following holds for all t ≥ 0 :
Revised: Then the following holds for all t ≥ 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where E denotes the conditional expectation on stochastic compression at t -th update step and
Revised: where E denotes the conditional expectation on stochastic compression at t -th update step and

Original: M y = 1 −
Revised: M y = 1 −

Original: [Equation]
Revised: [Equation]

Original: I − W ) and H (cid:63)x = 1 ( x (cid:63) − sm ∇ x f ( z (cid:63) )) and H (cid:63)y = 1 ( y (cid:63) + sm ∇ y f
Revised: I − W ) and H (cid:63)x = 1 ( x (cid:63) − sm ∇ x f ( z (cid:63) )) and H (cid:63)y = 1 ( y (cid:63) + sm ∇ y f

Original: ( z (cid:63) )) . .
Revised: ( z (cid:63) )) . .

Original: Proof of Lemma D. 
Revised: Proof of Lemma D.

Original: We follow [20] to prove Lemma D.1.
Revised: We follow [20] to prove Lemma D.1.

Original: We have H (cid:63)x = 1 ( x (cid:63) − sm ∇ x f ( x (cid:63) , y (cid:63) )) and H (cid:63)y = 1 ( y (cid:63) + sm ∇ y f
Revised: We have H (cid:63)x = 1 ( x (cid:63) − sm ∇ x f ( x (cid:63) , y (cid:63) )) and H (cid:63)y = 1 ( y (cid:63) + sm ∇ y f

Original: ( x (cid:63) , y (cid:63) )) .
Revised: ( x (cid:63) , y (cid:63) )) .

Original: First, we bound the terms appearing on the l.h.s.
Revised: First, we bound the terms appearing on the l.h.s.

Original: of (53) as
Revised: of (53) as

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: 54) and (cid:18)
Revised: 54) and (cid:18)

Original: 2 sγ
Revised: 2 sγ

Original: [Equation]
Revised: [Equation]

Original: Proofs of (54) and (55) are provided in Sections D.1 and D.2, respectively.
Revised: Proofs of (54) and (55) are provided in Sections D.1 and D.2, respectively.

Original: On adding (54) and (55), we obtain
Revised: On adding (54) and (55), we obtain

Original: [Equation]
Revised: [Equation]

Original: We now bound the terms on the r
Revised: We now bound the terms on the r

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: s. of (56).
Revised: s. of (56).

Original: First, observe that
Revised: First, observe that

Original: [Equation]
Revised: [Equation]

Original: ,t +1 (cid:13)(cid:13)(cid:13) 2 ( I − W )=2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)
Revised: ,t +1 (cid:13)(cid:13)(cid:13) 2 ( I − W )=2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)

Original: γ y
Revised: γ y

Original: ( I − W )
Revised: ( I − W )

Original: + 2 √ δα 2 y
Revised: + 2 √ δα 2 y

Original: (ˆ ν yk,t +1 − ν yk,t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (ˆ ν yk,t +1 − ν yk,t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)

Original: +
Revised: +

Original: γ 2 y M y
Revised: γ 2 y M y

Original: E (cid:13)(cid:13)(cid:13) ( I − W )(ˆ ν yk,t +1 − ν yk,t +1 ) (cid:13)(cid:13)(cid:13)≤2 (cid:13)(cid:13)(cid:13)(cid:13) (cid:113) γ y ( I − W ) + 2 √ δα 2 y (cid:13)(cid:13)(cid:13)(cid:13)
Revised: E (cid:13)(cid:13)(cid:13) ( I − W )(ˆ ν yk,t +1 − ν yk,t +1 ) (cid:13)(cid:13)(cid:13)≤2 (cid:13)(cid:13)(cid:13)(cid:13) (cid:113) γ y ( I − W ) + 2 √ δα 2 y (cid:13)(cid:13)(cid:13)(cid:13)

Original: E (cid:13)(cid:13)(cid:13) ˆ
Revised: E (cid:13)(cid:13)(cid:13) ˆ

Original: ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13) 2 + γ 2 y M y (cid:107) I − W (cid:107) 2 E (cid:13)(cid:13)(cid:13)
Revised: ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13) 2 + γ 2 y M y (cid:107) I − W (cid:107) 2 E (cid:13)(cid:13)(cid:13)

Original: ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)≤ (cid:32) (cid:16) γ y λ max ( I − W ) + 2 √ δα 2 y (cid:17) + γ 2 y M y λ
Revised: ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)≤ (cid:32) (cid:16) γ y λ max ( I − W ) + 2 √ δα 2 y (cid:17) + γ 2 y M y λ

Original: 2max
Revised: 2max

Original: ( I − W ) (cid:33)
Revised: ( I − W ) (cid:33)

Original: E (cid:13)(cid:13)(cid:13) ˆ
Revised: E (cid:13)(cid:13)(cid:13) ˆ

Original: ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13) 2 .
Revised: ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13) 2 .

Original: We also have
Revised: We also have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting this inequality in (57)
Revised: Substituting this inequality in (57)

Original: [Equation]
Revised: [Equation]

Original: we bound the last two terms in the r
Revised: we bound the last two terms in the r

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: s of (56) as
Revised: s of (56) as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:63)y (cid:13)(cid:13)(cid:13)
Revised: (cid:63)y (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By substituting (60) and (61) in (56), we get
Revised: By substituting (60) and (61) in (56), we get

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) 2 .
Revised: (cid:13)(cid:13)(cid:13) 2 .

Original: The coefﬁcient of (cid:13)(cid:13)(cid:13) ν yk,t +1 − H yk
Revised: The coefﬁcient of (cid:13)(cid:13)(cid:13) ν yk,t +1 − H yk

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: 2 in
Revised: 2 in

Original: [Equation]
Revised: [Equation]

Original: 62) is
Revised: 62) is

Original: [Equation]
Revised: [Equation]

Original: In deriving (63), the ﬁrst inequality follows from M y < 1 (to be proved later in Section E.4) and the second inequality follows from γ y λ max ( I − W )< 1 , since from assumption (52), it holds that 0 < γ y < 2 − 2 √ δα y λ max ( I − W ) ≤ 2 λ max ( I − W ) .
Revised: In deriving (63), the ﬁrst inequality follows from M y < 1 (to be proved later in Section E.4) and the second inequality follows from γ y λ max ( I − W )< 1 , since from assumption (52), it holds that 0 < γ y < 2 − 2 √ δα y λ max ( I − W ) ≤ 2 λ max ( I − W ) .

Original: As a result, (62) can be simpliﬁed as
Revised: As a result, (62) can be simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: proving Lemma D
Revised: proving Lemma D

Original: [Equation]
Revised: [Equation]

Original: The remainder of this section is devoted to prove (54) and (55)
Revised: The remainder of this section is devoted to prove (54) and (55)

Original: [Equation]
Revised: [Equation]

Original: D.1 Proof of (54)
Revised: D.1 Proof of (54)

Original: First, observe that
Revised: First, observe that

Original: [Equation]
Revised: [Equation]

Original: (ˆ y ik,t +1 ) − prox sr (cid:0) H (cid:63)i,y
Revised: (ˆ y ik,t +1 ) − prox sr (cid:0) H (cid:63)i,y

Original: (cid:1)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:1)(cid:13)(cid:13)(cid:13)(cid:13)

Original: ≤ m (cid:88) i =
Revised: ≤ m (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: ˆ y ik,t +1 − H (cid:63)i,y (cid:13)(cid:13) 2 (from non-expansivity of prox)
Revised: ˆ y ik,t +1 − H (cid:63)i,y (cid:13)(cid:13) 2 (from non-expansivity of prox)

Original: =
Revised: =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: ˆ y k,t +1 − H (cid:63)y (cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)
Revised: ˆ y k,t +1 − H (cid:63)y (cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)

Original: ν yk,t +1 − γ y 2 ( I − W )
Revised: ν yk,t +1 − γ y 2 ( I − W )

Original: ˆ ν yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)
Revised: ˆ ν yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)

Original: ν yk,t +1 − γ y 2 ( I − W )(ˆ ν yk,t +1 − ν yk,t +1 + ν yk,t +1 )
Revised: ν yk,t +1 − γ y 2 ( I − W )(ˆ ν yk,t +1 − ν yk,t +1 + ν yk,t +1 )

Original: − H (cid:63)y (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:16)
Revised: − H (cid:63)y (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:16)

Original: I − γ y 2 ( I − W ) (cid:17) ( ν yk,t +1 − H (cid:63)y ) − γ y 2 ( I − W )
Revised: I − γ y 2 ( I − W ) (cid:17) ( ν yk,t +1 − H (cid:63)y ) − γ y 2 ( I − W )

Original: (ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:16)
Revised: (ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:16)

Original: I − γ y 2 ( I − W ) (cid:17) ( ν yk,t +1 − H (cid:63)y ) (cid:13)(cid:13)(cid:13) 2 + γ 

Revised: I − γ y 2 ( I − W ) (cid:17) ( ν yk,t +1 − H (cid:63)y ) (cid:13)(cid:13)(cid:13) 2 + γ

Original: y
Revised: y

Original: (cid:13) (cid:13)(cid:13)
Revised: (cid:13) (cid:13)(cid:13)

Original: ( I − W )
Revised: ( I − W )

Original: (ˆ ν yk,t +1 − ν yk,t
Revised: (ˆ ν yk,t +1 − ν yk,t

Original: [Equation]
Revised: [Equation]

Original: By taking conditional expectation over stochastic compression at t -th iterate, we obtain
Revised: By taking conditional expectation over stochastic compression at t -th iterate, we obtain

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The last equality follows from Assumption 3.4.
Revised: The last equality follows from Assumption 3.4.

Original: By substituting the above equation in (66), we obtain
Revised: By substituting the above equation in (66), we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: We will now convert the square norm terms of (68) into matrix-norm based terms.
Revised: We will now convert the square norm terms of (68) into matrix-norm based terms.

Original: Towards that end, observe that (cid:16)
Revised: Towards that end, observe that (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:0) − I + γ y 2 ( I − W ) (cid:1) is a negative semideﬁnite matrix because 0 < γ y 2 λ max ( I − W ) <from the choice of γ y .
Revised: Note that (cid:0) − I + γ y 2 ( I − W ) (cid:1) is a negative semideﬁnite matrix because 0 < γ y 2 λ max ( I − W ) <from the choice of γ y .

Original: Using this fact in equation (69), we get ∀ x ∈ R m ,
Revised: Using this fact in equation (69), we get ∀ x ∈ R m ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: On substituting above two equalities (71) and (72) in (68)
Revised: On substituting above two equalities (71) and (72) in (68)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: To complete the proof of (54)
Revised: To complete the proof of (54)

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst show that the ﬁrst term on the r.h.s.
Revised: we ﬁrst show that the ﬁrst term on the r.h.s.

Original: of (73) is at most
Revised: of (73) is at most

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: where M
Revised: where M

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: W ) .
Revised: W ) .

Original: To this end, consider
Revised: To this end, consider

Original: [Equation]
Revised: [Equation]

Original: The largest eigenvalue of √ δα y I −
Revised: The largest eigenvalue of √ δα y I −

Original: [Equation]
Revised: [Equation]

Original: is
Revised: is

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Therefore, √ δα y I −
Revised: Therefore, √ δα y I −

Original: [Equation]
Revised: [Equation]

Original: I −
Revised: I −

Original: [Equation]
Revised: [Equation]

Original: y 2 ( I − W ) (cid:1) is negative semideﬁnite, and x
Revised: y 2 ( I − W ) (cid:1) is negative semideﬁnite, and x

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Substituting x = ν yk,t +1 − H (cid:63)y into the above inequality and using the deﬁnition of (cid:107) x (cid:107) 2 A , we obtain
Revised: Substituting x = ν yk,t +1 − H (cid:63)y into the above inequality and using the deﬁnition of (cid:107) x (cid:107) 2 A , we obtain

Original: [Equation]
Revised: [Equation]

Original: From (73) and (77), we see that
Revised: From (73) and (77), we see that

Original: [Equation]
Revised: [Equation]

Original: Multiplying throughout by M y , the proof of (54) is complete.
Revised: Multiplying throughout by M y , the proof of (54) is complete.

Original: D.2 Proof of (55)
Revised: D.2 Proof of (55)

Original: For every k , let G xk,t and G yk,t denote the stochastic gradient oracles at iterate t .
Revised: For every k , let G xk,t and G yk,t denote the stochastic gradient oracles at iterate t .

Original: For Algorithm 2,
Revised: For Algorithm 2,

Original: G xk,t = ∇ x F ( z k,t ; ξ k,t ) and G yk,t = ∇ y F ( z k,t , ξ k,t ) are obtained using general stochastic gradient oracle.
Revised: G xk,t = ∇ x F ( z k,t ; ξ k,t ) and G yk,t = ∇ y F ( z k,t , ξ k,t ) are obtained using general stochastic gradient oracle.

Original: In Algorithm 3, G xk,t and G yk,t are obtained from the SVRG oracle.
Revised: In Algorithm 3, G xk,t and G yk,t are obtained from the SVRG oracle.

Original: Step
Revised: Step

Original: [Equation]
Revised: [Equation]

Original: Computing
Revised: Computing

Original: [Equation]
Revised: [Equation]

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: On pre-multiplying both sides by (cid:112) ( I − W ) † and taking square norm on the resulting equality, we obtain
Revised: On pre-multiplying both sides by (cid:112) ( I − W ) † and taking square norm on the resulting equality, we obtain

Original: [Equation]
Revised: [Equation]

Original: ( I − W ) † ( I − W )(ˆ ν yk
Revised: ( I − W ) † ( I − W )(ˆ ν yk

Original: [Equation]
Revised: [Equation]

Original: t +1 − ν yk
Revised: t +1 − ν yk

Original: [Equation]
Revised: [Equation]

Original: 1 ) (cid:13)(cid:13)(cid:13)(cid:13) + 2 (cid:28)(cid:113)
Revised: 1 ) (cid:13)(cid:13)(cid:13)(cid:13) + 2 (cid:28)(cid:113)

Original: ( I − W ) † ( D yk
Revised: ( I − W ) † ( D yk

Original: [Equation]
Revised: [Equation]

Original: D (cid:63)y )
Revised: D (cid:63)y )

Original: [Equation]
Revised: [Equation]

Original: s (cid:113)
Revised: s (cid:113)

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: W )
Revised: W )

Original: [Equation]
Revised: [Equation]

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: ( ν
Revised: ( ν

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:63)y
Revised: (cid:63)y

Original: [Equation]
Revised: [Equation]

Original: (cid:113)
Revised: (cid:113)

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +1 )
Revised: +1 )

Original: (cid:29) .
Revised: (cid:29) .

Original: By taking conditional expectation over compression at t -th iterate and using the result
Revised: By taking conditional expectation over compression at t -th iterate and using the result

Original: E (cid:16) ˆ ν yk,t +1 − ν yk,t +1 (cid:17) = 0 , we obtain
Revised: E (cid:16) ˆ ν yk,t +1 − ν yk,t +1 (cid:17) = 0 , we obtain

Original: E (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)
Revised: E (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)

Original: ( I
Revised: ( I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: W )( ν
Revised: W )( ν

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: H (cid:63)y ) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: H (cid:63)y ) (cid:13)(cid:13)(cid:13)(cid:13)

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: W )
Revised: W )

Original: [Equation]
Revised: [Equation]

Original: W )
Revised: W )

Original: [Equation]
Revised: [Equation]

Original: ν
Revised: ν

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: − ν
Revised: − ν

Original: [Equation]
Revised: [Equation]

Original: t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We have (cid:18)(cid:113)
Revised: We have (cid:18)(cid:113)

Original: ( I
Revised: ( I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: (81) where the last equality follows from the deﬁnition of pseudoinverse.
Revised: (81) where the last equality follows from the deﬁnition of pseudoinverse.

Original: Consider (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)
Revised: Consider (cid:13)(cid:13)(cid:13)(cid:13)(cid:113)

Original: ( I − W ) † ( I − W
Revised: ( I − W ) † ( I − W

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, we see that (cid:13)(cid:13)(cid:13)(cid:112)
Revised: Similarly, we see that (cid:13)(cid:13)(cid:13)(cid:112)

Original: [Equation]
Revised: [Equation]

Original: yk
Revised: yk

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: yk
Revised: yk

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 2 I − W Substituting in (80), we obtain
Revised: 2 I − W Substituting in (80), we obtain

Original: [Equation]
Revised: [Equation]

Original: Now, we will simplify the last term of (83).
Revised: Now, we will simplify the last term of (83).

Original: From the property of adjoints, (cid:28)(cid:113)
Revised: From the property of adjoints, (cid:28)(cid:113)

Original: [Equation]
Revised: [Equation]

Original: Note that D (cid:63)y ∈ Range ( I − W ) using Proposition C.3.
Revised: Note that D (cid:63)y ∈ Range ( I − W ) using Proposition C.3.

Original: Further note that D yk,t ∈ Range ( I − W ) because of update process (Step 10) in Algorithm 1.
Revised: Further note that D yk,t ∈ Range ( I − W ) because of update process (Step 10) in Algorithm 1.

Original: Therefore, there exists ˜ D yk
Revised: Therefore, there exists ˜ D yk

Original: [Equation]
Revised: [Equation]

Original: t and ˜ D y such that
Revised: t and ˜ D y such that

Original: D yk,t =
Revised: D yk,t =

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: ˜ D yk,t and D (cid:63)y =
Revised: ˜ D yk,t and D (cid:63)y =

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: W ) ˜
Revised: W ) ˜

Original: D y .
Revised: D y .

Original: [Equation]
Revised: [Equation]

Original: This gives (cid:28)(cid:113)
Revised: This gives (cid:28)(cid:113)

Original: [Equation]
Revised: [Equation]

Original: On substituting above equality in (83), we obtain
Revised: On substituting above equality in (83), we obtain

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that 1 m 1 ∇
Revised: Note that 1 m 1 ∇

Original: y f
Revised: y f

Original: ( z (cid:63) )
Revised: ( z (cid:63) )

Original: = 1 m 1 (cid:80) mi
Revised: = 1 m 1 (cid:80) mi

Original: =1 ∇ y f i ( z (cid:63) ) =
Revised: =1 ∇ y f i ( z (cid:63) ) =

Original: J ∇ y F ( 1 z (cid:63) ) .
Revised: J ∇ y F ( 1 z (cid:63) ) .

Original: Then we can write H (cid:63)y = 

Revised: Then we can write H (cid:63)y =

Original: y (cid:63) + sJ ∇ y F ( 1 z (cid:63) ) .
Revised: y (cid:63) + sJ ∇ y F ( 1 z (cid:63) ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: In deriving equation (88), the second equality follows from the deﬁnition of D (cid:63)y =
Revised: In deriving equation (88), the second equality follows from the deﬁnition of D (cid:63)y =

Original: ( I − J ) ∇ y F ( 1 z (cid:63) ) , and the third equality follows from the update step of ν yk,t +

Revised: ( I − J ) ∇ y F ( 1 z (cid:63) ) , and the third equality follows from the update step of ν yk,t +

Original: (Step 9 in Algorithm 1).
Revised: (Step 9 in Algorithm 1).

Original: Substituting (88) in (87),
Revised: Substituting (88) in (87),

Original: [Equation]
Revised: [Equation]

Original: Taking square norm on both sides of (88), we obtain .
Revised: Taking square norm on both sides of (88), we obtain .

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table][Equation] [Table] [Table]
Revised: [Table]

Original: [Table] [Equation] [Table] [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: We have
Revised: We have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: γ y
Revised: γ y

Original: [Equation]
Revised: [Equation]

Original: By substituting above equality in (91)
Revised: By substituting above equality in (91)

Original: [Equation]
Revised: [Equation]

Original: we obtain 2 s
Revised: we obtain 2 s

Original: [Equation]
Revised: [Equation]

Original: ( I − W ) † + (cid:13)(cid:13)(cid:13) ν
Revised: ( I − W ) † + (cid:13)(cid:13)(cid:13) ν

Original: yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13) 2 = 2 sγ y
Revised: yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13) 2 = 2 sγ y

Original: (cid:13)(cid:13)(cid:13) D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13) D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13)

Original: ( I − W ) † + (cid:13)(cid:13)(cid:13) ν
Revised: ( I − W ) † + (cid:13)(cid:13)(cid:13) ν

Original: yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)
Revised: yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)

Original: ν yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13) 

Revised: ν yk,t +1 − H (cid:63)y (cid:13)(cid:13)(cid:13)

Original: I − γy 2 ( I − W )
Revised: I − γy 2 ( I − W )

Original: +
Revised: +

Original: γ y 2 E (cid:13)(cid:13)(cid:13) ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)
Revised: γ y 2 E (cid:13)(cid:13)(cid:13) ˆ ν yk,t +1 − ν yk,t +1 (cid:13)(cid:13)(cid:13)

Original: I − W − s 2 (cid:13)(cid:13)(cid:13)
Revised: I − W − s 2 (cid:13)(cid:13)(cid:13)

Original: D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13) 2 + (cid:13)(cid:13)(cid:13)
Revised: D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13) 2 + (cid:13)(cid:13)(cid:13)

Original: y k,t − 1 y (cid:63) + s G yk,t − s ∇ y F ( 1 z (cid:63) )
Revised: y k,t − 1 y (cid:63) + s G yk,t − s ∇ y F ( 1 z (cid:63) )

Original: (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)

Original: Hence we get
Revised: Hence we get

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we write
Revised: Now we write

Original: [Equation]
Revised: [Equation]

Original: in terms of
Revised: in terms of

Original: [Equation]
Revised: [Equation]

Original: By substituting this equality in (93), we obtain 2 s
Revised: By substituting this equality in (93), we obtain 2 s

Original: [Equation]
Revised: [Equation]

Original: Step 3:
Revised: Step 3:

Original: Computing √ δE
Revised: Computing √ δE

Original: [Equation]
Revised: [Equation]

Original: 2 and ﬁnishing the proof
Revised: 2 and ﬁnishing the proof

Original: Observe from Step 4 in Algorithm 4 that H yk,t +1 =
Revised: Observe from Step 4 in Algorithm 4 that H yk,t +1 =

Original: (1 − α y )
Revised: (1 − α y )

Original: H yk,t + α y
Revised: H yk,t + α y

Original: ˆ
Revised: ˆ

Original: ν yk,t +1 , and as a result,
Revised: ν yk,t +1 , and as a result,

Original: H yk,t +1 − H (cid:63)y =
Revised: H yk,t +1 − H (cid:63)y =

Original: (1 − α y )
Revised: (1 − α y )

Original: ( H yk,t − H (cid:63)y ) + α y
Revised: ( H yk,t − H (cid:63)y ) + α y

Original: (ˆ ν yk,t +1 − ν yk,t +1 )
Revised: (ˆ ν yk,t +1 − ν yk,t +1 )

Original: + α y ( ν yk,
Revised: + α y ( ν yk,

Original: t +1 − H (cid:63)y ) , and (cid:13)(cid:13)(cid:13)
Revised: t +1 − H (cid:63)y ) , and (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Taking conditional expectation over compression at t -th iterate on both sides and substituting
Revised: Taking conditional expectation over compression at t -th iterate on both sides and substituting

Original: E (cid:16) ˆ ν yk,t +1 − ν yk,t +1 (cid:17) = 0 , we see that
Revised: E (cid:16) ˆ ν yk,t +1 − ν yk,t +1 (cid:17) = 0 , we see that

Original: [Equation]
Revised: [Equation]

Original: 2 + α y (cid:13)(cid:13)(cid:13) ν yk
Revised: 2 + α y (cid:13)(cid:13)(cid:13) ν yk

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: (cid:63)y (cid:13)(cid:13)(cid:13)
Revised: (cid:63)y (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (1 −
Revised: (1 −

Original: [Equation]
Revised: [Equation]

Original: y,k ) (cid:13)(cid:13)(cid:13) ν yk
Revised: y,k ) (cid:13)(cid:13)(cid:13) ν yk

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: yk,t (cid:13)(cid:13)(cid:13)
Revised: yk,t (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: The last equality follows from the identity (cid:107) (1 − α ) x
Revised: The last equality follows from the identity (cid:107) (1 − α ) x

Original: + αy (cid:107) 

Revised: + αy (cid:107)

Original: =
Revised: =

Original: (1 − α ) (cid:107) x (cid:107) 2 + α (cid:107) y (cid:107) 2 − α (1 − α ) (cid:107) x − y (cid:107) 2 .
Revised: (1 − α ) (cid:107) x (cid:107) 2 + α (cid:107) y (cid:107) 2 − α (1 − α ) (cid:107) x − y (cid:107) 2 .

Original: On multiplying both sides of (97) by √ δ , we obtain √
Revised: On multiplying both sides of (97) by √ δ , we obtain √

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We know that
Revised: We know that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from (95) .
Revised: where the last equality follows from (95) .

Original: Now we add above equality with (98) and obtain the following expression:
Revised: Now we add above equality with (98) and obtain the following expression:

Original: [Equation]
Revised: [Equation]

Original: Rearranging the r.h.s.
Revised: Rearranging the r.h.s.

Original: of the above equation, we obtain (55).
Revised: of the above equation, we obtain (55).

Original: We have a similar recursion result in terms of x .
Revised: We have a similar recursion result in terms of x .

Original: Lemma D.2.
Revised: Lemma D.2.

Original: Let x k,t +1 , y k,t +1 , D xk,t +1 , D yk,t +1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t +1 be obtained from
Revised: Let x k,t +1 , y k,t +1 , D xk,t +1 , D yk,t +1 , H xk,t +1 , H yk,t +1 , H w,xk,t +1 , H w,yk,t +1 be obtained from

Original: Algorithm 1 using IPDHG( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s , γ x , γ y , α x , α y , G ).
Revised: Algorithm 1 using IPDHG( x k,t , y k,t , D xk,t , D yk,t , H xk,t , H yk,t , H w,xk,t , H w,yk,t , s , γ x , γ y , α x , α y , G ).

Original: Let
Revised: Let

Original: Assumption 3.4 and Assumption 3.5 hold.
Revised: Assumption 3.4 and Assumption 3.5 hold.

Original: Suppose α x ∈ (cid:0) 0 , (1 + δ ) − 1 (cid:1) and γ x ∈ (cid:32) 0 , min (cid:40) 2 −
Revised: Suppose α x ∈ (cid:0) 0 , (1 + δ ) − 1 (cid:1) and γ x ∈ (cid:32) 0 , min (cid:40) 2 −

Original: [Equation]
Revised: [Equation]

Original: Then the following holds for all t ≥ 0 :
Revised: Then the following holds for all t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: where E denotes the conditional expectation on stochastic compression at t -th update step and
Revised: where E denotes the conditional expectation on stochastic compression at t -th update step and

Original: M x = 1 −
Revised: M x = 1 −

Original: [Equation]
Revised: [Equation]

Original: We omit the proof of Lemma D.2 as it is similar to the proof of Lemma D.
Revised: We omit the proof of Lemma D.2 as it is similar to the proof of Lemma D.

Original: E Proofs for General Stochastic Setting
Revised: E Proofs for General Stochastic Setting

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we state and present proofs of the results for general stochastic setting discussed in
Revised: we state and present proofs of the results for general stochastic setting discussed in

Original: Section 4.
Revised: Section 4.

Original: We now make the following assumptions on the stochastic gradients.
Revised: We now make the following assumptions on the stochastic gradients.

Original: Assumption E.1.
Revised: Assumption E.1.

Original: (Unbiasedness and local bounded variance of stochastic gradients)
Revised: (Unbiasedness and local bounded variance of stochastic gradients)

Original: Stochastic gradients ∇ x f i ( x, y ; ξ i ) and ∇ y f i ( x, y ; ξ i ) for each node i are unbiased estimates of the respective true gradients: E (cid:2) ∇ x f i ( x, y ; ξ i ) (cid:3) =
Revised: Stochastic gradients ∇ x f i ( x, y ; ξ i ) and ∇ y f i ( x, y ; ξ i ) for each node i are unbiased estimates of the respective true gradients: E (cid:2) ∇ x f i ( x, y ; ξ i ) (cid:3) =

Original: ∇ x f i ( x, y ) , E (cid:2) ∇ y f i ( x, y ; ξ i ) (cid:3)
Revised: ∇ x f i ( x, y ) , E (cid:2) ∇ y f i ( x, y ; ξ i ) (cid:3)

Original: = ∇ y f i ( x, y ) and have bounded variance:
Revised: = ∇ y f i ( x, y ) and have bounded variance:

Original: [Equation]
Revised: [Equation]

Original: where ( x (cid:63) , y (cid:63) ) is the saddle point of (SPP).
Revised: where ( x (cid:63) , y (cid:63) ) is the saddle point of (SPP).

Original: Deﬁne σ 2 :=
Revised: Deﬁne σ 2 :=

Original: σ 2 x + σ 2 y .
Revised: σ 2 x + σ 2 y .

Original: Assumption E.2.
Revised: Assumption E.2.

Original: (Smoothness)
Revised: (Smoothness)

Original: Each f i ( · , y
Revised: Each f i ( · , y

Original: ξ i ) is L xx smooth in expectation; for all x 1 , x
Revised: ξ i ) is L xx smooth in expectation; for all x 1 , x

Original: d x , E
Revised: d x , E

Original: ∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: , y
Revised: , y

Original: [Equation]
Revised: [Equation]

Original: ξ
Revised: ξ

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , y )
Revised: , y )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , y )
Revised: , y )

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: Each − f i ( x,
Revised: Each − f i ( x,

Original: ; ξ i ) is L yy smooth in expectation; for all y 1 , y 2 ∈
Revised: ; ξ i ) is L yy smooth in expectation; for all y 1 , y 2 ∈

Original: y ,
Revised: y ,

Original: −∇
Revised: −∇

Original: For every x ∈ R d x , the following holds
Revised: For every x ∈ R d x , the following holds

Original: 2 for all y
Revised: 2 for all y

Original: , y
Revised: , y

Original: For every y ∈ R d y , the following
Revised: For every y ∈ R d y , the following

Original: for all
Revised: for all

Original: We deﬁne a quantity Φ k,t consisting of primal and dual updates which is instrumental in deriving the convergence rate of Algorithm 2.
Revised: We deﬁne a quantity Φ k,t consisting of primal and dual updates which is instrumental in deriving the convergence rate of Algorithm 2.

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 0 and k ≥
Revised: for all t ≥ 0 and k ≥

Original: [Equation]
Revised: [Equation]

Original: Lemma E.3.
Revised: Lemma E.3.

Original: Suppose { x k,t } t and { y k,t } t are the sequences generated by Algorithm 2 with
Revised: Suppose { x k,t } t and { y k,t } t are the sequences generated by Algorithm 2 with

Original: G = [ ∇ x F ( · ; ξ ); −∇ y F ( · ; ξ )] .
Revised: G = [ ∇ x F ( · ; ξ ); −∇ y F ( · ; ξ )] .

Original: Then, under Assumption 3.1, Assumption 3.2, Assumption E.1 and
Revised: Then, under Assumption 3.1, Assumption 3.2, Assumption E.1 and

Original: Assumption E.2, the following hold for all t ≥ 0 :
Revised: Assumption E.2, the following hold for all t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: where E denotes the conditional expectation on stochastic gradient at t -th update step.
Revised: where E denotes the conditional expectation on stochastic gradient at t -th update step.

Original: E.1 Proof of Lemma E. 
Revised: E.1 Proof of Lemma E.

Original: We will derive inequality (106) here.
Revised: We will derive inequality (106) here.

Original: The proof of inequality (107) is similar and is omitted.
Revised: The proof of inequality (107) is similar and is omitted.

Original: In the general stochastic setting, we have G i,xk,t = ∇ x f i ( z ik,t ; ξ ik,t ) , G i,yk,t = ∇ y f i ( z ik,t ; ξ ik,t ) and step size is s k .
Revised: In the general stochastic setting, we have G i,xk,t = ∇ x f i ( z ik,t ; ξ ik,t ) , G i,yk,t = ∇ y f i ( z ik,t ; ξ ik,t ) and step size is s k .

Original: We now have
Revised: We now have

Original: [Equation]
Revised: [Equation]

Original: − s k G xk,t + s k ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: − s k G xk,t + s k ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: (cid:13)(cid:13) 

Revised: (cid:13)(cid:13)

Original: = (cid:107) x k,t − 1 x (cid:63) (cid:107) 

Revised: = (cid:107) x k,t − 1 x (cid:63) (cid:107)

Original: + s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: + s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k E (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k E (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F ( z k,t ; ξ k,t ) (cid:105) =
Revised: − ∇ x F ( z k,t ; ξ k,t ) (cid:105) =

Original: (cid:107) x k,t − 1 x (cid:63) (cid:107) 

Revised: (cid:107) x k,t − 1 x (cid:63) (cid:107)

Original: + s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: + s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F ( z k,t ) (cid:105) ≤ (cid:107) x k,t − 1 x (cid:63) (cid:107) 

Revised: − ∇ x F ( z k,t ) (cid:105) ≤ (cid:107) x k,t − 1 x (cid:63) (cid:107)

Original: + 2 s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: + 2 s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F ( 1 x (cid:63) , 1 y (cid:63) ; ξ k,t ) (cid:107)+ 2 s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) ; ξ k,t ) − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: − ∇ x F ( 1 x (cid:63) , 1 y (cid:63) ; ξ k,t ) (cid:107)+ 2 s 2 k E (cid:107)∇ x F ( 1 x (cid:63) , 1 y (cid:63) ; ξ k,t ) − ∇ x F ( z k,t ; ξ k,t ) (cid:107)+ 2 s k (cid:104) x k,t − 1 x (cid:63) , ∇ x F ( 1 x (cid:63) , 1 y (cid:63) )

Original: − ∇ x F (
Revised: − ∇ x F (

Original: [Equation]
Revised: [Equation]

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now simplify the inner product
Revised: We now simplify the inner product

Original: [Equation]
Revised: [Equation]

Original: term present in the r
Revised: term present in the r

Original: [Equation]
Revised: [Equation]

Original: h.s of (108).
Revised: h.s of (108).

Original: Recall the deﬁnition of Bregman distance V f i ,y ( x 1 , x 2 ) :
Revised: Recall the deﬁnition of Bregman distance V f i ,y ( x 1 , x 2 ) :

Original: [Equation]
Revised: [Equation]

Original: Using µ x strong convexity of f i ( · , y )
Revised: Using µ x strong convexity of f i ( · , y )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We now compute
Revised: We now compute

Original: [Equation]
Revised: [Equation]

Original: where the second last step follows from (111) and (113).
Revised: where the second last step follows from (111) and (113).

Original: On substituting (109) and (115) in (108), we obtain
Revised: On substituting (109) and (115) in (108), we obtain

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)≤ (cid:107) x k,t − 1 x (cid:63) (cid:107)
Revised: ) (cid:13)(cid:13)≤ (cid:107) x k,t − 1 x (cid:63) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ms
Revised: ms

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: xx
Revised: xx

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Revised: [Equation]

Revised: f i

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  completing the proof.
Revised: ) completing the proof.

Original: We have the following corollary.
Revised: We have the following corollary.

Original: Corollary E.4.
Revised: Corollary E.4.

Original: Let s k =
Revised: Let s k =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f L 2 k/ 2 for every k ≥ 0 .
Revised: f L 2 k/ 2 for every k ≥ 0 .

Original: Then, under the setting of Lemma E.3 ,
Revised: Then, under the setting of Lemma E.3 ,

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 1 , where b x,k = µ x s k − 4 s 2 k L 2 yx = , b y,k
Revised: for all t ≥ 1 , where b x,k = µ x s k − 4 s 2 k L 2 yx = , b y,k

Original: = µ y s k − 4 s 2 k L 2 xy .
Revised: = µ y s k − 4 s 2 k L 2 xy .

Original: E.2 Proof of Corollary E. 
Revised: E.2 Proof of Corollary E.

Original: As the step size s k = 1 

Revised: As the step size s k = 1

Original: Lκ f 2 k/ 2 , we have s k ≤ 14 L .
Revised: Lκ f 2 k/ 2 , we have s k ≤ 14 L .

Original: We now show that the terms 2 s k − 8 s 2 k L xx and 2 s k − 8 s 2 k L yy appearing in (106) and (107) are non-negative:
Revised: We now show that the terms 2 s k − 8 s 2 k L xx and 2 s k − 8 s 2 k L yy appearing in (106) and (107) are non-negative:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we get 2 s k − 8 s 2 k L yy ≥ 0 .
Revised: Similarly, we get 2 s k − 8 s 2 k L yy ≥ 0 .

Original: Recall
Revised: Recall

Original: [Equation]
Revised: [Equation]

Original: We now show that b x,k < 1 .
Revised: We now show that b x,k < 1 .

Original: [Equation]
Revised: [Equation]

Original: Therefore, b x,k ∈ (0 , 1) for every k ≥ 0 .
Revised: Therefore, b x,k ∈ (0 , 1) for every k ≥ 0 .

Original: In a similar fashion
Revised: In a similar fashion

Original: [Equation]
Revised: [Equation]

Original: we obtain b y,k ∈ (0 , 1) for every k ≥ 0 .
Revised: we obtain b y,k ∈ (0 , 1) for every k ≥ 0 .

Original: On adding (106) and (107)
Revised: On adding (106) and (107)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: −b
Revised: −

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows from non-negativity of V f i ,y ik,t ( x (cid:63) , x ik,t ) , V − f i ,x ik,t ( y (cid:63) , y ik,t ) , 2 s k − 8 s 2 k L xx and 2 s k − 8 s 2 k L
Revised: The last inequality follows from non-negativity of V f i ,y ik,t ( x (cid:63) , x ik,t ) , V − f i ,x ik,t ( y (cid:63) , y ik,t ) , 2 s k − 8 s 2 k L xx and 2 s k − 8 s 2 k L

Original: yy .
Revised: yy .

Original: We now establish a recursion for E [Φ k,t ] .
Revised: We now establish a recursion for E [Φ k,t ] .

Original: Lemma E.5.
Revised: Lemma E.5.

Original: Suppose { x k,t } t and { y k,t } t are the sequences generated by algorithm 2 with G =
Revised: Suppose { x k,t } t and { y k,t } t are the sequences generated by algorithm 2 with G =

Original: [ ∇ x F ( · ; ξ ); −∇ y F ( · ; ξ )] .
Revised: [ ∇ x F ( · ; ξ ); −∇ y F ( · ; ξ )] .

Original: Suppose Assumptions 3.1-3.5 and Assumptions E.1-E.2 hold.
Revised: Suppose Assumptions 3.1-3.5 and Assumptions E.1-E.2 hold.

Original: Let step size s k is chosen according to Corollary E.4.
Revised: Let step size s k is chosen according to Corollary E.4.

Original: Then, for every k ≥ 0 , the following holds in total expectation:
Revised: Then, for every k ≥ 0 , the following holds in total expectation:

Original: [Equation]
Revised: [Equation]

Original: (120) for all t ≥ 0 , where ρ k is deﬁned in equation (128) .
Revised: (120) for all t ≥ 0 , where ρ k is deﬁned in equation (128) .

Original: On adding (53) and (103)
Revised: On adding (53) and (103)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By taking conditional expectation on stochastic gradient at t -th step on both sides of above inequality and applying Tower property
Revised: By taking conditional expectation on stochastic gradient at t -th step on both sides of above inequality and applying Tower property

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: − s k G xk,t + s k ∇ x F ( 1 x (cid:63) ) , 1 y (cid:63) )
Revised: − s k G xk,t + s k ∇ x F ( 1 x (cid:63) ) , 1 y (cid:63) )

Original: (cid:13)(cid:13) 

Revised: (cid:13)(cid:13)

Original: + 2 s 2 k γ x,k (cid:16) 1 − γ x,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D xk,t − D (cid:63)x (cid:13)(cid:13) 2 ( I − W ) † + √ δ (1 − α x,k ) (cid:13)(cid:13)
Revised: + 2 s 2 k γ x,k (cid:16) 1 − γ x,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D xk,t − D (cid:63)x (cid:13)(cid:13) 2 ( I − W ) † + √ δ (1 − α x,k ) (cid:13)(cid:13)

Original: H xk,t − H (cid:63)x,k (cid:13)(cid:13) 2 + E (cid:13)(cid:13)(cid:13)
Revised: H xk,t − H (cid:63)x,k (cid:13)(cid:13) 2 + E (cid:13)(cid:13)(cid:13)

Original: y k,t − 1 y (cid:63) + s k G yk,t − s k ∇ y F
Revised: y k,t − 1 y (cid:63) + s k G yk,t − s k ∇ y F

Original: ( 1 x (cid:63) , 1 y (cid:63) )) (cid:13)(cid:13)(cid:13)+ 2 s 2 k γ y,k (cid:16) 1 − γ y,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13)(cid:13) D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13) 2 ( I − W ) † + √ δ (1 − α y,k ) (cid:13)(cid:13)(cid:13) H yk,t − H (cid:63)y,k (cid:13)(cid:13)(cid:13)≤ (1 − b x,k ) (cid:107) x k,t − 1 x (cid:63) (cid:107) 

Revised: ( 1 x (cid:63) , 1 y (cid:63) )) (cid:13)(cid:13)(cid:13)+ 2 s 2 k γ y,k (cid:16) 1 − γ y,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13)(cid:13) D yk,t − D (cid:63)y (cid:13)(cid:13)(cid:13) 2 ( I − W ) † + √ δ (1 − α y,k ) (cid:13)(cid:13)(cid:13) H yk,t − H (cid:63)y,k (cid:13)(cid:13)(cid:13)≤ (1 − b x,k ) (cid:107) x k,t − 1 x (cid:63) (cid:107)

Original: + (1 − b y,k ) (cid:107) y k,t − 1 y (cid:63) (cid:107)+ 2 s 2 k γ x,k (cid:16) 1 − γ x,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13)
Revised: + (1 − b y,k ) (cid:107) y k,t − 1 y (cid:63) (cid:107)+ 2 s 2 k γ x,k (cid:16) 1 − γ x,k 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13)

Original: D xk
Revised: D xk

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from inequality (119).
Revised: where the last inequality follows from inequality (119).

Original: By taking total expectation on both sides of above inequality and using the deﬁnition of Φ k,t
Revised: By taking total expectation on both sides of above inequality and using the deﬁnition of Φ k,t

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: E.4 Feasibility of Parameters for Algorithm
Revised: E.4 Feasibility of Parameters for Algorithm

Original: Parameters setting: From Corollary E.4, the step size used in Algorithm 2 is s k =
Revised: Parameters setting: From Corollary E.4, the step size used in Algorithm 2 is s k =

Original: [Equation]
Revised: [Equation]

Original: κ
Revised: κ

Original: [Equation]
Revised: [Equation]

Original: for every k ≥ 0 .
Revised: /for every k ≥ 0 .

Original: We choose the parameters involved in COMM procedure and other parameters γ x,k , γ y,k as follows:
Revised: We choose the parameters involved in COMM procedure and other parameters γ x,k , γ y,k as follows:

Original: [Equation]
Revised: [Equation]

Original: Feasibility of parameters: Above choice of parameters should satisfy the following conditions:
Revised: Feasibility of parameters: Above choice of parameters should satisfy the following conditions:

Original: [Equation]
Revised: [Equation]

Original: In this section, we show that all parameters speciﬁed in (126) satisfy all requirements of (130)-(135).
Revised: In this section, we show that all parameters speciﬁed in (126) satisfy all requirements of (130)-(135).

Original: Feasibility of α x,k and α y,k .
Revised: Feasibility of α x,k and α y,k .

Original: From (117) and (118), we have 0 < b x,k < 1 .
Revised: From (117) and (118), we have 0 < b x,k < 1 .

Original: Therefore, α x,k < 

Revised: Therefore, α x,k <

Original: + δ .
Revised: + δ .

Original: Moreover, √ δ 1+ δ ≤ 1 / 2 .
Revised: Moreover, √ δ 1+ δ ≤ 1 / 2 .

Original: Therefore, α x,k ≤ b x,k 2 √ δ < b x,k / √ δ .
Revised: Therefore, α x,k ≤ b x,k 2 √ δ < b x,k / √ δ .

Original: Hence, α x,k < min (cid:110) b x,k √ δ , 11+ δ (cid:111) .
Revised: Hence, α x,k < min (cid:110) b x,k √ δ , 11+ δ (cid:111) .

Original: Similarly, α y,k < min (cid:110) b y,k √ δ , 11+ δ (cid:111) because b y,k ∈ (0 , 1) .
Revised: Similarly, α y,k < min (cid:110) b y,k √ δ , 11+ δ (cid:111) because b y,k ∈ (0 , 1) .

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: The last inequality uses the relation √ δ 1+ δ ≤ 12 .
Revised: The last inequality uses the relation √ δ 1+ δ ≤ 12 .

Original: Using (118), we have b x,k ≤ 1 4 κ x κ f 2 k/ 2 < 0 .
Revised: Using (118), we have b x,k ≤ 1 4 κ x κ f 2 k/ 2 < 0 .

Original: This allows us to use the inequality 2 x − 2 x 2 ≥ x/ 2 for all 0 ≤ x ≤ 0 .
Revised: This allows us to use the inequality 2 x − 2 x 2 ≥ x/ 2 for all 0 ≤ x ≤ 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: where the third inequality uses √ δ 1+ δ ≤ 12 and fourth inequality uses b x,k 2(1+ δ ) 2 < 1 .
Revised: where the third inequality uses √ δ 1+ δ ≤ 12 and fourth inequality uses b x,k 2(1+ δ ) 2 < 1 .

Original: We know that b y,k ∈ (0 , 1) .
Revised: We know that b y,k ∈ (0 , 1) .

Original: Therefore, by following similar steps, the chosen γ y,k is also feasible.
Revised: Therefore, by following similar steps, the chosen γ y,k is also feasible.

Original: As γ x,k < 2 − 2 √ δα x,k λ max ( I − W ) < 2 λ max ( I − W ) .
Revised: As γ x,k < 2 − 2 √ δα x,k λ max ( I − W ) < 2 λ max ( I − W ) .

Original: Notice that λ m − 1 ( I − W ) <
Revised: Notice that λ m − 1 ( I − W ) <

Original: λ max ( I − W )
Revised: λ max ( I − W )

Original: Therefore, γ
Revised: Therefore, γ

Original: [Equation]
Revised: [Equation]

Original: Similarly, γ y,k 2 λ m − 1 ( I − W ) < 1 .
Revised: Similarly, γ y,k 2 λ m − 1 ( I − W ) < 1 .

Original: Feasibility of M x,k and M y,k .
Revised: Feasibility of M x,k and M y,k .

Original: Recall M x,k = 1 − √ δα x,k 1 − γx,k 2 λ max ( I − W ) and M y,k = 1 − √
Revised: Recall M x,k = 1 − √ δα x,k 1 − γx,k 2 λ max ( I − W ) and M y,k = 1 − √

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: > 0 .
Revised: > 0 .

Original: Therefore, M x,k ∈ (0 , 1) .
Revised: Therefore, M x,k ∈ (0 , 1) .

Original: The feasibility of M y,k can be proved similarly.
Revised: The feasibility of M y,k can be proved similarly.

Original: Feasibility of 1 − b x,k M x,k and 1 − b y,k M y,k .
Revised: Feasibility of 1 − b x,k M x,k and 1 − b y,k M y,k .

Original: Observe that
Revised: Observe that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let θ = 1 − b x,k , σ = 1 − b x,k 4(1+ δ ) 2 and χ = √ δb x,k 1+ δ .
Revised: Let θ = 1 − b x,k , σ = 1 − b x,k 4(1+ δ ) 2 and χ = √ δb x,k 1+ δ .

Original: We now show that χ ≤ b − a .
Revised: We now show that χ ≤ b − a .

Original: χ = √ δb x,k 1 + δ ≤ b x,k = b x,k 2 + b x,k (cid:18) 1 − 4(1 + δ )(cid:19) − b x,k (cid:18) 1 − 4(1 + δ )(cid:19) =
Revised: χ = √ δb x,k 1 + δ ≤ b x,k = b x,k 2 + b x,k (cid:18) 1 − 4(1 + δ )(cid:19) − b x,k (cid:18) 1 − 4(1 + δ )(cid:19) =

Original: b x,k (cid:18) 1 − 4(1 + δ )(cid:19)
Revised: b x,k (cid:18) 1 − 4(1 + δ )(cid:19)

Original: + b x,k 2 − b x,k + b x,k 4(1 + δ )≤ b x,k (cid:18) 1 − 4(1 + δ )(cid:19) + b x,k 2 − b x,k + b x,k= b x,k (cid:18) 1 − 4(1 + δ )(cid:19) =
Revised: + b x,k 2 − b x,k + b x,k 4(1 + δ )≤ b x,k (cid:18) 1 − 4(1 + δ )(cid:19) + b x,k 2 − b x,k + b x,k= b x,k (cid:18) 1 − 4(1 + δ )(cid:19) =

Original: σ − θ.
Revised: σ − θ.

Original: Notice that 0 ≤ χ ≤ σ − θ , and σθ ≤ ( θ + χ )( σ − χ ) , i.e., (1 − b x,k ) (cid:18) 1 − b x,k 4(1 + δ )(cid:19) ≤ (cid:32) 1 − b x,k + √ δb x,k 1 + δ (cid:33) (cid:32) 1 − b x,k 4(1 + δ ) 2 − √ δb x,k 1 + δ (cid:33) .
Revised: Notice that 0 ≤ χ ≤ σ − θ , and σθ ≤ ( θ + χ )( σ − χ ) , i.e., (1 − b x,k ) (cid:18) 1 − b x,k 4(1 + δ )(cid:19) ≤ (cid:32) 1 − b x,k + √ δb x,k 1 + δ (cid:33) (cid:32) 1 − b x,k 4(1 + δ ) 2 − √ δb x,k 1 + δ (cid:33) .

Original: Substituting the above inequality in (141), we obtain
Revised: Substituting the above inequality in (141), we obtain

Original: [Equation]
Revised: [Equation]

Original: − b x,k
Revised: − b x,k

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from (117) .
Revised: where the last step follows from (117) .

Original: Notice that (cid:16) 1 − 1 √ 2 (cid:17) 18 κ 2 f 1 2 k/ 2 ∈ (0 , 1) and hence 1 − b x,k M x,k ∈ (0 , 1) .
Revised: Notice that (cid:16) 1 − 1 √ 2 (cid:17) 18 κ 2 f 1 2 k/ 2 ∈ (0 , 1) and hence 1 − b x,k M x,k ∈ (0 , 1) .

Original: Similarly, we obtain
Revised: Similarly, we obtain

Original: [Equation]
Revised: [Equation]

Original: We now prove another intermediate result that shows the convergence behavior of E [Φ k,t +1 ] .
Revised: We now prove another intermediate result that shows the convergence behavior of E [Φ k,t +1 ] .

Original: Lemma E.6.
Revised: Lemma E.6.

Original: Let Φ k,t
Revised: Let Φ k,t

Original: +

Revised: +

Original: and ρ be as deﬁned in (129) .
Revised: and ρ be as deﬁned in (129) .

Original: Under the settings of Lemma E.5,
Revised: Under the settings of Lemma E.5,

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 0 .
Revised: for all t ≥ 0 .

Original: E.5 Proof of Lemma E. 
Revised: E.5 Proof of Lemma E.

Original: First, we show that ρ k ≤ 1 − ρ 2 k/ 2 , where these quantities are deﬁned in (129).
Revised: First, we show that ρ k ≤ 1 − ρ 2 k/ 2 , where these quantities are deﬁned in (129).

Original: To prove this relation, we ﬁrst simplify the terms 1 − γ x,k 2 λ m − 1 ( I − W ) and 1 − α x,k appearing in the deﬁnition of ρ k :
Revised: To prove this relation, we ﬁrst simplify the terms 1 − γ x,k 2 λ m − 1 ( I − W ) and 1 − α x,k appearing in the deﬁnition of ρ k :

Original: [Equation]
Revised: [Equation]

Original: Similarly, we obtain
Revised: Similarly, we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: Let us recall ρ k : ρ k = max
Revised: Let us recall ρ k : ρ k = max

Original: [Equation]
Revised: [Equation]

Original: where ρ is deﬁned in equation (129).
Revised: where ρ is deﬁned in equation (129).

Original: Using Lemma E.5, we have
Revised: Using Lemma E.5, we have

Original: [Equation]
Revised: [Equation]

Original: where a k = 1 − ρ 2 k/ 2 and C 1 = 2 m ( σ 2 x + σ 2 y )
Revised: where a k = 1 − ρ 2 k/ 2 and C 1 = 2 m ( σ 2 x + σ 2 y )

Original: We now unroll the above recursion to obtain
Revised: We now unroll the above recursion to obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Using C 1 = 2 m ( σ 2 x + σ 2 y )
Revised: Using C 1 = 2 m ( σ 2 x + σ 2 y )

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: F Proof of Theorem 4. 
Revised: F Proof of Theorem 4.

Original: This proof is based on several intermediate results proved in Appendices C-E.
Revised: This proof is based on several intermediate results proved in Appendices C-E.

Original: Hence it would be useful to refer to those results in order to appreciate the proof of Theorem 4.1.
Revised: Hence it would be useful to refer to those results in order to appreciate the proof of Theorem 4.1.

Original: We divide the proof of Theorem 4.1 into two parts.
Revised: We divide the proof of Theorem 4.1 into two parts.

Original: We ﬁrst ﬁnd the total number outer iterations required by Algorithm 2 to achieve target accuracy (cid:15) .
Revised: We ﬁrst ﬁnd the total number outer iterations required by Algorithm 2 to achieve target accuracy (cid:15) .

Original: Then we derive the total gradient computation complexity of Algorithm 2.
Revised: Then we derive the total gradient computation complexity of Algorithm 2.

Original: F.0.1 Total Number of Outer Iterations
Revised: F.0.1 Total Number of Outer Iterations

Original: We have the following initializations at every k + 1 outer iterate:
Revised: We have the following initializations at every k + 1 outer iterate:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: By taking total expectation on both sides and using Lemma E.6, we obtain
Revised: By taking total expectation on both sides and using Lemma E.6, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: We now focus on bounding Φ k, 0 in terms of (cid:107) x k, 0 − 1 x (cid:63) (cid:107) 2 and (cid:107) y k, 0 − 1 y
Revised: We now focus on bounding Φ k, 0 in terms of (cid:107) x k, 0 − 1 x (cid:63) (cid:107) 2 and (cid:107) y k, 0 − 1 y

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: Recall from (129)
Revised: Recall from (129)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: We now bound the various terms appearing on the r.h.s.
Revised: We now bound the various terms appearing on the r.h.s.

Original: of the above equation.
Revised: of the above equation.

Original: First,
Revised: First,

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: We now have
Revised: We now have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting the above bounds in (158) gives
Revised: Substituting the above bounds in (158) gives

Original: [Equation]
Revised: [Equation]

Original: On substituting the above inequality in (156)
Revised: On substituting the above inequality in (156)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: By using above inequalities into (167)
Revised: By using above inequalities into (167)

Original: [Equation]
Revised: [Equation]

Original: we obtain ,
Revised: we obtain ,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: where A 1 =
Revised: where A 1 =

Original: [Equation]
Revised: [Equation]

Original: To proceed further, we derive lower bounds on M x,k and M y,k .
Revised: To proceed further, we derive lower bounds on M x,k and M y,k .

Original: Lower bound on M x,k .
Revised: Lower bound on M x,k .

Original: Using (118), we have
Revised: Using (118), we have

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: M x,k is lower bounded by 1 −
Revised: M x,k is lower bounded by 1 −

Original: [Equation]
Revised: [Equation]

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: where A 3 = m ( σ 2 x + σ 2 y ) 8 ˜ MρL 2 κ 2 f .
Revised: where A 3 = m ( σ 2 x + σ 2 y ) 8 ˜ MρL 2 κ 2 f .

Original: On substituting these bounds in (167), we obtain
Revised: On substituting these bounds in (167), we obtain

Original: [Equation]
Revised: [Equation]

Original: where e k =
Revised: where e k =

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using (178) recursively yields
Revised: Using (178) recursively yields

Original: [Equation]
Revised: [Equation]

Original: By choosing k = K ( (cid:15) )
Revised: By choosing k = K ( (cid:15) )

Original: =
Revised: =

Original: max { log
Revised: max { log

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: } , we obtain
Revised: } , we obtain

Original: [Equation]
Revised: [Equation]

Original: Above choice of K ( (cid:15) ) involves constants A 1 , A 2 , A 3 .
Revised: Above choice of K ( (cid:15) ) involves constants A 1 , A 2 , A 3 .

Original: We write each term in K ( (cid:15) ) in terms of parameters m, L, κ f , κ g , δ and σ as follows:
Revised: We write each term in K ( (cid:15) ) in terms of parameters m, L, κ f , κ g , δ and σ as follows:

Original: We have log
Revised: We have log

Original: [Equation]
Revised: [Equation]

Original: We also have 2 log
Revised: We also have 2 log

Original: [Equation]
Revised: [Equation]

Original: The last equality follows because 1 / ρ = O (cid:16) (1 + δ ) 2 κ 2 f κ g (cid:17) and
Revised: The last equality follows because 1 / ρ = O (cid:16) (1 + δ ) 2 κ 2 f κ g (cid:17) and

Original: ˜ M
Revised: ˜ M

Original: = O (1) .
Revised: = O (1) .

Original: F.0.2 Gradient Computation Complexity
Revised: F.0.2 Gradient Computation Complexity

Original: The gradient computation complexity T grad ( (cid:15) ) is bounded by the following computation
Revised: The gradient computation complexity T grad ( (cid:15) ) is bounded by the following computation

Original: T grad ( (cid:15) ) =
Revised: T grad ( (cid:15) ) =

Original: [Equation]
Revised: [Equation]

Original: Notice that above complexity does not contain 1 / ˜ M term because ˜ M ∈ (cid:2) 1115 ,
Revised: Notice that above complexity does not contain 1 / ˜ M term because ˜ M ∈ (cid:2) 1115 ,

Original: [Equation]
Revised: [Equation]

Original: F.0.3 Communication Complexity
Revised: F.0.3 Communication Complexity

Original: We ﬁnish the proof of Theorem 4.1 by computing the communication complexity as follows.
Revised: We ﬁnish the proof of Theorem 4.1 by computing the communication complexity as follows.

Original: [Equation]
Revised: [Equation]

Original: F.1 Algorithm 2 behavior in deterministic setting
Revised: F.1 Algorithm 2 behavior in deterministic setting

Original: In this section, we brieﬂy discuss that Algorithm 2 converges to the saddle point solution with linear rates when σ x = σ y = 0 , where σ x and σ y are the bounds on the variances of stochastic gradients of
Revised: In this section, we brieﬂy discuss that Algorithm 2 converges to the saddle point solution with linear rates when σ x = σ y = 0 , where σ x and σ y are the bounds on the variances of stochastic gradients of

Original: GSGO (see Assumption E.1).
Revised: GSGO (see Assumption E.1).

Original: Recall the recursive relation in Lemma E.5:
Revised: Recall the recursive relation in Lemma E.5:

Original: [Equation]
Revised: [Equation]

Original: On substituting k = 0 and σ x = σ y = 0 in above inequality, we obtain
Revised: On substituting k = 0 and σ x = σ y = 0 in above inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: 183) where ρ 0 is deﬁned in (128).
Revised: 183) where ρ 0 is deﬁned in (128).

Original: By unrolling above recursion in t , we get
Revised: By unrolling above recursion in t , we get

Original: [Equation]
Revised: [Equation]

Original: Note that M x, 0 (cid:107) x 0 ,t − 1 x (cid:63) (cid:107) 2 + M y, 0 (cid:107) y 0 ,t − 1 y
Revised: Note that M x, 0 (cid:107) x 0 ,t − 1 x (cid:63) (cid:107) 2 + M y, 0 (cid:107) y 0 ,t − 1 y

Original: [Equation]
Revised: [Equation]

Original: (cid:107)≤ Φ 0 ,t from (105).
Revised: (cid:107)≤ Φ 0 ,t from (105).

Original: Therefore,
Revised: Therefore,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: E [Φ 0 ,t ] min { M x, 0 ,M y, 0 } .
Revised: E [Φ 0 ,t ] min { M x, 0 ,M y, 0 } .

Original: Under the above settings, Algorithmneeds T grad (
Revised: Under the above settings, Algorithmneeds T grad (

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: (cid:17) gradient computations and communications to achieve E (cid:13)(cid:13) x 0 ,T grad ( (cid:15) ) − 1 x (cid:63) (cid:13)(cid:13) 2 + E (cid:13)(cid:13)
Revised: (cid:17) gradient computations and communications to achieve E (cid:13)(cid:13) x 0 ,T grad ( (cid:15) ) − 1 x (cid:63) (cid:13)(cid:13) 2 + E (cid:13)(cid:13)

Original: y 0 ,T grad ( (cid:15) ) − 1 y (cid:63) (cid:13)(cid:13) 2 ≤ (cid:15) .
Revised: y 0 ,T grad ( (cid:15) ) − 1 y (cid:63) (cid:13)(cid:13) 2 ≤ (cid:15) .

Original: We now write T grad ( (cid:15) ) in terms of κ f , κ g and δ .
Revised: We now write T grad ( (cid:15) ) in terms of κ f , κ g and δ .

Original: Using (151), we have ρ 0 ≤ 1 − ρ , where ρ is deﬁned in (129).
Revised: Using (151), we have ρ 0 ≤ 1 − ρ , where ρ is deﬁned in (129).

Original: Notice thatlog(1 /ρ 0 ) ≤ 1 − log(1 − ρ ) ≤ 5 ρ .
Revised: Notice thatlog(1 /ρ 0 ) ≤ 1 − log(1 − ρ ) ≤ 5 ρ .

Original: Therefore,
Revised: Therefore,

Original: T grad ( (cid:15) ) =(cid:32) min (cid:40)(cid:18)
Revised: T grad ( (cid:15) ) =(cid:32) min (cid:40)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:41)(cid:33) −log (cid:18) Φ
Revised: (cid:41)(cid:33) −log (cid:18) Φ

Original: [Equation]
Revised: [Equation]

Original: G Proofs for the Finite Sum Setting
Revised: G Proofs for the Finite Sum Setting

Original: In this section, we prove all results related to convergence analysis of Algorithm 3 in Section 5.
Revised: In this section, we prove all results related to convergence analysis of Algorithm 3 in Section 5.

Original: For theoretical analysis of Algorithm 3, we make the following smoothness assumptions on f ij particular to the ﬁnite sum setting.
Revised: For theoretical analysis of Algorithm 3, we make the following smoothness assumptions on f ij particular to the ﬁnite sum setting.

Original: Assumption G.1.
Revised: Assumption G.1.

Original: Assume that each f ij ( x, y ) is L xx smooth in x ; for every ﬁxed y ,
Revised: Assume that each f ij ( x, y ) is L xx smooth in x ; for every ﬁxed y ,

Original: [Equation]
Revised: [Equation]

Original: Assumption G.2.
Revised: Assumption G.2.

Original: Assume that each − f ij ( x, y ) is L yy smooth in y i.e., for every ﬁxed x , (cid:107)−∇
Revised: Assume that each − f ij ( x, y ) is L yy smooth in y i.e., for every ﬁxed x , (cid:107)−∇

Original: [Equation]
Revised: [Equation]

Original: Assumption G.3.
Revised: Assumption G.3.

Original: Assume that each ∇ x f ij ( x, y ) is L xy Lipschitz in y i.e., for every ﬁxed x , (cid:107)∇
Revised: Assume that each ∇ x f ij ( x, y ) is L xy Lipschitz in y i.e., for every ﬁxed x , (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: − ∇
Revised: − ∇

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107)≤ L xy (cid:107) y
Revised: ) (cid:107)≤ L xy (cid:107) y

Original: [Equation]
Revised: [Equation]

Original: y 2 (cid:107) , ∀ y 1 , y 2 ∈ R d y .
Revised: y 2 (cid:107) , ∀ y 1 , y 2 ∈ R d y .

Original: Assumption G.4.
Revised: Assumption G.4.

Original: Assume that each ∇ y f ij ( x, y ) is L yx Lipschitz in x i.e., for every ﬁxed y ,
Revised: Assume that each ∇ y f ij ( x, y ) is L yx Lipschitz in x i.e., for every ﬁxed y ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , y )
Revised: , y )

Original: [Equation]
Revised: [Equation]

Original: ∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: We begin with few intermediate results which will help us in getting the ﬁnal convergence result.
Revised: We begin with few intermediate results which will help us in getting the ﬁnal convergence result.

Original: Lemma G.5.
Revised: Lemma G.5.

Original: Let { x t } t , { y t } t be the sequences generated by Algorithm 3 with G xt and G yt obtained from SVRGO.
Revised: Let { x t } t , { y t } t be the sequences generated by Algorithm 3 with G xt and G yt obtained from SVRGO.

Original: Then, under Assumptions 3.1-3.2 and Assumptions G.1-G.4, the following holds for all
Revised: Then, under Assumptions 3.1-3.2 and Assumptions G.1-G.4, the following holds for all

Original: [Equation]
Revised: [Equation]

Original: where p min := min i,j { p ij } .
Revised: where p min := min i,j { p ij } .

Original: G.1 Proof of Lemma G. 
Revised: G.1 Proof of Lemma G.

Original: We begin the proof by bounding the primal ( x ) and dual ( y ) updates on the l.h.s.
Revised: We begin the proof by bounding the primal ( x ) and dual ( y ) updates on the l.h.s.

Original: of (185) separately.
Revised: of (185) separately.

Original: In particular, we show that
Revised: In particular, we show that

Original: [Equation]
Revised: [Equation]

Original: (cid:63) )
Revised: (cid:63) )

Original: (cid:107)≤ (1 − µ x s ) (cid:107) x
Revised: (cid:107)≤ (1 − µ x s ) (cid:107) x

Original: t − 1 x (cid:63) (cid:107)+ 2 sn 2 p min m (cid:88) i = n (cid:88)
Revised: t − 1 x (cid:63) (cid:107)+ 2 sn 2 p min m (cid:88) i = n (cid:88)

Original: j = (cid:13)(cid:13) ∇ x f ij ( z it )
Revised: j = (cid:13)(cid:13) ∇ x f ij ( z it )

Original: − ∇ x f ij ( z (cid:63) )
Revised: − ∇ x f ij ( z (cid:63) )

Original: (cid:13)(cid:13) 

Revised: (cid:13)(cid:13)

Original: + 2 sn 2 p
Revised: + 2 sn 2 p

Original: min m (cid:88) i = n (cid:88)
Revised: min m (cid:88) i = n (cid:88)

Original: j = (cid:13)(cid:13)
Revised: j = (cid:13)(cid:13)

Original: ∇ x f ij (˜ z it ) − ∇ x f ij ( z (cid:63) ) (cid:13)(cid:13)− 2 s m (cid:88) i =
Revised: ∇ x f ij (˜ z it ) − ∇ x f ij ( z (cid:63) ) (cid:13)(cid:13)− 2 s m (cid:88) i =

Original: V f i ,y it ( x (cid:63) , x it ) + 2 s ( F ( 1 x (cid:63) , y t ) − F ( z t ) + F ( x t , 1 y (cid:63) )
Revised: V f i ,y it ( x (cid:63) , x it ) + 2 s ( F ( 1 x (cid:63) , y t ) − F ( z t ) + F ( x t , 1 y (cid:63) )

Original: − F ( 1 z (cid:63) ))
Revised: − F ( 1 z (cid:63) ))

Original: (186) and
Revised: (186) and

Original: E (cid:107) y t − 1 y (cid:63) + s G yt − s ∇ y F ( 1 x (cid:63) , 1 y (cid:63) )
Revised: E (cid:107) y t − 1 y (cid:63) + s G yt − s ∇ y F ( 1 x (cid:63) , 1 y (cid:63) )

Original: (cid:107)≤ (1 − sµ y ) (cid:107) y t − 1 y (cid:63) (cid:107) 

Revised: (cid:107)≤ (1 − sµ y ) (cid:107) y t − 1 y (cid:63) (cid:107)

Original: + 2 s ( − F ( x t , 1 y (cid:63) )
Revised: + 2 s ( − F ( x t , 1 y (cid:63) )

Original: + F ( 1 z (cid:63) ) − F ( 1 x (cid:63) , y t )
Revised: + F ( 1 z (cid:63) ) − F ( 1 x (cid:63) , y t )

Original: + F ( z t ))
Revised: + F ( z t ))

Original: − 2 s m (cid:88) i =
Revised: − 2 s m (cid:88) i =

Original: V − f i ,x it ( y (cid:63) , y it ) + 2 sn 2 p
Revised: V − f i ,x it ( y (cid:63) , y it ) + 2 sn 2 p

Original: min m (cid:88) i = n (cid:88)
Revised: min m (cid:88) i = n (cid:88)

Original: j = (cid:13)(cid:13)
Revised: j = (cid:13)(cid:13)

Original: ∇ y f ij ( z it ) − ∇ y f ij ( z
Revised: ∇ y f ij ( z it ) − ∇ y f ij ( z

Original: [Equation]
Revised: [Equation]

Original: Observe that (186) and (187) are similar
Revised: Observe that (186) and (187) are similar

Original: [Equation]
Revised: [Equation]

Original: and we only prove (186) in Section G.1.1 below.
Revised: and we only prove (186) in Section G.1.1 below.

Original: Adding (186) and (187), we obtain
Revised: Adding (186) and (187), we obtain

Original: [Equation]
Revised: [Equation]

Original: To ﬁnish the proof of Lemma G.5, we bound the last two terms of (189) as shown in Section G.1.2.
Revised: To ﬁnish the proof of Lemma G.5, we bound the last two terms of (189) as shown in Section G.1.2.

Original: G.1.1 Proof of (186)
Revised: G.1.1 Proof of (186)

Original: First, consider the primal update term
Revised: First, consider the primal update term

Original: [Equation]
Revised: [Equation]

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality and second last equality follows respectively from step (1) of SVRGO and deﬁnition of f i ( x, y ) .
Revised: where the ﬁrst equality and second last equality follows respectively from step (1) of SVRGO and deﬁnition of f i ( x, y ) .

Original: Substituting the above in the last term of (190)
Revised: Substituting the above in the last term of (190)

Original: [Equation]
Revised: [Equation]

Original: we see that
Revised: we see that

Original: [Equation]
Revised: [Equation]

Original: Substituting (111) (i.e., Bregman distance) and (113) (i.e., strong convexity of f ) in (192), we obtain
Revised: Substituting (111) (i.e., Bregman distance) and (113) (i.e., strong convexity of f ) in (192), we obtain

Original: E (cid:107) x
Revised: E (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: s ∇
Revised: s ∇

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: ≤ m
Revised: ≤ m

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − 2 s
Revised: − 2 s

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: − f
Revised: − f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:63)
Revised: (cid:63)

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: i = (cid:16) f i ( x it , y (cid:63) )
Revised: i = (cid:16) f i ( x it , y (cid:63) )

Original: − f i ( z (cid:63) ) − µ x 2 (cid:13)(cid:13) x
Revised: − f i ( z (cid:63) ) − µ x 2 (cid:13)(cid:13) x

Original: it − x (cid:63) (cid:13)(cid:13) 2 (cid:17) = m (cid:88)
Revised: it − x (cid:63) (cid:13)(cid:13) 2 (cid:17) = m (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13) x it − x (cid:63) (cid:13)(cid:13) 2 + sm (cid:88) i =
Revised: (cid:13)(cid:13) x it − x (cid:63) (cid:13)(cid:13) 2 + sm (cid:88) i =

Original: E (cid:13)(cid:13)(cid:13)(cid:13) 1 np il (cid:0) ∇ x f il ( z it ) − ∇ x f il (˜ z it ) (cid:1) + ∇ x f i (˜ z it ) − ∇ x f i ( x (cid:63) , y (cid:63) )
Revised: E (cid:13)(cid:13)(cid:13)(cid:13) 1 np il (cid:0) ∇ x f il ( z it ) − ∇ x f il (˜ z it ) (cid:1) + ∇ x f i (˜ z it ) − ∇ x f i ( x (cid:63) , y (cid:63) )

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: + 2 s ( F ( 1 x (cid:63) , y t ) − F ( z t ))
Revised: + 2 s ( F ( 1 x (cid:63) , y t ) − F ( z t ))

Original: − 2 s m (cid:88) i =
Revised: − 2 s m (cid:88) i =

Original: V f i ,y it ( x (cid:63) , x it ) + 2 s ( F ( x t , 1 y (cid:63) )
Revised: V f i ,y it ( x (cid:63) , x it ) + 2 s ( F ( x t , 1 y (cid:63) )

Original: − F ( 1 z (cid:63) )) − sµ x (cid:107) x
Revised: − F ( 1 z (cid:63) )) − sµ x (cid:107) x

Original: t − 1 x (cid:63) (cid:107)= (1 − µ x s ) (cid:107) x t − 1 x (cid:63) (cid:107) 

Revised: t − 1 x (cid:63) (cid:107)= (1 − µ x s ) (cid:107) x t − 1 x (cid:63) (cid:107)

Original: + sm (cid:88) i =
Revised: + sm (cid:88) i =

Original: E (cid:13) (cid:13)(cid:13)(cid:13) 1 np il (cid:0) ∇ x f il ( z i t ) − ∇ x f il (˜ z i t ) (cid:1) + ∇ x f i (˜ z it ) − ∇ x f i ( x (cid:63) , y (cid:63) ) (cid:13) (cid:13)(cid:13)(cid:13)
Revised: E (cid:13) (cid:13)(cid:13)(cid:13) 1 np il (cid:0) ∇ x f il ( z i t ) − ∇ x f il (˜ z i t ) (cid:1) + ∇ x f i (˜ z it ) − ∇ x f i ( x (cid:63) , y (cid:63) ) (cid:13) (cid:13)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: m (cid:88)
Revised: m (cid:88)

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now we bound the second term on the r.h.s.
Revised: Now we bound the second term on the r.h.s.

Original: of (193) in terms of (cid:107) x t − 1 x (cid:63) (cid:107) 2 and (cid:107) y t − 1 y (cid:63) (cid:107) 2 as follows:
Revised: of (193) in terms of (cid:107) x t − 1 x (cid:63) (cid:107) 2 and (cid:107) y t − 1 y (cid:63) (cid:107) 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: 194) where p min = min i,j { p ij } .
Revised: 194) where p min = min i,j { p ij } .

Original: Let u i =
Revised: Let u i =

Original: (cid:110) ∇ x f il (˜ z it ) −∇ x f il ( z (cid:63) )
Revised: (cid:110) ∇ x f il (˜ z it ) −∇ x f il ( z (cid:63) )

Original: np il : l ∈ { 1 , 2 , . . . , n } (cid:111) be a random variable with probability distribution P i
Revised: np il : l ∈ { 1 , 2 , . . . , n } (cid:111) be a random variable with probability distribution P i

Original: = { p il : l ∈ { 1 , 2 , . . . , n }} .
Revised: = { p il : l ∈ { 1 , 2 , . . . , n }} .

Original: [Equation]
Revised: [Equation]

Original: We know that E (cid:107) u i − Eu i (cid:107) 2 ≤ E (cid:107) u
Revised: We know that E (cid:107) u i − Eu i (cid:107) 2 ≤ E (cid:107) u

Original: i (cid:107) 2 .
Revised: i (cid:107) 2 .

Original: Therefore,
Revised: Therefore,

Original: E (cid:13)(cid:13)(cid:13)(cid:13)
Revised: E (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: f ij (˜ z
Revised: f ij (˜ z

Original: [Equation]
Revised: [Equation]

Original: ) − ∇ x f ij ( z (cid:63) ) np ij −
Revised: ) − ∇ x f ij ( z (cid:63) ) np ij −

Original: [Equation]
Revised: [Equation]

Original: ) (cid:1)(cid:13)(cid:13)(cid:13)(cid:13) ≤ E (cid:13)(cid:13)(cid:13)(cid:13) ∇ x f ij 
Revised: ) (cid:1)(cid:13)(cid:13)(cid:13)(cid:13) ≤ E (cid:13)(cid:13)(cid:13)(cid:13) ∇ x f ij ( [Equation] np

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: =
Revised: =

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Revised: n (cid:88)

Original:  (cid:13)(cid:13)(cid:13)(cid:13)
Revised: j = (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: it
Revised: f ij (

Original: [Equation]
Revised: [Equation]

Original:  )p
Revised: z it )

Original: [Equation]
Revised: [Equation]

Original:  (cid:13)(cid:13)(cid:13)(cid:13) 
Revised: ij ( z (cid:63) ) p ij (cid:13)(cid:13)(cid:13)(cid:13) p ij =nn (cid:88)

Original: [Equation]
Revised: [Equation]

Original:   p
Revised: =  p

Original: [Equation]
Revised: [Equation]

Original: =n
Revised: (cid:13)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ n
Revised: (cid:63)

Original: [Equation]
Revised: [Equation]

Original:  p min n 
Revised: (cid:13) (cid:13)≤ n 2 p min n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:63)
Revised: = (cid:13)(cid:13)

Original: [Equation] 
Revised: [Equation] [Equation] it ) [Equation]

Revised: f ij (

Revised: [Equation]

Revised: (cid:63)

Original: [Equation]
Revised: [Equation]

Revised: (cid:13)(cid:13) 2 .

Original: By substituting the above inequality in (194)
Revised: By substituting the above inequality in (194)

Original: [Equation]
Revised: [Equation]

Original: we obtain s 2 m (cid:88)
Revised: we obtain s 2 m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Substituting this inequality in (193) we obtain (186).
Revised: Substituting this inequality in (193) we obtain (186).

Original: G.1.2 Finishing the Proof of Lemma G. 
Revised: G.1.2 Finishing the Proof of Lemma G.

Original: We now compute upper bounds on the last two terms present in (189) using smoothness assumptions.
Revised: We now compute upper bounds on the last two terms present in (189) using smoothness assumptions.

Original: First, observe that
Revised: First, observe that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Proposition C.
Revised: where the last inequality follows from Proposition C.

Original: [Equation]
Revised: [Equation]

Original: Proposition C.5 and Assumptions G.3-G
Revised: Proposition C.5 and Assumptions G.3-G

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Adding up the above inequality for j = 1 to n and using (27)-(28), we obtain n (cid:88)
Revised: Adding up the above inequality for j = 1 to n and using (27)-(28), we obtain n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: it
Revised: it

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ( x (cid:63)
Revised: ( x (cid:63)

Original: [Equation]
Revised: [Equation]

Original: y it )
Revised: y it )

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: it , y it )
Revised: it , y it )

Original: [Equation]
Revised: [Equation]

Original: (cid:10)
Revised: (cid:10)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: it
Revised: it

Original: [Table] [Table]
Revised: [Table] [Table]

Original: − x it (cid:11)(cid:1) + 4 L yy (cid:0) − nf i ( x it , y (cid:63) )
Revised: − x it (cid:11)(cid:1) + 4 L yy (cid:0) − nf i ( x it , y (cid:63) )

Original: + nf i ( x it , y it ) − (cid:10) − n ∇ y f i ( x it , y it ) , y (cid:63) − y it (cid:11)(cid:1) + 2 nL 2 xy (cid:13)(cid:13)
Revised: + nf i ( x it , y it ) − (cid:10) − n ∇ y f i ( x it , y it ) , y (cid:63) − y it (cid:11)(cid:1) + 2 nL 2 xy (cid:13)(cid:13)

Original: y it − y (cid:63) (cid:13)(cid:13) 2 + 2 nL 2 yx (cid:13)(cid:13) x
Revised: y it − y (cid:63) (cid:13)(cid:13) 2 + 2 nL 2 yx (cid:13)(cid:13) x

Original: it − x (cid:63) (cid:13)(cid:13)= 4 nL xx V f i ,y it ( x (cid:63) , x it ) + 4 nL
Revised: it − x (cid:63) (cid:13)(cid:13)= 4 nL xx V f i ,y it ( x (cid:63) , x it ) + 4 nL

Original: yy V − f i ,x it ( y (cid:63) , y i t )
Revised: yy V − f i ,x it ( y (cid:63) , y i t )

Original: + 2 nL 2 xy (cid:13) (cid:13)
Revised: + 2 nL 2 xy (cid:13) (cid:13)

Original: y it − y (cid:63) (cid:13) (cid:13) 2 + 2 nL 2 yx (cid:13) (cid:13) x
Revised: y it − y (cid:63) (cid:13) (cid:13) 2 + 2 nL 2 yx (cid:13) (cid:13) x

Original: it − x (cid:63) (cid:13) (cid:13) 2 , (199) where the second last step follows from the structure of f i ( x, y )
Revised: it − x (cid:63) (cid:13) (cid:13) 2 , (199) where the second last step follows from the structure of f i ( x, y )

Original: =
Revised: =

Original: 1 n (cid:80) nj =1 f ij ( x, y ) .
Revised: 1 n (cid:80) nj =1 f ij ( x, y ) .

Original: Therefore, 2 sn 2 p min m (cid:88) i = n (cid:88)
Revised: Therefore, 2 sn 2 p min m (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: ∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ) − ∇
Revised: ) − ∇

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ( z (cid:63) )
Revised: ( z (cid:63) )

Original: (cid:13)(cid:13) 

Revised: (cid:13)(cid:13)

Original: + (cid:13)(cid:13) ∇
Revised: + (cid:13)(cid:13) ∇

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ( z
Revised: ( z

Original: [Equation]
Revised: [Equation]

Original: ) − ∇
Revised: ) − ∇

Original: [Equation]
Revised: [Equation]

Original: f ij ( z (cid:63)
Revised: f ij ( z (cid:63)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13) 2 (cid:17) ≤ 8 s 2 L xx np min m (cid:88) i =
Revised: (cid:13)(cid:13) 2 (cid:17) ≤ 8 s 2 L xx np min m (cid:88) i =

Original: V f
Revised: V f

Original: [Equation]
Revised: [Equation]

Original: it (
Revised: it (

Original: [Equation]
Revised: [Equation]

Original: (cid:63)
Revised: (cid:63)

Original: [Equation]
Revised: [Equation]

Original: np min m (cid:88)
Revised: np min m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we bound the last term of (189) as
Revised: we bound the last term of (189) as

Original: [Equation]
Revised: [Equation]

Original: On substituting (200) and (201) in (189), we obtain
Revised: On substituting (200) and (201) in (189), we obtain

Original: [Equation]
Revised: [Equation]

Original: proving Lemma G.5.
Revised: proving Lemma G.5.

Original: We now have the following corollary.
Revised: We now have the following corollary.

Original: Corollary G.6.
Revised: Corollary G.6.

Original: Let s = µnp min 24 L 2 .
Revised: Let s = µnp min 24 L 2 .

Original: Then under the settings of Lemma G.5,
Revised: Then under the settings of Lemma G.5,

Original: [Equation]
Revised: [Equation]

Original: G.2 Proof of Corollary G. 
Revised: G.2 Proof of Corollary G.

Original: From the statement of the corollary, we have s = µnp min 24 L 2 ≤ np min 24 Lκ < np min 4 L ≤ np min 4 L xx .
Revised: From the statement of the corollary, we have s = µnp min 24 L 2 ≤ np min 24 Lκ < np min 4 L ≤ np min 4 L xx .

Original: This implies that
Revised: This implies that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Notice that V f i ,y it ( x (cid:63) , x it ) ≥ 0 .
Revised: Notice that V f i ,y it ( x (cid:63) , x it ) ≥ 0 .

Original: Therefore, (cid:16) 2 s − 8 s 2 L xx np min (cid:17) (cid:80) mi
Revised: Therefore, (cid:16) 2 s − 8 s 2 L xx np min (cid:17) (cid:80) mi

Original: =1 V f i ,y it
Revised: =1 V f i ,y it

Original: ( x (cid:63) , x it ) ≥ 0 .
Revised: ( x (cid:63) , x it ) ≥ 0 .

Original: We also have s ≤ np min 4 L yy because L = max { L xx , L yy , L xy , L yx } .
Revised: We also have s ≤ np min 4 L yy because L = max { L xx , L yy , L xy , L yx } .

Original: Therefore, 8 s 2 L yy np min ≤ 2 s .
Revised: Therefore, 8 s 2 L yy np min ≤ 2 s .

Original: Due to the concavity of f i ( x, y ) in y , V − f i ,x it ( y (cid:63) , y it ) is nonnegative.
Revised: Due to the concavity of f i ( x, y ) in y , V − f i ,x it ( y (cid:63) , y it ) is nonnegative.

Original: Therefore, (cid:16) 2 s −
Revised: Therefore, (cid:16) 2 s −

Original: [Equation]
Revised: [Equation]

Original: By substituting these lower bounds in (185), we get the desired result.
Revised: By substituting these lower bounds in (185), we get the desired result.

Original: Parameters setup
Revised: Parameters setup

Original: Let p min =
Revised: Let p min =

Original: min i,j { p ij } .
Revised: min i,j { p ij } .

Original: We deﬁne the following quantities which are instrumental in simplifying the bounds and in Algorithm 3 implementation.
Revised: We deﬁne the following quantities which are instrumental in simplifying the bounds and in Algorithm 3 implementation.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is worth mentioning that γ x and γ y are well deﬁned for δ =
Revised: It is worth mentioning that γ x and γ y are well deﬁned for δ =

Original: [Equation]
Revised: [Equation]

Original: Lemma G.7.
Revised: Lemma G.7.

Original: Parameters Feasibility
Revised: Parameters Feasibility

Original: The parameters deﬁned in (207) , (208) , (209) and (210) satisfy the followings:
Revised: The parameters deﬁned in (207) , (208) , (209) and (210) satisfy the followings:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: G.3 Proof of Lemma G. 
Revised: G.3 Proof of Lemma G.

Original: In this section, we show that the chosen parameters α x α y , M x , M y , γ x and γ y satisfy the following conditions given in Lemma G.7:
Revised: In this section, we show that the chosen parameters α x α y , M x , M y , γ x and γ y satisfy the following conditions given in Lemma G.7:

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst show that b x ∈ (0 , 1) and b y ∈
Revised: We ﬁrst show that b x ∈ (0 , 1) and b y ∈

Original: From deﬁnition,
Revised: From deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: We now focus on the lower bound on b x and b y .
Revised: We now focus on the lower bound on b x and b y .

Original: [Equation]
Revised: [Equation]

Original: In a similar fashion, we get b y < 1 and b y ≥ np
Revised: In a similar fashion, we get b y < 1 and b y ≥ np

Original: [Equation]
Revised: [Equation]

Original: Feasibility of α x and α y .
Revised: Feasibility of α x and α y .

Original: We have, 0 < b x < 1 .
Revised: We have, 0 < b x < 1 .

Original: Therefore, α x < 11+ δ .
Revised: Therefore, α x < 11+ δ .

Original: Moreover, √ δ 1+ δ ≤ 1 / 2 .
Revised: Moreover, √ δ 1+ δ ≤ 1 / 2 .

Original: Therefore, α x ≤ b x 2 √ δ < b x / √ δ .
Revised: Therefore, α x ≤ b x 2 √ δ < b x / √ δ .

Original: Hence, α x < min (cid:110) b x √ δ , 11+ δ (cid:111) .
Revised: Hence, α x < min (cid:110) b x √ δ , 11+ δ (cid:111) .

Original: Similarly, α y < min (cid:110) b y √ δ , 11+ δ (cid:111) because b y ∈ (0 , 1) .
Revised: Similarly, α y < min (cid:110) b y √ δ , 11+ δ (cid:111) because b y ∈ (0 , 1) .

Original: Feasibility of γ x,k and γ y,k .
Revised: Feasibility of γ x,k and γ y,k .

Original: We consider two cases to verify the feasibility of γ x and γ y .
Revised: We consider two cases to verify the feasibility of γ x and γ y .

Original: Case I: b x ≤ √ δ .
Revised: Case I: b x ≤ √ δ .

Original: This gives γ x = b x 4 √ δ (1+ δ ) λ max ( I − W ) .
Revised: This gives γ x = b x 4 √ δ (1+ δ ) λ max ( I − W ) .

Original: Consider α
Revised: Consider α

Original: [Equation]
Revised: [Equation]

Original: √ δλ
Revised: √ δλ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using (228), we have b x ≤ 1 24 κ 2 x < 0 .
Revised: Using (228), we have b x ≤ 1 24 κ 2 x < 0 .

Original: This allows us to use the inequality 2 x − 2 x 2 ≥ x/ 2 for all 0 ≤ x ≤ 0 .
Revised: This allows us to use the inequality 2 x − 2 x 2 ≥ x/ 2 for all 0 ≤ x ≤ 0 .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: where the second last inequality uses b x ≤ √ δ .
Revised: where the second last inequality uses b x ≤ √ δ .

Original: We know that b y ∈ (0 , 1) .
Revised: We know that b y ∈ (0 , 1) .

Original: Therefore, by following similar steps, the chosen γ y is also feasible.
Revised: Therefore, by following similar steps, the chosen γ y is also feasible.

Original: Case II: b x > √ δ
Revised: Case II: b x > √ δ

Original: This give γ x =
Revised: This give γ x =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: Therefore, γ x < min (cid:110) α x − (1+ δ ) α 2 x √ δλ max ( I − W ) , 2 − 2 √ δα x λ max ( I − W ) (cid:111) .
Revised: Therefore, γ x < min (cid:110) α x − (1+ δ ) α 2 x √ δλ max ( I − W ) , 2 − 2 √ δα x λ max ( I − W ) (cid:111) .

Original: As γ x < 2 − 2 √ δα x λ max ( I − W ) < 2 λ max ( I − W ) .
Revised: As γ x < 2 − 2 √ δα x λ max ( I − W ) < 2 λ max ( I − W ) .

Original: Notice that λ m − 1 ( I − W ) <
Revised: Notice that λ m − 1 ( I − W ) <

Original: λ max ( I − W )
Revised: λ max ( I − W )

Original: Therefore, γ x 2 λ m − 1 ( I − W ) < γ x 2 λ max
Revised: Therefore, γ x 2 λ m − 1 ( I − W ) < γ x 2 λ max

Original: ( I − W ) < 1 .
Revised: ( I − W ) < 1 .

Original: Similarly, γ y 2 λ m − 1 ( I − W ) < 1 .
Revised: Similarly, γ y 2 λ m − 1 ( I − W ) < 1 .

Original: Feasibility of M x and M y .
Revised: Feasibility of M x and M y .

Original: Recall M x = 1 − √ δα x 1 − γx 2 λ max ( I − W ) and M y = 1 − √ δα y 1 − γy 2 λ max
Revised: Recall M x = 1 − √ δα x 1 − γx 2 λ max ( I − W ) and M y = 1 − √ δα y 1 − γy 2 λ max

Original: I − W ) .
Revised: I − W ) .

Original: We have γ x < 2 −√ δα x λ max ( I − W ) γ x λ max ( I − W ) <
Revised: We have γ x < 2 −√ δα x λ max ( I − W ) γ x λ max ( I − W ) <

Original: 1 − √ δα x 1 − γ x λ max ( I − W )
Revised: 1 − √ δα x 1 − γ x λ max ( I − W )

Original: [Equation]
Revised: [Equation]

Original: Moreover, √ δα x
Revised: Moreover, √ δα x

Original: [Equation]
Revised: [Equation]

Original: > 0 .
Revised: > 0 .

Original: Therefore, M x ∈ (0 , 1) .
Revised: Therefore, M x ∈ (0 , 1) .

Original: Similar steps follow to prove the feasibility of M y .
Revised: Similar steps follow to prove the feasibility of M y .

Original: Feasibility of 1 − b x M x and 1 − b y M y .
Revised: Feasibility of 1 − b x M x and 1 − b y M y .

Original: We derive upper bounds on 1 − b x M x and 1 − b y M y to verify the feasibility.
Revised: We derive upper bounds on 1 − b x M x and 1 − b y M y to verify the feasibility.

Original: We divide the derivation into two cases.
Revised: We divide the derivation into two cases.

Original: Case I:
Revised: Case I:

Original: [Equation]
Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: Recall M x :
Revised: Recall M x :

Original: [Equation]
Revised: [Equation]

Original: We know that √ δ b x ≥ 1 .
Revised: We know that √ δ b x ≥ 1 .

Original: Therefore, √ δ (1+ δ ) b x > 1 which in turn implies that
Revised: Therefore, √ δ (1+ δ ) b x > 1 which in turn implies that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By using above relation in (242), we obtain
Revised: By using above relation in (242), we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second last inequality uses √ δ 1+ δ ≤ 12 .
Revised: where the second last inequality uses √ δ 1+ δ ≤ 12 .

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Revised: and,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Case II: b x > √ δ .
Revised: Case II: b x > √ δ .

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: As 8(1 + δ ) − 1 > 8(1 + δ ) − 1 − δ = 7(1 + δ ) .
Revised: As 8(1 + δ ) − 1 > 8(1 + δ ) − 1 − δ = 7(1 + δ ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Notice that above lower bound matches with lower bound in (244).
Revised: Notice that above lower bound matches with lower bound in (244).

Original: Therefore, by following steps similar to Case I, we obtain
Revised: Therefore, by following steps similar to Case I, we obtain

Original: [Equation]
Revised: [Equation]

Original: Feasibility of 1 − γ x 2 λ m − 1 ( I − W ) and 1 − γ y 2 λ m − 1 ( I − W ) .
Revised: Feasibility of 1 − γ x 2 λ m − 1 ( I − W ) and 1 − γ y 2 λ m − 1 ( I − W ) .

Original: If b x ≤ √ δ , then γ x = b x 4 √ δ (1+
Revised: If b x ≤ √ δ , then γ x = b x 4 √ δ (1+

Original: [Equation]
Revised: [Equation]

Original: ) λ max ( I − W ) .
Revised: ) λ max ( I − W ) .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If b x > √ δ , then γ x = 

Revised: If b x > √ δ , then γ x =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now have a result on recursive relationship on E (cid:104)
Revised: We now have a result on recursive relationship on E (cid:104)

Original: ˜Φ t (cid:105) .
Revised: ˜Φ t (cid:105) .

Original: Lemma G.8.
Revised: Lemma G.8.

Original: Let { x t } t , { y t } t be the sequences generated by Algorithm 3.
Revised: Let { x t } t , { y t } t be the sequences generated by Algorithm 3.

Original: Suppose Assumptions 3.1-3.5 and Assumptions G.1-G.4 hold.
Revised: Suppose Assumptions 3.1-3.5 and Assumptions G.1-G.4 hold.

Original: Then for every t ≥ 1 :
Revised: Then for every t ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: where ρ is deﬁned in equation (214) .
Revised: where ρ is deﬁned in equation (214) .

Original: G.4 Proof of Lemma G. 
Revised: G.4 Proof of Lemma G.

Original: Iterates { x t , y t } of Algorithm 3 are obtained by calling Algorithm 1 at iterate t − 1 .
Revised: Iterates { x t , y t } of Algorithm 3 are obtained by calling Algorithm 1 at iterate t − 1 .

Original: Therefore,
Revised: Therefore,

Original: Lemma D.1 and Lemma D.2 also holds for Algorithm 3.
Revised: Lemma D.1 and Lemma D.2 also holds for Algorithm 3.

Original: Adding inequalities (53) and (103) (Lemma
Revised: Adding inequalities (53) and (103) (Lemma

Original: D.1 and Lemma D.2), we have
Revised: D.1 and Lemma D.2), we have

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of Φ t
Revised: By the deﬁnition of Φ t

Original: [Equation]
Revised: [Equation]

Original: the above inequality can be rewritten as
Revised: the above inequality can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Taking conditional expectation on stochastic gradient at t -th step on both sides of above inequality and applying Tower property
Revised: Taking conditional expectation on stochastic gradient at t -th step on both sides of above inequality and applying Tower property

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: t +1 ]
Revised: t +1 ]

Original: [Equation]
Revised: [Equation]

Original: + E (cid:107)
Revised: + E (cid:107)

Original: y t − 1 y
Revised: y t − 1 y

Original: (cid:63) + s G yt − s ∇ y F ( 1 z (cid:63) ) (cid:107) 

Revised: (cid:63) + s G yt − s ∇ y F ( 1 z (cid:63) ) (cid:107)

Original: + 2 s 2 γ y (cid:16) 1 − γ y 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D yt − D (cid:63)y (cid:13)(cid:13) 2 ( I − W ) † + √ δ
Revised: + 2 s 2 γ y (cid:16) 1 − γ y 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D yt − D (cid:63)y (cid:13)(cid:13) 2 ( I − W ) † + √ δ

Original: (1 − α y )
Revised: (1 − α y )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: H yt − H (cid:63)y (cid:13)(cid:13)≤ (cid:32) 1 − µ x s + 4 s 2 L 2 yx np min (cid:33) (cid:107) x
Revised: H yt − H (cid:63)y (cid:13)(cid:13)≤ (cid:32) 1 − µ x s + 4 s 2 L 2 yx np min (cid:33) (cid:107) x

Original: t − 1 x (cid:63) (cid:107) 2 + (cid:32) 1 − sµ y
Revised: t − 1 x (cid:63) (cid:107) 2 + (cid:32) 1 − sµ y

Original: + 4 s 2 L 2 xy np min (cid:33) (cid:107)
Revised: + 4 s 2 L 2 xy np min (cid:33) (cid:107)

Original: y t − 1 y
Revised: y t − 1 y

Original: (cid:63) (
Revised: (cid:63) (

Original: cid:107)+ 4 s 2 ( L 2 xx + L 2 yx ) np min (cid:107) ˜ x
Revised: cid:107)+ 4 s 2 ( L 2 xx + L 2 yx ) np min (cid:107) ˜ x

Original: t − 1 x (cid:63) (cid:107) 2 + 4 s 

Revised: t − 1 x (cid:63) (cid:107) 2 + 4 s

Original: ( L 2 yy + L 2 xy )
Revised: ( L 2 yy + L 2 xy )

Original: np min (cid:107) ˜
Revised: np min (cid:107) ˜

Original: y t − 1 y (cid:63) (cid:107)+ 2 sγ x (cid:16) 1 − γ x 2 λ m − 1 ( I − W ) (cid:17) (cid:107) D xt − D
Revised: y t − 1 y (cid:63) (cid:107)+ 2 sγ x (cid:16) 1 − γ x 2 λ m − 1 ( I − W ) (cid:17) (cid:107) D xt − D

Original: cid:63)x (cid:107) 

Revised: cid:63)x (cid:107)

Original: ( I − W ) † + 2 s 2 γ y (cid:16) 1 − γ y 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D yt − D (cid:63)y (cid:13)(cid:13) 2 ( I − W ) † + √ δ
Revised: ( I − W ) † + 2 s 2 γ y (cid:16) 1 − γ y 2 λ m − 1 ( I − W ) (cid:17) (cid:13)(cid:13) D yt − D (cid:63)y (cid:13)(cid:13) 2 ( I − W ) † + √ δ

Original: (1 − α x ) (cid:107) H xt − H (cid:63)x (cid:107) 2 + √ δ (1 − α y )
Revised: (1 − α x ) (cid:107) H xt − H (cid:63)x (cid:107) 2 + √ δ (1 − α y )

Original: [Equation]
Revised: [Equation]

Original: The last step holds due to Corollary G.6.
Revised: The last step holds due to Corollary G.6.

Original: From SVRG oracle, we have
Revised: From SVRG oracle, we have

Original: [Equation]
Revised: [Equation]

Original: , and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using above equality and (260)
Revised: Using above equality and (260)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: We have ˜ c x = 8 s 2 ( L 2 xx + L 2 yx ) np min p .
Revised: We have ˜ c x = 8 s 2 ( L 2 xx + L 2 yx ) np min p .

Original: The coefﬁcient of (cid:107) ˜ x t − 1 x (cid:63)
Revised: The coefﬁcient of (cid:107) ˜ x t − 1 x (cid:63)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the coefﬁcient of (cid:107) ˜ y t − 1 y (cid:63) (cid:107) 2 in (262) is
Revised: and the coefﬁcient of (cid:107) ˜ y t − 1 y (cid:63) (cid:107) 2 in (262) is

Original: [Equation]
Revised: [Equation]

Original: Substituting the above simpliﬁed coefﬁcients into (262), we see that
Revised: Substituting the above simpliﬁed coefﬁcients into (262), we see that

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: By taking total expectation on both sides and using the deﬁnition of b x and b y
Revised: By taking total expectation on both sides and using the deﬁnition of b x and b y

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: 267) where
Revised: 267) where

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of ˜Φ t =
Revised: By the deﬁnition of ˜Φ t =

Original: Φ t + ˜ c
Revised: Φ t + ˜ c

Original: x (cid:107) ˜ x
Revised: x (cid:107) ˜ x

Original: t − 1 x (cid:63) (cid:107) 

Revised: t − 1 x (cid:63) (cid:107)

Original: + ˜ c y (cid:107) ˜
Revised: + ˜ c y (cid:107) ˜

Original: y t − 1 y (cid:63) (cid:107) 2 , (267) reduces to
Revised: y t − 1 y (cid:63) (cid:107) 2 , (267) reduces to

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:105) .
Revised: (cid:105) .

Original: Therefore, E (cid:104)
Revised: Therefore, E (cid:104)

Original: ˜Φ t +1 (cid:105) ≤ ρ t +1 ˜Φ 0 .
Revised: ˜Φ t +1 (cid:105) ≤ ρ t +1 ˜Φ 0 .

Original: H Proof of Theorem 5. 
Revised: H Proof of Theorem 5.

Original: This proof is based on several intermediate results proved in Appendices C-E.
Revised: This proof is based on several intermediate results proved in Appendices C-E.

Original: Hence it would be useful to refer to those results in order to appreciate the proof of Theorem 5.1.
Revised: Hence it would be useful to refer to those results in order to appreciate the proof of Theorem 5.1.

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: where last inequality follows from Lemma G.8 and M :
Revised: where last inequality follows from Lemma G.8 and M :

Original: = min { M x , M y } .
Revised: = min { M x , M y } .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: (272) for T ( (cid:15) )
Revised: (272) for T ( (cid:15) )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Recall
Revised: Recall

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using Lemma G.7, ρ can be upper bounded as ρ ≤ max (cid:40)
Revised: Using Lemma G.7, ρ can be upper bounded as ρ ≤ max (cid:40)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using (231) and (232)
Revised: Using (231) and (232)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: =: 1 − ˜ C.
Revised: =: 1 − ˜ C.

Original: [Equation]
Revised: [Equation]

Original: By taking log on both sides
Revised: By taking log on both sides

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where the fourth inequality uses the fact that (1 / − log(1 − x )) ≤ 5 /x for all 0 < x < 1 .
Revised: where the fourth inequality uses the fact that (1 / − log(1 − x )) ≤ 5 /x for all 0 < x < 1 .

Original: Using
Revised: Using

Original: Lemma G.
Revised: Lemma G.

Original: [Equation]
Revised: [Equation]

Original: we have M x ≥ 1 − 4 b x 7 .
Revised: we have M x ≥ 1 − 4 b x 7 .

Original: Therefore, M x > 1 − 47 = 

Revised: Therefore, M x > 1 − 47 =

Original: because 0 < b x < 1 .
Revised: because 0 < b x < 1 .

Original: Moreover,
Revised: Moreover,

Original: M y > 37 as 0 <
Revised: M y > 37 as 0 <

Original: b y < 1 .
Revised: b y < 1 .

Original: Therefore, log (cid:16) ˜Φ 0 M(cid:15) (cid:17) ≤ log (cid:16) 7˜Φ 0 3 (cid:15) (cid:17) .
Revised: Therefore, log (cid:16) ˜Φ 0 M(cid:15) (cid:17) ≤ log (cid:16) 7˜Φ 0 3 (cid:15) (cid:17) .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: H.1 Discussion on the analysis techniques
Revised: H.1 Discussion on the analysis techniques

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we discuss and compare the analysis techniques of our work with those in existing works.
Revised: we discuss and compare the analysis techniques of our work with those in existing works.

Original: In [20] a convex composite minimization problem is studied and inexact PDHG method is applied to its saddle point formulation.
Revised: In [20] a convex composite minimization problem is studied and inexact PDHG method is applied to its saddle point formulation.

Original: In this work, we study a different problem (1) where a smooth function depends jointly on primal and dual variables.
Revised: In this work, we study a different problem (1) where a smooth function depends jointly on primal and dual variables.

Original: We prove that it is equivalent to study unconstrained saddle point problem (3) to get the solution of (1).
Revised: We prove that it is equivalent to study unconstrained saddle point problem (3) to get the solution of (1).

Original: However, [20] uses a well known equivalence between a convex minimization problem and its Lagrangian formulation [19].
Revised: However, [20] uses a well known equivalence between a convex minimization problem and its Lagrangian formulation [19].

Original: We deﬁne additional quantities D (cid:63)y , H (cid:63)y and Bregman distance functions V f
Revised: We deﬁne additional quantities D (cid:63)y , H (cid:63)y and Bregman distance functions V f

Original: i ,y ( x 1 , x 2 ) , V − f i ,x ( y 1 , y 2 ) in
Revised: i ,y ( x 1 , x 2 ) , V − f i ,x ( y 1 , y 2 ) in

Original: Appendix C to obtain appropriate bounds.
Revised: Appendix C to obtain appropriate bounds.

Original: Algorithm 2 analysis: In contrast to [20], we get complicated upper bounds depending on primal and dual iterates in Lemma E.3 which yields different set of parameters.
Revised: Algorithm 2 analysis: In contrast to [20], we get complicated upper bounds depending on primal and dual iterates in Lemma E.3 which yields different set of parameters.

Original: We prove the feasibility of these parameters and derive useful lower and upper bounds in Appendix E.4.
Revised: We prove the feasibility of these parameters and derive useful lower and upper bounds in Appendix E.4.

Original: [20] uses diminishing step size and an induction approach to prove the convergence to exact solution.
Revised: [20] uses diminishing step size and an induction approach to prove the convergence to exact solution.

Original: However, we use a different method summarized below to derive the convergence rate of Algorithm (2).
Revised: However, we use a different method summarized below to derive the convergence rate of Algorithm (2).

Original: We ﬁrst derive a relation which connects iterate t information with the restart iterates (see Lemma E.6).
Revised: We ﬁrst derive a relation which connects iterate t information with the restart iterates (see Lemma E.6).

Original: Then by appropriately choosing the restart iterates x k, 0 , y k, 0 , D xk, 0 , D yk, 0 , H xk, 0 and H yk, 0 and inner iterates t k , we get the complexity result in Appendix F.
Revised: Then by appropriately choosing the restart iterates x k, 0 , y k, 0 , D xk, 0 , D yk, 0 , H xk, 0 and H yk, 0 and inner iterates t k , we get the complexity result in Appendix F.

Original: It is worth noting that proof techniques of Lemma E. and Theorem 4.1 are different from those in [20] due to different algorithm structure involving a restart scheme, complicated bounds, and different sets of parameters.
Revised: It is worth noting that proof techniques of Lemma E. and Theorem 4.1 are different from those in [20] due to different algorithm structure involving a restart scheme, complicated bounds, and different sets of parameters.

Original: Algorithm 3 analysis:
Revised: Algorithm 3 analysis:

Original: Using smoothness, strong convexity strong concavity assumptions, and deﬁnitions of
Revised: Using smoothness, strong convexity strong concavity assumptions, and deﬁnitions of