Original: Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox
Revised: Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox

Original: Abstract
Revised: Abstract

Original: Inspired by a recent breakthrough of Mishchenko et al.
Revised: Inspired by a recent breakthrough of Mishchenko et al.

Original: [2022], who for the first time showed that local gradient steps can lead to provable communication acceleration, we propose an alternative algorithm which obtains the same communication acceleration as their method (ProxSkip).
Revised: [2022], who for the first time showed that local gradient steps can lead to provable communication acceleration, we propose an alternative algorithm which obtains the same communication acceleration as their method (ProxSkip).

Original: Our approach is very different, however: it is based on the celebrated method of Chambolle and Pock [2011], with several nontrivial modifications: i) we allow for an inexact computation of the prox operator of a certain smooth strongly convex function via a suitable gradient-based method (e.g., GD, Fast GD or FSFOM), ii) we perform a careful modification of the dual update step in order to retain linear convergence.
Revised: Our approach is very different, however: it is based on the celebrated method of Chambolle and Pock [2011], with several nontrivial modifications: i) we allow for an inexact computation of the prox operator of a certain smooth strongly convex function via a suitable gradient-based method (e.g., GD, Fast GD or FSFOM), ii) we perform a careful modification of the dual update step in order to retain linear convergence.

Original: Our general results offer the new state-of-the-art rates for the class of strongly convex-concave saddle-point problems with bilinear coupling characterized by the absence of smoothness in the dual function.
Revised: Our general results offer the new state-of-the-art rates for the class of strongly convex-concave saddle-point problems with bilinear coupling characterized by the absence of smoothness in the dual function.

Original: When applied to federated learning, we obtain a theoretically better alternative to ProxSkip: our method requires fewer local steps ( O ( κ 1 / 3 ) or
Revised: When applied to federated learning, we obtain a theoretically better alternative to ProxSkip: our method requires fewer local steps ( O ( κ 1 / 3 ) or

Original: O ( κ 1 / 4 ) , compared to O ( κ 1 / 2 ) of ProxSkip), and performs a deterministic number of local steps instead.
Revised: O ( κ 1 / 4 ) , compared to O ( κ 1 / 2 ) of ProxSkip), and performs a deterministic number of local steps instead.

Original: Like ProxSkip, our method can be applied to optimization over a connected network, and we obtain theoretical improvements here as well.
Revised: Like ProxSkip, our method can be applied to optimization over a connected network, and we obtain theoretical improvements here as well.

Original:  Introduction
Revised:  Introduction

Original: Communication efficiency of distributed stochastic gradient descent (SGD) can be improved, often dramatically, via a simple trick: instead of synchronizing the parameters across the parallel workers after every SGD step, let the workers perform multiple optimization steps using their local loss and data only before synchronizing.
Revised: Communication efficiency of distributed stochastic gradient descent (SGD) can be improved, often dramatically, via a simple trick: instead of synchronizing the parameters across the parallel workers after every SGD step, let the workers perform multiple optimization steps using their local loss and data only before synchronizing.

Original: This trick dates back at least to two or three decades ago [Mangasarian, 1995] and may be much older.
Revised: This trick dates back at least to two or three decades ago [Mangasarian, 1995] and may be much older.

Original: Due to its simplicity, it has been repeatedly rediscovered [Povey et al., 2014, Moritz et al., 2016, Ma et al., 2017].
Revised: Due to its simplicity, it has been repeatedly rediscovered [Povey et al., 2014, Moritz et al., 2016, Ma et al., 2017].

Original: It is the basis of the famous federated averaging (FedAvg) algorithm of McMahan et al.
Revised: It is the basis of the famous federated averaging (FedAvg) algorithm of McMahan et al.

Original: [2016, 2017], which is the workhorse of federated learning [Koneˇcný et al., 2016b,a]; see also the recent surveys on federated learning [Li et al., 2020a, Kairouz et al., 2019] and federated optimization [Wang et al., 2021b].
Revised: [2016, 2017], which is the workhorse of federated learning [Koneˇcný et al., 2016b,a]; see also the recent surveys on federated learning [Li et al., 2020a, Kairouz et al., 2019] and federated optimization [Wang et al., 2021b].

Original: 1.1 Towards provable communication acceleration via delayed parameter synchronization
Revised: 1.1 Towards provable communication acceleration via delayed parameter synchronization

Original: Until recently, this simple trick resisted all attempts at an appropriate theoretical justification.
Revised: Until recently, this simple trick resisted all attempts at an appropriate theoretical justification.

Original: Through collective effort of the federated learning community, the bounds of various local SGD methods were progressively getting better [Haddadpour and Mahdavi, 2019, Li et al., 2019a,b, Khaled et al., 2019a,b, 2020, Li et al., 2020b, Woodworth et al., 2020, Stich, 2020, Gorbunov et al., 2020a, Malinovsky et al., 2020, Pathak and Wainwright, 2020, Karimireddy et al., 2020, Malinovsky et al., 2021, Mishchenko et al., 2021a, Wang and Joshi, 2021, Horváth et al., 2022], and the assumptions required to achieve them weaker.
Revised: Through collective effort of the federated learning community, the bounds of various local SGD methods were progressively getting better [Haddadpour and Mahdavi, 2019, Li et al., 2019a,b, Khaled et al., 2019a,b, 2020, Li et al., 2020b, Woodworth et al., 2020, Stich, 2020, Gorbunov et al., 2020a, Malinovsky et al., 2020, Pathak and Wainwright, 2020, Karimireddy et al., 2020, Malinovsky et al., 2021, Mishchenko et al., 2021a, Wang and Joshi, 2021, Horváth et al., 2022], and the assumptions required to achieve them weaker.

Original: A brief overview of the progress is provided in [Mishchenko et al., 2022].
Revised: A brief overview of the progress is provided in [Mishchenko et al., 2022].

Original: However, all known theoretical rates are worse than the rate of gradient descent, which synchronizes after every gradient step.
Revised: However, all known theoretical rates are worse than the rate of gradient descent, which synchronizes after every gradient step.

Original: In a recent breakthrough, Mishchenko et al.
Revised: In a recent breakthrough, Mishchenko et al.

Original: [2022] developed a novel local
Revised: [2022] developed a novel local

Original: SGD method, called ProxSkip, which performs a random number of local gradient (or stochastic gradient) steps before synchronization, and proved that it enjoys strong communication acceleration properties.
Revised: SGD method, called ProxSkip, which performs a random number of local gradient (or stochastic gradient) steps before synchronization, and proved that it enjoys strong communication acceleration properties.

Original: In particular, while the method needs O ( κ log 1 ε ) iterations, only O ( √ κ log 1 ε ) of them involve communication.
Revised: In particular, while the method needs O ( κ log 1 ε ) iterations, only O ( √ κ log 1 ε ) of them involve communication.

Original: 1.2 Problem formulation
Revised: 1.2 Problem formulation

Original: In this paper, we consider the composite optimization problem
Revised: In this paper, we consider the composite optimization problem

Original: [Equation]
Revised: [Equation]

Original: where G : R d x → R is a smooth and strongly convex function, F : R d y → R ∪ { + ∞} is a proper, closed and convex function, and K : R d x → R d y is a linear map.
Revised: where G : R d x → R is a smooth and strongly convex function, F : R d y → R ∪ { + ∞} is a proper, closed and convex function, and K : R d x → R d y is a linear map.

Original: Let us define
Revised: Let us define

Original: [Equation]
Revised: [Equation]

Original:  where ∥ · ∥ refers to the standard Euclidean norm.
Revised: (2) where ∥ · ∥ refers to the standard Euclidean norm.

Original: Note that L 2 xy ≥ λ max ( KK ⊤ ) =
Revised: Note that L 2 xy ≥ λ max ( KK ⊤ ) =

Original: λ max ( K ⊤ K ) .
Revised: λ max ( K ⊤ K ) .

Original: It will be useful to formalize our assumptions at this point as we will refer to the various constants involved in them throughout the text.
Revised: It will be useful to formalize our assumptions at this point as we will refer to the various constants involved in them throughout the text.

Original: Assumption 1.
Revised: Assumption 1.

Revised: Function G : R

Original: Function G : R d x → R is µ x -strongly convex, i.e.,
Revised: [Equation] → R is [Equation] x -strongly convex, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: The function G : R d x → R is L x -smooth, i.e.
Revised: The function G : R d x → R is L x -smooth, i.e.

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.
Revised: Assumption 3.

Original: Function F : R d y → R ∪ { + ∞} is proper, closed and convex.
Revised: Function F : R d y → R ∪ { + ∞} is proper, closed and convex.

Original: Assumption 4 (See Kovalev et al.
Revised: Assumption 4 (See Kovalev et al.

Original: [2021a]) .
Revised: [2021a]) .

Original: There exists a constant µ xy > 0 such that
Revised: There exists a constant µ xy > 0 such that

Original: [Equation]
Revised: [Equation]

Original: , if ∂F⋆
Revised: ∂F

Original: [Equation]
Revised: [Equation]

Original: y) ∈ range K for all y ∈ R d y , λ min ( KK
Revised: y

Original: [Equation]
Revised: [Equation]

Original: ) , otherwise.

Original: 1.3 ProxSkip
Revised: 1.3 ProxSkip

Original: The most general version of the ProxSkip method 5 of Mishchenko et al.
Revised: The most general version of the ProxSkip method 5 of Mishchenko et al.

Original: [2022] was designed to solve problems of the form (1).
Revised: [2022] was designed to solve problems of the form (1).

Original: In each iteration, ProxSkip evaluates the gradient of G and then flips a biased coin: with probability p , it additionally evaluates the proximity operator of F ⋆ , and performs a matrix-vector multiplication involving K .
Revised: In each iteration, ProxSkip evaluates the gradient of G and then flips a biased coin: with probability p , it additionally evaluates the proximity operator of F ⋆ , and performs a matrix-vector multiplication involving K .

Original: The method becomes relevant to the standard optimization formulation of federated learning (FL), i.e., to the finite-sum optimization problem
Revised: The method becomes relevant to the standard optimization formulation of federated learning (FL), i.e., to the finite-sum optimization problem

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: through its application to its consensus reformulation
Revised: through its application to its consensus reformulation

Original: [Equation]
Revised: [Equation]

Original: (6) where d x def = nd , x def = ( x 1 , . . 
Revised: [Table] where d x def = nd [Table] x def = ( x 1 , . . .

Original: , x n ) ∈ R nd , G ( x ) def = (cid:80) ni
Revised: , x n ) ∈ R nd , G ( x ) def = (cid:80) ni

Original: =

Revised: =

Original: f i ( x i ) , and F def = ψ is the indicator function of the constraint x 1 = · · · = x n , i.e., ψ (
Revised: f i ( x i ) , and F def = ψ is the indicator function of the constraint x 1 = · · · = x n , i

Original: [Equation]
Revised: [Table]

Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: The evaluation of the proximity operator of F is equivalent to averaging of the vectors x 1 , . . .
Revised: The evaluation of the proximity operator of F is equivalent to averaging of the vectors x 1 , . . .

Original: , x n , which necessitates communication.
Revised: , x n , which necessitates communication.

Original: Therefore, if p is small, ProxSkip communicates very rarely.
Revised: Therefore, if p is small, ProxSkip communicates very rarely.

Original: Since G is block separable, the gradient steps involving G , taken in between two communications, correspond to gradient steps with respect to the local loss functions { f i } taken by the clients.
Revised: Since G is block separable, the gradient steps involving G , taken in between two communications, correspond to gradient steps with respect to the local loss functions { f i } taken by the clients.

Original: See [Mishchenko et al., 2022] for the details; we will elaborate on this as well in Section 6.
Revised: See [Mishchenko et al., 2022] for the details; we will elaborate on this as well in Section 6.

Original: ProxSkip inexactly solves problem (1) in O ( κ log 1 / ε ) iterations, out of which only O (cid:0) √ κχ log 1 ε (cid:1) involve communication, where χ is a condition number measuring the connectivity of the graph (the standard setup in FL corresponds to a fully connected graph, in which case χ = 1 ; see (26) for definition).
Revised: ProxSkip inexactly solves problem (1) in O ( κ log 1 / ε ) iterations, out of which only O (cid:0) √ κχ log 1 ε (cid:1) involve communication, where χ is a condition number measuring the connectivity of the graph (the standard setup in FL corresponds to a fully connected graph, in which case χ = 1 ; see (26) for definition).

Original:  Summary of Contributions
Revised:  Summary of Contributions

Original: Inspired by the results of Mishchenko et al.
Revised: Inspired by the results of Mishchenko et al.

Original: [2022], we propose an alternative and substantially different algorithm which obtains the same guarantees for the number of prox evaluations (wrt F ) as ProxSkip, but has better guarantees for the number  gradient steps (wrt G ) in between the prox evaluations.
Revised: [2022], we propose an alternative and substantially different algorithm which obtains the same guarantees for the number of prox evaluations (wrt F ) as ProxSkip, but has better guarantees for the number of gradient steps (wrt G ) in between the prox evaluations.

Original: Below we summarize the main contributions:
Revised: Below we summarize the main contributions:

Original: Saddle-point formulation.
Revised: Saddle-point formulation.

Original: Unlike Mishchenko et al.
Revised: Unlike Mishchenko et al.

Original: [2022], we consider the saddle-point reformulation of (1)
Revised: [2022], we consider the saddle-point reformulation of (1)

Original: [Equation]
Revised: [Equation]

Original: where F ⋆ ( y ) def =
Revised: where F ⋆ ( y ) def =

Original: sup y ′ ∈ R dy {⟨ y, y ′ ⟩ − F ( y ′ ) } is the convex conjugate of F .
Revised: sup y ′ ∈ R dy {⟨ y, y ′ ⟩ − F ( y ′ ) } is the convex conjugate of F .

Original: Since F is proper, closed and convex, so is F ⋆ .
Revised: Since F is proper, closed and convex, so is F ⋆ .

Original: We assume throughout that (7) is solvable, i.e., there exists at least one solution ( x ⋆ , y ⋆ ) .
Revised: We assume throughout that (7) is solvable, i.e., there exists at least one solution ( x ⋆ , y ⋆ ) .

Original: Such a solution then satisfies the first-oder optimality conditions
Revised: Such a solution then satisfies the first-oder optimality conditions

Original: [Equation]
Revised: [Equation]

Original: where ∂ denotes the subdifferential.
Revised: where ∂ denotes the subdifferential.

Original: By working with this reformulation, we can tap into the rich and powerful philosophical and technical toolbox offered by proximal-point theory, fixed point theory, and primal-dual methods, which facilitates the algorithm development and analysis.
Revised: By working with this reformulation, we can tap into the rich and powerful philosophical and technical toolbox offered by proximal-point theory, fixed point theory, and primal-dual methods, which facilitates the algorithm development and analysis.

Original: This ultimately enables us to shed new light on the nature of local gradient-type steps as inexact computations of the prox operator of G in a new appropriately designed Accelerated Primal-Dual Algorithm (APDA; see
Revised: This ultimately enables us to shed new light on the nature of local gradient-type steps as inexact computations of the prox operator of G in a new appropriately designed Accelerated Primal-Dual Algorithm (APDA; see

Original: Algorithm 1).
Revised: Algorithm 1).

Original: Modifications of Chambolle-Pock Our Algorithms 1 and 2 are inspired by the celebrated
Revised: Modifications of Chambolle-Pock Our Algorithms 1 and 2 are inspired by the celebrated

Original: Chambolle-Pock method [Chambolle and Pock, 2011], but with several important and nontrivial modifications.
Revised: Chambolle-Pock method [Chambolle and Pock, 2011], but with several important and nontrivial modifications.

Original: While Chambolle-Pock achieves linear convergence when both G and F ⋆ are strongly convex, F ⋆ is merely convex in our setting.
Revised: While Chambolle-Pock achieves linear convergence when both G and F ⋆ are strongly convex, F ⋆ is merely convex in our setting.

Original: Our modifications are: i) Inspired by the ideas of Kovalev et al.
Revised: Our modifications are: i) Inspired by the ideas of Kovalev et al.

Original: [2021a], we perform a careful modification of the dual update step (update of y ) in order to retain linear convergence despite lack of strong convexity in F ⋆ .
Revised: [2021a], we perform a careful modification of the dual update step (update of y ) in order to retain linear convergence despite lack of strong convexity in F ⋆ .

Original: On the other hand, in contrast to the method of Kovalev et al.
Revised: On the other hand, in contrast to the method of Kovalev et al.

Original: [2021a], we do not assume F ⋆ to be smooth.
Revised: [2021a], we do not assume F ⋆ to be smooth.

Original: This modification leads to a new method, which we call APDA (Algorithm 1).
Revised: This modification leads to a new method, which we call APDA (Algorithm 1).

Original: APDA relies on the evaluation of the prox operators of both G and F ⋆ .
Revised: APDA relies on the evaluation of the prox operators of both G and F ⋆ .

Original: ii)
Revised: ii)

Original: Next, we remove the reliance on the prox operator of G , and instead allow for its inexact evaluation via a suitable user-defined gradient-based method, which we call M (see (2) and Lemma 1).
Revised: Next, we remove the reliance on the prox operator of G , and instead allow for its inexact evaluation via a suitable user-defined gradient-based method, which we call M (see (2) and Lemma 1).

Original: We call the resulting method “APDA with Inexact Prox”
Revised: We call the resulting method “APDA with Inexact Prox”

Original: (Algorithm 2).
Revised: (Algorithm 2).

Original: The choice of method M will have a strong impact on the number of inexact/local steps, and this is one of the places in which we improve upon the results of Mishchenko et al.
Revised: The choice of method M will have a strong impact on the number of inexact/local steps, and this is one of the places in which we improve upon the results of Mishchenko et al.

Original: General theory Our general complexity results for Algorithms 1 and 2, covered in Theorems 7 and 2, respectively, contrasted with the key baselines, are summarized in Table 1).
Revised: General theory Our general complexity results for Algorithms 1 and 2, covered in Theorems 7 and 2, respectively, contrasted with the key baselines, are summarized in Table 1).

Original: While the method of [Chambolle and Pock, 2011] needs F ⋆ to be strongly convex to obtain a linear rate, we only need convexity.
Revised: While the method of [Chambolle and Pock, 2011] needs F ⋆ to be strongly convex to obtain a linear rate, we only need convexity.

Original: While the AltGDA [Zhang et al., 2022] and APDG
Revised: While the AltGDA [Zhang et al., 2022] and APDG

Original: [Kovalev et al., 2021a]
Revised: [Kovalev et al., 2021a]

Original: methods enjoy linear rates without strong convexity of F ⋆ , both require F ⋆ to be L y -smooth.
Revised: methods enjoy linear rates without strong convexity of F ⋆ , both require F ⋆ to be L y -smooth.

Original: In contrast, we do not need this assumption (i.e., we allow L y = ∞ ).
Revised: In contrast, we do not need this assumption (i.e., we allow L y = ∞ ).

Original: Our methods are the first to obtain linear convergence rates in the regime when G is L x -smooth and µ x -strongly-convex, and F ⋆ is merely proper, closed and convex, without requiring it to be L y -smooth, nor µ y -strongly-convex.
Revised: Our methods are the first to obtain linear convergence rates in the regime when G is L x -smooth and µ x -strongly-convex, and F ⋆ is merely proper, closed and convex, without requiring it to be L y -smooth, nor µ y -strongly-convex.

Original: This is important in some applications.
Revised: This is important in some applications.

Original: Our two methods offer two alternative ways of dealing with this regime: while APDA (Algorithm 1) relies on the evaluation of the prox of G , APDA with Inexact Prox
Revised: Our two methods offer two alternative ways of dealing with this regime: while APDA (Algorithm 1) relies on the evaluation of the prox of G , APDA with Inexact Prox

Original: (Algorithm 2) does not.
Revised: (Algorithm 2) does not.

Original: As we shall see, the latter method has an important application in federated learning.
Revised: As we shall see, the latter method has an important application in federated learning.

Original: Federated learning and a third method When applied to the distributed/federated problem (5) (see Section 6), APDA with Inexact Prox
Revised: Federated learning and a third method When applied to the distributed/federated problem (5) (see Section 6), APDA with Inexact Prox

Original: (Algorithm 2) turns out to be a theoretically better alternative to ProxSkip [Mishchenko et al., 2022].
Revised: (Algorithm 2) turns out to be a theoretically better alternative to ProxSkip [Mishchenko et al., 2022].

Original: In the centralized case, our method requires the same optimal number of communication rounds ( (cid:101) O ( √ κ ) , where κ = L x / µ x ) as ProxSkip, but requires fewer local gradient-type steps ( O ( κ 1 / 3 ) or O ( κ 1 / 4 ) , compared to O ( κ 1 / 2 ) of ProxSkip, depending on the choice of the inner method M ).
Revised: In the centralized case, our method requires the same optimal number of communication rounds ( (cid:101) O ( √ κ ) , where κ = L x / µ x ) as ProxSkip, but requires fewer local gradient-type steps ( O ( κ 1 / 3 ) or O ( κ 1 / 4 ) , compared to O ( κ 1 / 2 ) of ProxSkip, depending on the choice of the inner method M ).

Original: Like ProxSkip, our method can be applied to optimization over a connected network, and we obtain theoretical improvements in this decentralized scenario as well.
Revised: Like ProxSkip, our method can be applied to optimization over a connected network, and we obtain theoretical improvements in this decentralized scenario as well.

Original: However, in the decentralized regime, neither ProxSkip nor Algorithm 2 obtain the optimal bound for the number of communication rounds.
Revised: However, in the decentralized regime, neither ProxSkip nor Algorithm 2 obtain the optimal bound for the number of communication rounds.

Original: For this reason, we propose a third method (Algorithm 5) which employs an accelerated gossip routine to remedy this situation.
Revised: For this reason, we propose a third method (Algorithm 5) which employs an accelerated gossip routine to remedy this situation.

Original: It is also notable that while
Revised: It is also notable that while

Original: ProxSkip uses a random number of local steps, all our methods perform a deterministic number of local steps.
Revised: ProxSkip uses a random number of local steps, all our methods perform a deterministic number of local steps.

Original: Our complexity results are summarized in Table 2.[Table]
Revised: Our complexity results are summarized in Table 2.

Original:  From Proximal Point Method to Chambolle-Pock
Revised:  From Proximal Point Method to Chambolle-Pock

Original: In this section, we briefly motivate the development of the celebrated Chambolle-Pock method which acts as a starting point of our algorithm design. 
Revised: In this section, we briefly motivate the development of the celebrated Chambolle-Pock method which acts as a starting point of our algorithm design. [Table] 4 √ κ (cid:1) (1) Thisistrueonlywhen [Table]

Revised: ≤ χ .

Original:  3.1 Proximal-Point Method for finding zeros of monotone operators
Revised: (1) GD=GradientDescent;FGD=FastGradientDescent(i.e. ,Nesterov’sacceleratedGD);FSFOM= afixed-stepfirst-ordermethodfrom[KimandFessler,2021]. 3.1 Proximal-Point Method for finding zeros of monotone operators

Original: Our starting point is the general problem of finding a zero of an (set-valued) operator A : H → 2 H , where H is a Hilbert space, i.e., find z ∈ H such that
Revised: Our starting point is the general problem of finding a zero of an (set-valued) operator A : H → 2 H , where H is a Hilbert space, i.e., find z ∈ H such that

Original: [Equation]
Revised: [Equation]

Original: If A is maximally monotone, its resolvent (Id + ηA ) − 1 is single valued, nonexpansive, and has full domain.
Revised: If A is maximally monotone, its resolvent (Id + ηA ) − 1 is single valued, nonexpansive, and has full domain.

Original: Moreover, 0 ∈ A ( z ) iff z
Revised: Moreover, 0 ∈ A ( z ) iff z

Original: =
Revised: =

Original: (Id + ηA ) − 1 ( z ) .
Revised: (Id + ηA ) − 1 ( z ) .

Original: The corresponding fixed point iteration, i.e., z k +1 =
Revised: The corresponding fixed point iteration, i.e., z k +1 =

Original: (Id + ηA ) − 1 ( z k ) , is called the proximal point method (PPM)
Revised: (Id + ηA ) − 1 ( z k ) , is called the proximal point method (PPM)

Original: [Rockafellar, 1976].
Revised: [Rockafellar, 1976].

Original: This can be equivalently written as z k ∈ (Id + ηA )
Revised: This can be equivalently written as z k ∈ (Id + ηA )

Original: ( z k +1 ) , and subsequently as
Revised: ( z k +1 ) , and subsequently as

Original: [Equation]
Revised: [Equation]

Original: From now on, for simplicity only, we will ignore the fact that in general, A ( z k +1 ) is a set, and will write z k +

Revised: From now on, for simplicity only, we will ignore the fact that in general, A ( z k +1 ) is a set, and will write z k +

Original: =
Revised: =

Original: z k − ηA
Revised: z k − ηA

Original: ( z k +1 ) instead to mean the same thing, i.e., that there exists u ∈ A ( z k +1 )
Revised: ( z k +1 ) instead to mean the same thing, i.e., that there exists u ∈ A ( z k +1 )

Original: such that z k +1 =
Revised: such that z k +1 =

Original: z k − ηu .
Revised: z k − ηu .

Original: 3.2 PPM applied to the saddle-point problem
Revised: 3.2 PPM applied to the saddle-point problem

Original: The optimality conditions (8) of the saddle point problem (7) can be written in the form (9) with z = ( x ; y ) ∈ R
Revised: The optimality conditions (8) of the saddle point problem (7) can be written in the form (9) with z = ( x ; y ) ∈ R

Original: d x × R d y as follows 7 :
Revised: d x × R d y as follows 7 :

Original: [Equation]
Revised: [Equation]

Original: Allowing for different stepsizes η x , η y for each block of the vector z =
Revised: Allowing for different stepsizes η x , η y for each block of the vector z =

Original: ( x ; y ) , PPM applied to (10) takes the form
Revised: ( x ; y ) , PPM applied to (10) takes the form

Original: [Equation]
Revised: [Equation]

Original: The main advantage of this method is its unboundedly fast convergence rate under weak assumptions.
Revised: The main advantage of this method is its unboundedly fast convergence rate under weak assumptions.

Original: According to Theorem 4, the proof of which we provide in the appendix for completeness, if G and
Revised: According to Theorem 4, the proof of which we provide in the appendix for completeness, if G and

Original: Algorithm 1: Input : Initial point ( x  0 , y  0 ) ∈
Revised: F ⋆ are proper and closed, G is µ x strongly convex and F ⋆ is µ y strongly convex, then any choice of stepsizes η x > 0 and η y > 0 (yes, without an upper bound!) PPM find an ε -accurate solution in

Original: [Equation]
Revised: [Equation]

Original: ¯ y 
Revised: iterations. Unfortunately, PPM is not implementable since in order to compute x k +1 , we need to know y k +1 , and vice versa.

Original: [Equation]
Revised: Algorithm

Original: = y [Equation] Step sizes η
Revised: 1: Input : Initial point ( x 0 , y 0 )

Original: [Equation] η y {+, ¯ y 
+}
Revised: ∈ R d x × R d y {+, ¯ y 
+}

Original: [Equation] β y > 0 , θ ∈ [0 , 1] 2: for k = 0 , 1 , . . .
Revised: = y 0 ; Step sizes η x , η y , β y > 0 , θ ∈ [0 , 1] 2: for k = 0 , 1 , . . .

Original: do 3:
Revised: do 3:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:1) 6: end for F ⋆ are proper and closed, G is µ x strongly convex and F ⋆ is µ y strongly convex, then any choice of stepsizes η x > 0 and η y > 0 (yes, without an upper bound!) PPM find an ε -accurate solution in O (cid:18)(cid:18)
Revised: 5: ¯

Original: [Equation]
Revised: [Equation]

Original: (11) iterations.
Revised: (cid:1) : end for

Original: Unfortunately, PPM is not implementable since in order to compute x k +1 , we need to know y k +1 , and vice versa.

Original: 3.3 Chambolle-Pock: Making PPM implementable, and fast
Revised: 3.3 Chambolle-Pock: Making PPM implementable, and fast

Original: In order to overcome the above problem, Chambolle and Pock [2011] proposed to replace y k +1 with y k
Revised: In order to overcome the above problem, Chambolle and Pock [2011] proposed to replace y k +1 with y k

Original: (see Algorithm 1 in [Chambolle and Pock, 2011]), which leads to
Revised: (see Algorithm 1 in [Chambolle and Pock, 2011]), which leads to

Original: [Equation]
Revised: [Equation]

Original: Although this method is implementable, it has its own disadvantages, one of which is its weak iteration complexity bound
Revised: Although this method is implementable, it has its own disadvantages, one of which is its weak iteration complexity bound

Original: [Equation]
Revised: [Equation]

Original:  Chambolle and Pock [2011] proposed to fix this problem via an extrapolation step of the dual variable (see Algorithm 3 in [Chambolle and Pock, 2011]):
Revised: [Equation] Chambolle and Pock [2011] proposed to fix this problem via an extrapolation step of the dual variable (see Algorithm 3 in [Chambolle and Pock, 2011]):

Original: [Equation]
Revised: [Equation]

Original: This new method enjoys the much better iteration complexity bound
Revised: This new method enjoys the much better iteration complexity bound

Original: [Equation]
Revised: [Equation]

Original:  Accelerated Primal-Dual Algorithm (Algorithm 1)
Revised:  Accelerated Primal-Dual Algorithm (Algorithm 1)

Original: Recall that the Chambolle-Pock method requires F ⋆ to be strongly-convex to obtain a linear convergence rate.
Revised: Recall that the Chambolle-Pock method requires F ⋆ to be strongly-convex to obtain a linear convergence rate.

Original: However, in our setting, F ⋆ is not strongly convex 8 (see Assumption 3), and ChambollePock method does not converge linearly in this scenario.
Revised: However, in our setting, F ⋆ is not strongly convex 8 (see Assumption 3), and ChambollePock method does not converge linearly in this scenario.

Original: 4.1 Modifying Chambolle-Pock to preserve linear rate without strong convexity of F ⋆
Revised: 4.1 Modifying Chambolle-Pock to preserve linear rate without strong convexity of F ⋆

Original: To obtain a linear rate, we modify the dual update step of the algorithm using a trick proposed by
Revised: To obtain a linear rate, we modify the dual update step of the algorithm using a trick proposed by

Original: Kovalev et al.
Revised: Kovalev et al.

Original: [2021a] that was shown to work in the regime when F ⋆ is smooth; the innovation here is that we do not need this assumption (see Table 1).
Revised: [2021a] that was shown to work in the regime when F ⋆ is smooth; the innovation here is that we do not need this assumption (see Table 1).

Original: From this point onwards, we will also need to assume G to be L x -smooth (see Assumption 2).
Revised: From this point onwards, we will also need to assume G to be L x -smooth (see Assumption 2).

Original: In particular, we propose to modify the update step for y k +1 in the Chambolle-Pock method as follows:
Revised: In particular, we propose to modify the update step for y k +1 in the Chambolle-Pock method as follows:

Original: [Equation]
Revised: [Equation]

Original: This is a new method, which we call APDA (formalized as Algorithm 1).
Revised: This is a new method, which we call APDA (formalized as Algorithm 1).

Original: 4.2 APDA converges linearly
Revised: .2 APDA converges linearly

Original: Our first result shows that APDA indeed converges linearly, without the need for F ⋆ to be strongly convex.
Revised: Our first result shows that APDA indeed converges linearly, without the need for F ⋆ to be strongly convex.

Original: Theorem 1 (Convergence of APDA; informal) .
Revised: Theorem 1 (Convergence of APDA; informal) .

Original: Let Assumptions 1, 2,3 and 4 hold.
Revised: Let Assumptions 1, 2,3 and 4 hold.

Original: Then, with a suitable selection of stepsizes, APDA (Algorithm 1) solves problem (7) in
Revised: Then, with a suitable selection of stepsizes, APDA (Algorithm 1) solves problem (7) in

Original: [Equation]
Revised: [Equation]

Original: iterations.
Revised: iterations.

Original: The formal statement and proof can be found in the appendix (see Theorem 7).
Revised: The formal statement and proof can be found in the appendix (see Theorem 7).

Original:  Accelerated Primal-Dual Algorithm with Inexact Prox (Algorithm 2)
Revised:  Accelerated Primal-Dual Algorithm with Inexact Prox (Algorithm 2)

Original: The key disadvantage of APDA is that it requires the evaluations of the proximity operator of G , which can be very expensive in some applications.
Revised: The key disadvantage of APDA is that it requires the evaluations of the proximity operator of G , which can be very expensive in some applications.

Original: To remedy the situation, we first notice that Stepof APDA can be equivalently written in the form
Revised: To remedy the situation, we first notice that Line 3 of APDA can be equivalently written in the form

Original: [Equation]
Revised: [Equation]

Original:  that is, this step involves the evaluation of the prox of G .
Revised: 15) that is, this step involves the evaluation of the prox of G .

Original: The key idea of this section is to replace this by an inexact prox computation via a suitably selected iterative method M (this is the method performing the inner iterations in Table 1).
Revised: The key idea of this section is to replace this by an inexact prox computation via a suitably selected iterative method M (this is the method performing the inner iterations in Table 1).

Original: This leads to our next method: APDA with Inexact Prox
Revised: This leads to our next method: APDA with Inexact Prox

Original: (Algorithm 2).
Revised: (Algorithm 2).

Original: Algorithm 2 APDA with Inexact Prox

Original: Find ˆ x k as a final point of T iteration of some method M for following problem:
Revised: of some method M for following problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: for

Original: 5.1 Gradient methods for finding a stationary point of convex functions
Revised: 5.1 Gradient methods for finding a stationary point of convex functions

Original: A key feature of Algorithm 2 is its reliance on a subroutine M for an inexact evaluation of the prox of G via solving the auxiliary problem (15).
Revised: A key feature of Algorithm 2 is its reliance on a subroutine M for an inexact evaluation of the prox of G via solving the auxiliary problem (15).

Original: Our theory requires the method M to be able to output, after T iterations, a point ˆ x k such that
Revised: Our theory requires the method M to be able to output, after T iterations, a point ˆ x k such that

Original: [Equation]
Revised: [Equation]

Original: where α ≥ 2 .
Revised: where α ≥ 2 .

Original: In other words, we require a reduction of the squared norm of the gradient with a fast sublinear rate.
Revised: In other words, we require a reduction of the squared norm of the gradient with a fast sublinear rate.

Original: In the next lemma, we present three examples of such methods.
Revised: In the next lemma, we present three examples of such methods.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let Ψ : R d x → R be an L -smooth convex function, and let w ⋆ be a minimizer of Ψ .
Revised: Let Ψ : R d x → R be an L -smooth convex function, and let w ⋆ be a minimizer of Ψ .

Original: Then there exists a gradient-based method M which after T iterations outputs a point w T satisfying
Revised: Then there exists a gradient-based method M which after T iterations outputs a point w T satisfying

Original: [Equation]
Revised: [Equation]

Original: for all starting points x 0 ∈ R
Revised: for all starting points x 0 ∈ R

Original: d x and some universal constant A > 0 .
Revised: d x and some universal constant A > 0 .

Original: In particular,
Revised: In particular,

Original: (i) if M is GD, then ∥∇ Ψ( w T ) ∥ 2 ≤ 4 L 2 ∥ w 0 − w ⋆ ∥ 2 T
Revised: (i) if M is GD, then ∥∇ Ψ( w T ) ∥ 2 ≤ 4 L 2 ∥ w 0 − w ⋆ ∥ 2 T

Original: (ii) if M is a combination 9 of Fast GD [Nesterov, 2004] and GD, then ∥∇ Ψ( w T )
Revised: (ii) if M is a combination 9 of Fast GD [Nesterov, 2004] and GD, then ∥∇ Ψ( w T )

Original: ∥ 2 ≤ 64 L 2 ∥ w 0 − w ⋆ ∥ 2 T, (iii) if M is a combination 10 of Fast GD [Nesterov, 2004] and FSFOM [Kim and Fessler, 2021], then ∥∇ Ψ(w T ) ∥ 2 ≤ 256 L 2 ∥ w 0 − w ⋆ ∥ 2 T
Revised: ∥ 2 ≤ 64 L 2 ∥ w 0 − w ⋆ ∥ 2 T, (iii) if M is a combination 10 of Fast GD [Nesterov, 2004] and FSFOM [Kim and Fessler, 2021], then ∥∇ Ψ(

Revised: w T ) ∥ 2 ≤ 256 L 2 ∥ w 0 − w ⋆ ∥ 2 T.

Original: Let w ⋆k =
Revised: Let w ⋆k =

Original: arg
Revised: arg

Original: min w ∈ R dx Ψ k
Revised: min w ∈ R dx Ψ k

Original: ( w ) .
Revised: ( w ) .

Original: Since Ψ k is (cid:0) L x + η − 1 x (cid:1) -smooth, Lemma 1 implies that T iterations of a method M satisfying (17) applied to the auxiliary problem (15) with starting point w 

Revised: Since Ψ k is (cid:0) L x + η − 1 x (cid:1) -smooth, Lemma 1 implies that T iterations of a method M satisfying (17) applied to the auxiliary problem (15) with starting point w

Original: = x k yield point w T
Revised: = x k yield point w T

Original: = ˆ x k for which
Revised: = ˆ x k for which

Original: [Equation]
Revised: [Equation]

Original: 5.2 APDA with Inexact Prox converges linearly
Revised: 5.2 APDA with Inexact Prox converges linearly

Original: Now we can provide the main theorem with the total complexity of gradient computation ∇ G and proximity operator computation ∂F ⋆ .
Revised: Now we can provide the main theorem with the total complexity of gradient computation ∇ G and proximity operator computation ∂F ⋆ .

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Assumptions 1, 2, 3, 4 hold.
Revised: Let Assumptions 1, 2, 3, 4 hold.

Original: Then there exist parameters of Algorithm 2 such that the total # of evaluations of prox F ⋆ and the total number evaluations of the gradient of ∇ G to find an ε solution of problem (7) are
Revised: Then there exist parameters of Algorithm 2 such that the total # of evaluations of prox F ⋆ and the total number evaluations of the gradient of ∇ G to find an ε solution of problem (7) are

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: In particular,
Revised: In particular,

Original: (i) if the inner method M is GD, then the total number of ∇ G computations is equal to
Revised: (i) if the inner method M is GD, then the total number of ∇ G computations is equal to

Original: [Equation]
Revised: [Equation]

Original: (ii) if the inner method M is combination of Fast GD and GD, then the total number of ∇ G computations is equal to
Revised: (ii) if the inner method M is combination of Fast GD and GD, then the total number of ∇ G computations is equal to

Original: [Equation]
Revised: [Equation]

Original: (iii)if the inner method M is combination of Fast GD and FSFOM, then the total number of ∇ G computations is equal to
Revised: (iii) if the inner method M is combination of Fast GD and FSFOM, then the total number of ∇ G computations is equal to

Original: [Equation]
Revised: [Equation]

Original: The proof relies on several lemmas; their statements and the proof of the theorem can be found in the appendix.
Revised: The proof relies on several lemmas; their statements and the proof of the theorem can be found in the appendix.

Original: Note that our way of performing inexact computation of prox of G allows us to keep the same complexity as APDA (Algorithm 1) in terms of the number of evaluations of the prox of F ⋆ .
Revised: Note that our way of performing inexact computation of prox of G allows us to keep the same complexity as APDA (Algorithm 1) in terms of the number of evaluations of the prox of F ⋆ .

Original: When
Revised: When

Original: M is chosen to be Gradient Decent ( α = 2 ), the number of computations of the gradient of ∇ G , given by (20), is larger than the number of computations of the prox of G for APDA.
Revised: M is chosen to be Gradient Decent ( α = 2 ), the number of computations of the gradient of ∇ G , given by (20), is larger than the number of computations of the prox of G for APDA.

Original: Fortunately, we can reduce this if GD is replaced with a faster method.
Revised: Fortunately, we can reduce this if GD is replaced with a faster method.

Original: For example, if we choose M to be a simple combination of Fast GD (FGD) and GD, in which case α = 3 , Theorem 2 says that the number of computations of the gradient of ∇ G is can be reduced to (21) (this choice is mentioned in the second-to-last row of Table 2).
Revised: For example, if we choose M to be a simple combination of Fast GD (FGD) and GD, in which case α = 3 , Theorem 2 says that the number of computations of the gradient of ∇ G is can be reduced to (21) (this choice is mentioned in the second-to-last row of Table 2).

Original: A further reduction is possible if we instead employ a combination of
Revised: A further reduction is possible if we instead employ a combination of

Original: FSFOM and Fast GD; see (22) and the last row of Table 2.
Revised: FSFOM and Fast GD; see (22) and the last row of Table 2.

Original:  Accelerated Primal Dual Algorithm with Inexact Prox and Accelerated Gossip (Algorithm 5)
Revised:  Accelerated Primal Dual Algorithm with Inexact Prox and Accelerated Gossip (Algorithm 5)

Original: In this section, we consider the most significant applications of Algorithm 2: decentralized optimization over a network G =
Revised: In this section, we consider the most significant applications of Algorithm 2: decentralized optimization over a network G =

Original: ( V , E ) , and federated learning.
Revised: ( V , E ) , and federated learning.

Original: In particular, we consider the finite-sum optimization problem
Revised: In particular, we consider the finite-sum optimization problem

Original: [Equation]
Revised: [Equation]

Original: (see (5)) interpreted as follows: n = |V| is the number of clients/agents in the network.
Revised: (see (5)) interpreted as follows: n = |V| is the number of clients/agents in the network.

Original: Communication can only happen between clients connected by an edge.
Revised: Communication can only happen between clients connected by an edge.

Original: The prevalent paradigm in federated learning, where a single server orchestrates communication in rounds, arises as a special case of this with G being the fully connected network.
Revised: The prevalent paradigm in federated learning, where a single server orchestrates communication in rounds, arises as a special case of this with G being the fully connected network.

Original: If ˆ W is the Laplacian 11 of graph G , and let W =
Revised: If ˆ W is the Laplacian 11 of graph G , and let W =

Original: ˆ W ⊗ I dn .
Revised: ˆ W ⊗ I dn .

Original: Then problem (5) can be rewritten in following equivalent way:
Revised: Then problem (5) can be rewritten in following equivalent way:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where x ⊤ =
Revised: where x ⊤ =

Original: ( x ⊤ 1 , x ⊤ 2 , . . .
Revised: ( x ⊤ 1 , x ⊤ 2 , . . .

Original: , x ⊤ n ) , P ( x ) = (cid:80) ni
Revised: , x ⊤ n ) , P ( x ) = (cid:80) ni

Original: =

Revised: =

Original: f i ( x i ) and ψ ( x ) = 

Revised: f i ( x i ) and ψ ( x ) =

Original: iff x = 0 , and ψ ( x ) =
Revised: iff x = 0 , and ψ ( x ) =

Original: + ∞ otherwise.
Revised: + ∞ otherwise.

Original: Problem (23) is a special case of (1).
Revised: Problem (23) is a special case of (1).

Original: By dualizing the nonsmooth (but proper, closed and convex) penalty, we get the equivalent saddle point formulation
Revised: By dualizing the nonsmooth (but proper, closed and convex) penalty, we get the equivalent saddle point formulation

Original: [Equation]
Revised: [Equation]

Original: As we can see, this problem (24) is the particular case of the problem (7).
Revised: As we can see, this problem (24) is the particular case of the problem (7).

Original: It means that we can solve it by Algorithm 2, for example.
Revised: It means that we can solve it by Algorithm 2, for example.

Original: Moreover, we do not have to compute the prox of ψ ⋆ due to the fact that ψ ⋆ ( · ) ≡ 0 , because ψ ( · ) is the indicator function of { 0 } .
Revised: Moreover, we do not have to compute the prox of ψ ⋆ due to the fact that ψ ⋆ ( · ) ≡ 0 , because ψ ( · ) is the indicator function of { 0 } .

Original: We thus arrive at the final formulation
Revised: We thus arrive at the final formulation

Original: [Equation]
Revised: [Equation]

Original: 6.1 Application of Algorithm 2 to (25)
Revised: 6.1 Application of Algorithm 2 to (25)

Original: Before providing the complexity results related to the application of Algorithm 2 to problem (25), note that L xy =
Revised: Before providing the complexity results related to the application of Algorithm 2 to problem (25), note that L xy =

Original: (cid:112) λ max ( W ) and µ xy = (cid:113) λ
Revised: (cid:112) λ max ( W ) and µ xy = (cid:113) λ

Original: +min ( W ) and define
Revised: +min ( W ) and define

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: According to Theorem 2, the total number of evaluations of the prox of ψ ⋆ , i.e., the communication complexity, and the total number of evaluations of ∇ P , i.e., computation complexity, are
Revised: According to Theorem 2, the total number of evaluations of the prox of ψ ⋆ , i.e., the communication complexity, and the total number of evaluations of ∇ P , i.e., computation complexity, are

Original: [Equation]
Revised: [Equation]

Original: (27) respectively.
Revised: (27) respectively.

Original: For example, in centralized case, when G is the complete graph ( χ = 1 ) and M is chosen to be GD ( α = 2 ), we obtain the same complexities as ProxSkip [Mishchenko et al., 2022]:
Revised: For example, in centralized case, when G is the complete graph ( χ = 1 ) and M is chosen to be GD ( α = 2 ), we obtain the same complexities as ProxSkip [Mishchenko et al., 2022]:

Original: [Equation]
Revised: [Equation]

Original:  However, this can be improved by using a more elaborate subroutine M .
Revised: κ ) . However, this can be improved by using a more elaborate subroutine M .

Original: If instead of GD we use either FGD + GD ( α = 3 ) or FGD + FSFOM ( α = 4 ) in place of M , the number of communication rounds will be the same as in the case of ProxSkip (or Algorithm 2 used with M = GD).
Revised: If instead of GD we use either FGD + GD ( α = 3 ) or FGD + FSFOM ( α = 4 ) in place of M , the number of communication rounds will be the same as in the case of ProxSkip (or Algorithm 2 used with M = GD).

Original: However, the total number of gradient computations gets improved to ♯ comp = (cid:101) O (cid:16) κ 56 (cid:17) in the first case, and to ♯ comp = (cid:101) O (cid:16) κ 34 (cid:17) in the second case.
Revised: However, the total number of gradient computations gets improved to ♯ comp = (cid:101) O (cid:16) κ 56 (cid:17) in the first case, and to ♯ comp = (cid:101) O (cid:16) κ 34 (cid:17) in the second case.

Original: 6.2 Improvement on Algorithm 2 via accelerated gossip
Revised: 6.2 Improvement on Algorithm 2 via accelerated gossip

Original: Compared to the ProxSkip computation complexity (cid:101) O (cid:0) √ κχ (cid:1) , in the general decentralized case (i.e., χ > 1 ), complexity (27) of our Algorithm 2 is worse if κ ≤ χ .
Revised: Compared to the ProxSkip computation complexity (cid:101) O (cid:0) √ κχ (cid:1) , in the general decentralized case (i.e., χ > 1 ), complexity (27) of our Algorithm 2 is worse if κ ≤ χ .

Original: To tackle this problem, we propose to enhance Algorithm 2 using the accelerated gossip technique [Scaman et al., 2017].
Revised: To tackle this problem, we propose to enhance Algorithm 2 using the accelerated gossip technique [Scaman et al., 2017].

Original: Based on this approach, we propose one more (and final) method: Algorithm 5.
Revised: Based on this approach, we propose one more (and final) method: Algorithm 5.

Original: For this method, we prove the following result.
Revised: For this method, we prove the following result.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let Assumptions 1 and 2 hold for function P .
Revised: Let Assumptions 1 and 2 hold for function P .

Original: Then there exist parameters of Algorithmsuch that in order to find an ε -solution of problem (7) , the total number of communications and the total number of gradient computations can be bounded by
Revised: Then there exist parameters of Algorithmsuch that in order to find an ε -solution of problem (7) , the total number of communications and the total number of gradient computations can be bounded by

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: In particular, if the inner method M is
Revised: In particular, if the inner method M is

Original: (i) GD, then

Original: the

Original: number

Original: of local steps

Original: is

Original: equal to (cid:101)

Original: O

Original:  (cid:0) κ 1 /  (cid:1) and the total number of gradient computa- tions is 
Revised: (i) GD, then the number of local steps is equal to (cid:101) O (cid:0) κ 1 / 2 (cid:1) and the total number of gradient computations is equal to (cid:101)

Original: equal to (cid:101) O ( κ ) ; (ii) FGD + GD, then the number of local steps is equal to (cid:101) O (cid:0) κ 1 / 3 (cid:1) and the total number of gradient computations is equal to (cid:101) O (cid:0) κ 5 / 6 (cid:1) ; (iii)FGD + FSFOM, then the number of local steps is equal to (cid:101) O (cid:0) κ 1 / 4 (cid:1) and the total number of gradient computations is equal to (cid:101) O (cid:0) κ 3 / 4 (cid:1) .
Revised: O ( κ ) ; (ii) FGD + GD, then the number of local steps is equal to (cid:101) O (cid:0) κ 1 / 3 (cid:1) and the total number of gradient computations is equal to (cid:101) O (cid:0) κ 5 / 6 (cid:1) ; (iii) FGD + FSFOM, then the number of local steps is equal to (cid:101) O (cid:0) κ 1 / 4 (cid:1) and the total number of gradient computations is equal to (cid:101) O (cid:0) κ 3 / 4 (cid:1) .

Original: The communication complexities obtained this way are substantially better than those of decentralized
Revised: The communication complexities obtained this way are substantially better than those of decentralized

Original: ProxSkip; see Table 3 and the commentary in the next subsection.
Revised: ProxSkip; see Table 3 and the commentary in the next subsection.

Original: Recall that in Table 2, we already compared ProxSkip and our Algorithm 2 in the centralized case.
Revised: Recall that in Table 2, we already compared ProxSkip and our Algorithm 2 in the centralized case.

Original: In
Revised: In

Original: Table 3 we add to this a comparison in the decentralized case, and include our Algorithm 5 as well.
Revised: Table 3 we add to this a comparison in the decentralized case, and include our Algorithm 5 as well.

Original: In particular, in Table 3 we compare the complexity results of our methods (Algorithms 2 and 5) for solving the decentralized optimization problem (5) to two selected benchmarks: D-SGD [Koloskova et al., 2020] and ProxSkip [Mishchenko et al., 2022].
Revised: In particular, in Table 3 we compare the complexity results of our methods (Algorithms 2 and 5) for solving the decentralized optimization problem (5) to two selected benchmarks: D-SGD [Koloskova et al., 2020] and ProxSkip [Mishchenko et al., 2022].

Original: First, observe that ProxSkip has vastly superior communication complexity to D-SGD, both in the centralized case (i.e., for fully-connected network; χ = 1 ), where the improvement is from O ( κ ) to (cid:101) O ( √ κ ) , and the decentralized case ( χ > 1 ), where the improvement is from (cid:101) O ( κχ ) to (cid:101) O ( √ κχ ) .
Revised: First, observe that ProxSkip has vastly superior communication complexity to D-SGD, both in the centralized case (i.e., for fully-connected network; χ = 1 ), where the improvement is from O ( κ ) to (cid:101) O ( √ κ ) , and the decentralized case ( χ > 1 ), where the improvement is from (cid:101) O ( κχ ) to (cid:101) O ( √ κχ ) .

Original: In the decentralized case, both our methods match the (cid:101) O ( √ κχ ) communication complexity of
Revised: In the decentralized case, both our methods match the (cid:101) O ( √ κχ ) communication complexity of

Original: ProxSkip.
Revised: ProxSkip.

Original: However, Algorithm 2 does so only when √ κχ ≥ χ (i.e., when κ > χ ).
Revised: However, Algorithm 2 does so only when √ κχ ≥ χ (i.e., when κ > χ ).

Original: On the other hand, both our methods have an improved bound on the number of local gradient computations, depending on what subroutine M they employ.
Revised: On the other hand, both our methods have an improved bound on the number of local gradient computations, depending on what subroutine M they employ.

Original: The improvement is from (cid:101) O ( √ κ ) to (cid:101)
Revised: The improvement is from (cid:101) O ( √ κ ) to (cid:101)

Original: O ( 3 √ κ ) (when M = FGD + GD) to (cid:101)
Revised: O ( 3 √ κ ) (when M = FGD + GD) to (cid:101)

Original: O ( 4 √ κ ) (when M = FGD + FSFOM).
Revised: O ( 4 √ κ ) (when M = FGD + FSFOM).

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: The work of all authors was supported by the KAUST Baseline Research Grant awarded to P.
Revised: The work of all authors was supported by the KAUST Baseline Research Grant awarded to P.

Original: Richtárik.
Revised: Richtárik.

Original: The work of A. Sadiev was supported by the Visiting Student Research Program (VSRP) at KAUST.
Revised: The work of A. Sadiev was supported by the Visiting Student Research Program (VSRP) at KAUST.

Original: The work of A. Sadiev was also partially supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian
Revised: The work of A. Sadiev was also partially supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian

Original: Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q ) and the agreement with the Ivannikov Institute for System Programming of the Russian Academy of Sciences dated November 2, 2021 No. 70-2021-00142.
Revised: Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q ) and the agreement with the Ivannikov Institute for System Programming of the Russian Academy of Sciences dated November 2, 2021 No. 70-2021-00142.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] We crearly state all theorem (b) Did you describe the limitations of your work?
Revised: [Yes] We crearly state all theorem (b) Did you describe the limitations of your work?

Original: [Yes] Assumptions are our limitations.
Revised: [Yes] Assumptions are our limitations.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our work is mainly theoretical, with no foreseeable negative societal impacts.
Revised: Our work is mainly theoretical, with no foreseeable negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 1.
Revised: [Yes] See Section 1.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix. 3.
Revised: [Yes] See Appendix. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Appendix A Additional Related Work
Revised: Appendix Contents 1 Introduction 1.

Revised: Towards provable communication acceleration via delayed parameter synchronization 1.

Revised: Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: ProxSkip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: 2 Summary of Contributions 3 From Proximal Point Method to Chambolle-Pock 3.

Revised: Proximal-Point Method for finding zeros of monotone operators . . . . . . . . . .

Revised: PPM applied to the saddle-point problem . . . . . . . . . . . . . . . . . . . . . .

Revised: Chambolle-Pock: Making PPM implementable, and fast . . . . . . . . . . . . . .

Revised: 4 Accelerated Primal-Dual Algorithm (Algorithm 1) 4.

Revised: Modifying Chambolle-Pock to preserve linear rate without strong convexity of F ⋆ .

Revised: APDA converges linearly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: 5 Accelerated Primal-Dual Algorithm with Inexact Prox (Algorithm 2) 5.

Revised: Gradient methods for finding a stationary point of convex functions . . . . . . . .

Revised: APDA with Inexact Prox converges linearly . . . . . . . . . . . . . . . . . . . . .

Revised: 6 Accelerated Primal Dual Algorithm with Inexact Prox and Accelerated Gossip (Algorithm 5) 6.

Revised: Application of Algorithm 2 to (25) . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: Improvement on Algorithm 2 via accelerated gossip . . . . . . . . . . . . . . . . .

Original: An alternative approach to achieving improved communication efficiency is through the use of communication compression via (unbiased) quantization [Zhu et al., 2017, Alistarh et al., 2017,
Revised: A.1 Summary of complexity results for Algorithms 2 and 5 applied to decentralized optimization B Analysis of the Proximal-Point Method B.1 The Proximal-Point Method

Revised: B.2 Bonding the distance of the primal iterates to the primal solution

Revised: B.3 Bonding the distance of the dual iterates to the dual solution

Revised: B.4 Complexity of the Proximal-Point Method

Revised: C Analysis of the Chambolle-Pock Method

Original: {+C.
+}
Revised: {+C.
+}

Revised: The Chambolle-Pock Method

Revised: C.2 Bonding the distance of the primal iterates to the primal solution

Revised: C.3 Bonding the distance of the dual iterates to dual solution

Revised: C.4 Formal statement and proof

Revised: Pock θ = case) . . . . . .

Revised: C

Revised: 5 Formal statement and proof of (13) (Chambolle-Pock in θ > case) . . . . . .

Revised: D.1 Three lemmas . .

Original: {+D.
+}
Revised: {+D.
+}

Revised: Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: D.3 Proof of Theorem 1 (informal) . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: E Analysis of the Accelerated Primal-Dual Algorithm with Inexact Prox (Algorithm 2)

Revised: E.1 Three Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: E.2 Detailed theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: E.3 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: F Analysis of the

Revised: Algorithm with Inexact Prox and

Revised: Gossip (Algorithm 5)

Revised: F.

Revised: Gossip matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: F.

Revised: Theorem 3 follows from Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: G Proof of Lemma

Revised: A Additional Related Work

Revised: An alternative approach to achieving improved communication efficiency is through the use of communication compression via (unbiased) quantization [Zhu et al., 2017, Alistarh et al., 2017,

Original: Horváth et al., 2019a, Mishchenko et al., 2021b], sketching [Hanzely et al., 2018, Safaryan et al., 2021] or sparsification [Wangni et al., 2017, Wang et al., 2018, Mishchenko et al., 2020].
Revised: Horváth et al., 2019a, Mishchenko et al., 2021b], sketching [Hanzely et al., 2018, Safaryan et al., 2021] or sparsification [Wangni et al., 2017, Wang et al., 2018, Mishchenko et al., 2020].

Original: Modern variants offering with variance reduction for the variance caused by compression Mishchenko et al.
Revised: Modern variants offering with variance reduction for the variance caused by compression Mishchenko et al.

Original: [2019], Horváth et al.
Revised: [2019], Horváth et al.

Original: [2019b], Gorbunov et al.
Revised: [2019b], Gorbunov et al.

Original: [2020b], Safaryan et al.
Revised: [2020b], Safaryan et al.

Original: [2021], Mishchenko et al.
Revised: [2021], Mishchenko et al.

Original: [2021b], Wang et al.
Revised: [2021b], Wang et al.

Original: [2021a], Shulgin and Richtárik [2021], adaptivity [Mishchenko et al., 2021b], bidirectional compression [Horváth et al., 2019a, Tang et al., 2019, Philippenko and Dieuleveut, 2020] or acceleration Li et al.
Revised: [2021a], Shulgin and Richtárik [2021], adaptivity [Mishchenko et al., 2021b], bidirectional compression [Horváth et al., 2019a, Tang et al., 2019, Philippenko and Dieuleveut, 2020] or acceleration Li et al.

Original: [2020c] enjoy better theoretical rates and practical performance.
Revised: [2020c] enjoy better theoretical rates and practical performance.

Original: Variancereduction for communication compression has been extended to work over arbitrary connected networks [Kovalev et al., 2021b], and to second-order methods [Islamov et al., 2021, Safaryan et al., 2022, Qian et al., 2022, Islamov et al., 2022].
Revised: Variancereduction for communication compression has been extended to work over arbitrary connected networks [Kovalev et al., 2021b], and to second-order methods [Islamov et al., 2021, Safaryan et al., 2022, Qian et al., 2022, Islamov et al., 2022].

Original: The current state-of-the-art communication complexity in the smooth nonconvex regime is offered by the MARINA
Revised: The current state-of-the-art communication complexity in the smooth nonconvex regime is offered by the MARINA

Original: [Gorbunov et al., 2021, Szlendak et al., 2022] and DASHA [Tyurin and Richtárik, 2022] methods.
Revised: [Gorbunov et al., 2021, Szlendak et al., 2022] and DASHA [Tyurin and Richtárik, 2022] methods.

Original: Greedy (biased) compressors, such as Top-K sparsification [Alistarh et al., 2018] or Rank-K approximation [Vogels et al., 2019], require a different approach via an error-feedback/compensation mechanism [Stich et al., 2018, Stich and Karimireddy, 2019, Tang et al., 2019, Gorbunov et al., 2020c].
Revised: Greedy (biased) compressors, such as Top-K sparsification [Alistarh et al., 2018] or Rank-K approximation [Vogels et al., 2019], require a different approach via an error-feedback/compensation mechanism [Stich et al., 2018, Stich and Karimireddy, 2019, Tang et al., 2019, Gorbunov et al., 2020c].

Original: For a more modern treatment of error feedback offering current state-of-the-art rates, we refer the reader to [Richtárik et al., 2021, Fatkhullin et al., 2021, Richtárik et al., 2022, Qian et al., 2020].
Revised: For a more modern treatment of error feedback offering current state-of-the-art rates, we refer the reader to [Richtárik et al., 2021, Fatkhullin et al., 2021, Richtárik et al., 2022, Qian et al., 2020].

Original: An alternative approach based on the transformation of a biased compressor into a related induced unbiased compressor was proposed in [Horváth and Richtárik, 2021], and a unified treatment of variance reduction and error-feedback was proposed in [Condat et al., 2022].
Revised: An alternative approach based on the transformation of a biased compressor into a related induced unbiased compressor was proposed in [Horváth and Richtárik, 2021], and a unified treatment of variance reduction and error-feedback was proposed in [Condat et al., 2022].

Original: For a systems-oriented survey, we recommend the reader the work of Xu et al.
Revised: For a systems-oriented survey, we recommend the reader the work of Xu et al.

Original: In this work we do not consider the communication compression approach to communication efficiency since this area is much more understood, and many methods already improve on the theoretical communication complexity of vanilla GD and SGD, often by significant data and dimensiondependent margins.
Revised: In this work we do not consider the communication compression approach to communication efficiency since this area is much more understood, and many methods already improve on the theoretical communication complexity of vanilla GD and SGD, often by significant data and dimensiondependent margins.

Original: Instead, we focus on the practice of delayed parameter synchronization via local training, and contribute to the theoretical foundations of this immensely popular yet poorly understood approach to achieving communication efficiency.
Revised: Instead, we focus on the practice of delayed parameter synchronization via local training, and contribute to the theoretical foundations of this immensely popular yet poorly understood approach to achieving communication efficiency.

Original: A.1 Summary of complexity results for Algorithms 2 and 5 applied to decentralized optimization B Analysis of the Proximal-Point Method
Revised: A.1 Summary of complexity results for Algorithms 2 and 5 applied to decentralized optimization B Analysis of the Proximal-Point Method

Original: In this section we justify the claims we made in Section 3.2 about the Proximal-Point Method (Algorithm 3).
Revised: In this section we justify the claims we made in Section 3.2 about the Proximal-Point Method (Algorithm 3).

Original: In particular, we prove the complexity result (11).
Revised: In particular, we prove the complexity result (11).

Original: The result is not new of course, but we could not find a source for the proof we include here.
Revised: The result is not new of course, but we could not find a source for the proof we include here.

Original: B.1 The Proximal-Point Method
Revised: B.1 The Proximal-Point Method

Original: We have described the Proximal-Point Method informally in Section 3.2.
Revised: We have described the Proximal-Point Method informally in Section 3.2.

Original: Here we state it formally as
Revised: Here we state it formally as

Original: Algorithm 3.
Revised: Algorithm 3.

Original: Algorithm  1: Input : Initialpoint ( x 0 , y 0 )
Revised: Algorithm 3 Proximal-Point Method 1: Input : Initial

Original:  ∈ R d x × R d y ; Stepsizesη x , η y >2: for k = 0 , 1 ,. . .
Revised: ( x 0 , y 0 ) ∈ R d x × R d y ; Stepsizes x , η y >2: for k = 0 , 1 ,

Original: do 3: x k +

Revised: do 3: x k +

Original: = x
Revised: = x

Original: k − η x (cid:0) ∂G ( x k +1 )
Revised: k − η x (cid:0) ∂G ( x k +1 )

Original: + K ⊤ y k +1 (cid:1) 4: y k
Revised: + K ⊤ y k +1 (cid:1) 4: y k

Original: +

Revised: +

Original: =
Revised: =

Original: y k − η y (cid:0) ∂F
Revised: y k − η y (cid:0) ∂F

Original: ⋆ ( y k +1 ) − 
Revised: ⋆ ( y k +1 ) − Kx

Original: Kx k +1 (cid:1) 5: end for[Table] [Figure] [Table]
Revised: k +1 (cid:1) 5: end for

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: B.2 Bonding the distance of the primal iteratesto the primal solution
Revised: B.2 Bonding the distance of the primal iterates

Revised: to the primal solution

Original: In our first lemma, we will provide a bound on ∥ x k +1 − x ⋆ ∥ 2 . 
Revised: In our first lemma, we will provide a bound on ∥ x k +1 − x ⋆ ∥ 2 . [Table] [Table] [Table]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let Assumption 1 hold and choose any η x , η y > 0 .
Revised: Let Assumption 1 hold and choose any η x , η y > 0 .

Original: Then the iterates of the Proximal-Point
Revised: Then the iterates of the Proximal-Point

Original: Method (Algorithm 3) for all k ≥ 0 satisfy
Revised: Method (Algorithm 3) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By writing x k +1 as x k + ( x k +1 − x k ) , and using line 3 of Algorithm 3, which reads x k +1 = x k − η x (cid:0) ∂G ( x k +1 )
Revised: By writing x k +1 as x k + ( x k +1 − x k ) [Equation] and using line 3 of Algorithm [Equation]

Original: {+which reads x k +
 = x k − η x (cid:0) ∂G ( x k +1 )+} + K ⊤ y k +1 (cid:1),
Revised: {+which reads x k +
 = x k − η x (cid:0) ∂G ( x k +1 )+} + K ⊤ y k +1 (cid:1)

Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: We now split the inner product into two parts by applying the optimality condition (see (8))
Revised: We now split the inner product into two parts by applying the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original:  obtaining
Revised: , obtaining

Original: [Equation]
Revised: [Equation]

Original: Finally, this allows us to replace the first inner product using strong convexity of G as follows: 
Revised: Finally [Equation] this allows us to replace the first inner product using strong convexity of G as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B.3 Bonding the distance of the dual iterates to the dual solution
Revised: B.3 Bonding the distance of the dual iterates to the dual solution

Original: In our second lemma, we will provide a bound on ∥
Revised: In our second lemma, we will provide a bound on ∥

Original: y k +1 − y ⋆ ∥ 2 .
Revised: y k +1 − y ⋆ ∥ 2 .

Original: For this, we will rely on an additional assumption (strong convexity of F ⋆ ):
Revised: For this, we will rely on an additional assumption (strong convexity of F ⋆ ):

Original: Assumption 5.
Revised: Assumption 5.

Original: The function F ⋆ : R d y → R is µ y -strongly convex (but can be non-differentiable), i.e.,
Revised: The function F ⋆ : R d y → R is µ y -strongly convex (but can be non-differentiable), i.e.,

Original: [Equation]
Revised: [Equation]

Original: where g F ⋆ ( y ′′ ) ∈ ∂F ⋆ ( y ′′ ) is any subgradient of F ⋆ at y ′′ ∈ R d y .
Revised: where g F ⋆ ( y ′′ ) ∈ ∂F ⋆ ( y ′′ ) is any subgradient of F ⋆ at y ′′ ∈ R d y .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let Assumptions 3 and 5 hold and choose any η x , η y > 0 .
Revised: Let Assumptions 3 and 5 hold and choose any η x , η y > 0 .

Original: Then the iterates of the
Revised: Then the iterates of the

Original: Proximal-Point Method (Algorithm 3) for all k ≥ 0 satisfy
Revised: Proximal-Point Method (Algorithm 3) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By writing y k +1 as y k
Revised: By writing y k +1 as y k

Original: + ( y k +1 − y k ) , and using line 4 of Algorithm 3, which reads y k
Revised: + ( y k +1 − y k ) , and using line 4 of Algorithm 3, which reads y k

Original: +

Revised: +

Original: =
Revised: =

Original: y k − η y (cid:0) ∂F
Revised: y k − η y (cid:0) ∂F

Original: ⋆ ( y k +1 )
Revised: ⋆ ( y k +1 )

Original: − Kx k +1 (cid:1) , we get[Equation] =
Revised: − Kx k +1 (cid:1) , we get

Original: [Equation]
Revised: [Equation]

Original: We now split the inner product into two parts by applying the optimality condition (see (8))
Revised: We now split the inner product into two parts by applying the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original: obtaining
Revised: obtaining

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: this allows us to replace the first inner product using strong convexity of F ⋆ as follows:
Revised: this allows us to replace the first inner product using strong convexity of F ⋆ as follows:

Original: [Equation]
Revised: [Equation]

Original: B.4 Complexity of the Proximal-Point Method
Revised: B.4 Complexity of the Proximal-Point Method

Original: We now formulate the main result describing the iteration complexity of the Proximal-Point Method.
Revised: We now formulate the main result describing the iteration complexity of the Proximal-Point Method.

Original: As we shall see, it follows by combining the above two lemmas.
Revised: As we shall see [Equation] it follows by combining the above two lemmas.

Original: Note that the theorem postulates arbitrarily fast linear convergence , with the speed controlled via the primal and dual stepsizes η x and η y , respectively.
Revised: Note that the theorem postulates arbitrarily fast linear convergence , with the speed controlled via the primal and dual stepsizes η x and η y , respectively.

Original: The larger the stepsizes, the faster the rate becomes.
Revised: The larger the stepsizes, the faster the rate becomes.

Original: In the limit, as η x → + ∞ and η y → + ∞ , the result obtained predicts convergence in a single iteration
Revised: In the limit, as η x → + ∞ and η y → + ∞ , the result obtained predicts convergence in a single iteration.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let Assumptions 1 and 5 hold, and choose any η x , η y > 0 .
Revised: Let Assumptions 1 and 5 hold, and choose any η x , η y > 0 .

Original: Then the iterates of the
Revised: Then the iterates of the

Original: Proximal-Point Method (Algorithm 3) for all k ≥ 0 satisfy
Revised: Proximal-Point Method (Algorithm 3) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: where the Lyapunov function is defined by
Revised: where the Lyapunov function is defined by

Original: [Equation]
Revised: [Equation]

Original: This implies the following statement:
Revised: This implies the following statement:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By adding the inequalities from Lemma 2 and Lemma 3, we obtain
Revised: By adding the inequalities from Lemma 2 and Lemma 3, we obtain

Original: [Equation]
Revised: [Equation]

Original: Thus, if we denote ∆ k def = 1 η x ∥ x k − x ⋆ ∥ 2 + 1 η y ∥
Revised: Thus, if we denote ∆ k def = 1 η x ∥ x k − x ⋆ ∥ 2 + 1 η y ∥

Original: y k − y ⋆ ∥ 2 and m def = min { 1 + 2 µ x η x , 1 + 2 µ y η y } , the above inequality can be written in the following way:
Revised: y k − y ⋆ ∥ 2 and m def = min { 1 + 2 µ x η x , 1 + 2 µ y η y } , the above inequality can be written in the following way:

Original: [Equation]
Revised: [Equation]

Original: Using standard arguments, the above implies that
Revised: Using standard arguments, the above implies that

Original: [Equation]
Revised: [Equation]

Original: C Analysis of the Chambolle-Pock Method
Revised: C Analysis of the Chambolle-Pock Method

Original: In this section we justify the claims we made in Section 3.3 about the Chambolle-Pock Method (Algorithm 4).
Revised: In this section we justify the claims we made in Section 3.3 about the Chambolle-Pock Method (Algorithm 4).

Original: In particular, we provide formal statements and proofs of the complexity results (12) and (13) mentioned in Sections C.4 and C.5, respectively.
Revised: In particular, we provide formal statements and proofs of the complexity results (12) and (13) mentioned in Sections C.4 and C.5, respectively.

Original: C.1 The Chambolle-Pock Method
Revised: C.1 The Chambolle-Pock Method

Original: We have described the Chambolle-Pock Method informally in Section 3.2.
Revised: We have described the Chambolle-Pock Method informally in Section 3.2.

Original: Here we state it formally as Algorithm 4.
Revised: Here we state it formally as Algorithm 4.

Original: and
Revised: and

Original: ] 1: Input : Initial point ( x 0 , y 0 )
Revised: ] 1: Input : Initial point ( x 0 , y 0 )

Original: ∈ R d x × R d y , ¯ y 

Revised: ∈ R d x × R d y , ¯ y

Original: = y 0 ; Stepsizes η x , η y > 0 , Extrapolation parameter θ ∈ [0 , 1] 2: for k = 0 , 1 , . . .
Revised: = y 0 ; Stepsizes η x , η y > 0 , Extrapolation parameter θ ∈ [0 , 1] 2: for k = 0 , 1 , . . .

Original: do 3: x
Revised: do 3: x

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: = x
Revised: = x

Original: [Equation]
Revised: [Equation]

Original: − η
Revised: − η

Original: [Equation]
Revised: [Equation]

Original: (cid:0) ∂G ( x k +1 )
Revised: (cid:0) ∂G ( x k +1 )

Original: + K ⊤ ¯ y k
Revised: + K ⊤ ¯ y k

Original: (cid:1) 4: y k
Revised: (cid:1) 4: y k

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: = y
Revised: = y

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: (cid:0) ∂F
Revised: (cid:0) ∂F

Original: [Equation]
Revised: [Equation]

Original: ( y k +1 )
Revised: ( y k +1 )

Original: − Kx k +1 (cid:1) 5: ¯ y k
Revised: − Kx k +1 (cid:1) 5: ¯ y k

Original: +

Revised: +

Original: =
Revised: =

Original: y k
Revised: y k

Original: +1 + θ (cid:0) y k
Revised: +1 + θ (cid:0) y k

Original: +1 − y k (cid:1) 6: end for
Revised: +1 − y k (cid:1) 6: end for

Original: C.2 Bonding the distance of the primal iterates to the primal solution
Revised: C.2 Bonding the distance of the primal iterates to the primal solution

Original: In our first lemma, we will provide a bound on ∥ x k +1 − x ⋆ ∥ 2 .
Revised: In our first lemma, we will provide a bound on ∥ x k +1 − x ⋆ ∥ 2 .

Original: The result is identical to Lemmawith a single exception: instead of the fresh dual point y k +1 , we now have ¯ y k on the right-hand side.
Revised: The result is identical to Lemmawith a single exception: instead of the fresh dual point y k +1 , we now have ¯ y k on the right-hand side.

Original: This lemma applies both to the θ = 0 and θ > 0 cases.
Revised: This lemma applies both to the θ = 0 and θ > 0 cases.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let Assumption 1 hold and choose any η x > 0 .
Revised: Let Assumption 1 hold and choose any η x > 0 .

Original: Then the iterates of the Chambolle-Pock
Revised: Then the iterates of the Chambolle-Pock

Original: Method (Algorithm 4) for all k ≥ 0 satisfy
Revised: Method (Algorithm 4) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: ⊤ ¯ y k − K ⊤ y ⋆ , x k +1 − x ⋆ ⟩ .
Revised: ⊤ ¯ y k − K ⊤ y ⋆ , x k +1 − x ⋆ ⟩ .

Original: Proof.
Revised: Proof.

Original: Identical to the proof of Lemma 2; one just needs to replace y k +1 by ¯ y k everywhere.
Revised: Identical to the proof of Lemma 2; one just needs to replace y k +1 by ¯ y k everywhere.

Original: C.3 Bonding the distance of the dual iterates to dual solution
Revised: C.3 Bonding the distance of the dual iterates to dual solution

Original: We do not need a new bound on ∥ y k +1 − y ⋆ ∥ 2 for Algorithm 4 since Lemma 3 we proved for the
Revised: We do not need a new bound on ∥ y k +1 − y ⋆ ∥ 2 for Algorithm 4 since Lemma 3 we proved for the

Original: Proximal-Point Method applies here as well.
Revised: Proximal-Point Method applies here as well.

Original: C.4 Formal statement and proof of (12) (Chambolle-Pock in the θ = 0 case)
Revised: C.4 Formal statement and proof of (12) (Chambolle-Pock in the θ = 0 case)

Original: We are now ready to state and prove the main complexity result for the Chambolle-Pock Method in the θ = 0 case.
Revised: We are now ready to state and prove the main complexity result for the Chambolle-Pock Method in the θ = 0 case.

Original: This is the formal version of the informal complexity result (12).
Revised: This is the formal version of the informal complexity result (12).

Original: Theorem 5 (Complexity of the Chambolle-Pock Method in the θ = 0 case) .
Revised: Theorem 5 (Complexity of the Chambolle-Pock Method in the θ = 0 case) .

Original: Let Assumptions 1,hold.
Revised: Let Assumptions 1,hold.

Original: Consider the Chambolle-Pock Method (Algorithm 4) with the extrapolation parameter set as θ = 0 , and the primal and dual stepsizes set as
Revised: Consider the Chambolle-Pock Method (Algorithm 4) with the extrapolation parameter set as θ = 0 , and the primal and dual stepsizes set as

Original: [Equation]
Revised: [Equation]

Original: Then for the Lyapunov function
Revised: Then for the Lyapunov function

Original: [Equation]
Revised: [Equation]

Original: and all k ≥ 

Revised: and all k ≥

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: This implies the following statement:
Revised: This implies the following statement:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By adding the inequalities from Lemma 4 (and noting that ¯ y k
Revised: By adding the inequalities from Lemma 4 (and noting that ¯ y k

Original: =
Revised: =

Original: y k in the θ = 0 case) and
Revised: y k in the θ = 0 case) and

Original: Lemma 3, we obtain
Revised: Lemma 3, we obtain

Original: [Equation]
Revised: [Equation]

Original: where in the last step we have used the bound − 1 η x ∥ x k +1 − x k ∥ 2 ≤ 0 .
Revised: where in the last step we have used the bound − 1 η x ∥ x k +1 − x k ∥ 2 ≤ 0 .

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: the definition of L xy as the norm of K (see (2)), and applying
Revised: the definition of L xy as the norm of K (see (2)), and applying

Original: Young’s inequality, we can bound the inner product by
Revised: Young’s inequality, we can bound the inner product by

Original: [Equation]
Revised: [Equation]

Original: for any C > 0 .
Revised: for any C > 0 .

Original: Plugging this into (32), and rearranging the inequality so that all terms involving ∥ x k +1 − x ⋆ ∥ 2 appear on the left-hand side, we get
Revised: Plugging this into (32), and rearranging the inequality so that all terms involving ∥ x k +1 − x ⋆ ∥ 2 appear on the left-hand side, we get

Original: [Equation]
Revised: [Equation]

Original: Taking C
Revised: Taking C

Original: =
Revised: =

Original: L xy µ x and η y = µ x L 2 xy
Revised: L xy µ x and η y = µ x L 2 xy

Original: [Equation]
Revised: [Equation]

Original: we obtain the simplified bound
Revised: we obtain the simplified bound

Original: [Equation]
Revised: [Equation]

Original: If we let
Revised: If we let

Original: [Equation]
Revised: [Equation]

Original: then inequality (33) implies
Revised: then inequality (33) implies

Original: [Equation]
Revised: [Equation]

Original: Using standard arguments, the above implies that
Revised: Using standard arguments, the above implies that

Original: [Equation]
Revised: [Equation]

Original: C.5 Formal statement and proof of (13) (Chambolle-Pock in the θ > 0 case)
Revised: C.5 Formal statement and proof of (13) (Chambolle-Pock in the θ > 0 case)

Original: We now study the case when the extrapolation parameter θ is set to a positive value
Revised: We now study the case when the extrapolation parameter θ is set to a positive value

Original: [Equation]
Revised: [Equation]

Original: and show that this helps to get better rates.
Revised: and show that this helps to get better rates.

Original: Lemma 5.
Revised: Lemma 5.

Original: Under Assumption
Revised: Under Assumption

Original: [Equation]
Revised: [Equation]

Original: the iterates of the Chambolle-Pock Method (Algorithm 4) with θ > 0 for any C > 0 and all k ≥ 1 satisfy
Revised: the iterates of the Chambolle-Pock Method (Algorithm 4) with θ > 0 for any C > 0 and all k ≥ 1 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using line 5 from Algorithm
Revised: Using line 5 from Algorithm

Original: [Equation]
Revised: [Equation]

Original: which reads ¯ y k
Revised: which reads ¯ y k

Original: = y k
Revised: = y k

Original: + θ (cid:0) y k − y k − 1 (cid:1)
Revised: + θ (cid:0) y k − y k − 1 (cid:1)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: the definition of L xy as the norm of K (see (2)), and applying
Revised: the definition of L xy as the norm of K (see (2)), and applying

Original: Young’s inequality, we can bound the last inner product by
Revised: Young’s inequality, we can bound the last inner product by

Original: [Equation]
Revised: [Equation]

Original: It only remains to plug this inequality into (35).
Revised: It only remains to plug this inequality into (35).

Original: We are now ready to state and prove the general theorem.
Revised: We are now ready to state and prove the general theorem.

Original: Theorem 6 (Complexity of the Chambolle-Pock Method in the θ > 0 case) .
Revised: Theorem 6 (Complexity of the Chambolle-Pock Method in the θ > 0 case) .

Original: Let Assumptions 1,hold.
Revised: Let Assumptions 1,hold.

Original: Consider the Chambolle-Pock Method (Algorithm 4) with the extrapolation parameter set as
Revised: Consider the Chambolle-Pock Method (Algorithm 4) with the extrapolation parameter set as

Original: [Equation]
Revised: [Equation]

Original: and primal and dual stepsizes set as
Revised: and primal and dual stepsizes set as

Original: [Equation]
Revised: [Equation]

Original: Then for the Lyapunov function defined for k ≥ 0 via
Revised: Then for the Lyapunov function defined for k ≥ 0 via

Original: [Equation]
Revised: [Equation]

Original: and for k = 0 via
Revised: and for k = 0 via

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will proceed in several steps.
Revised: We will proceed in several steps.

Original: Showing that ∆ k ≥ 0 for all k ≥ 0 .
Revised: Showing that ∆ k ≥ 0 for all k ≥ 0 .

Original: First, let us show that ∆ k ≥ 0 for every k .
Revised: First, let us show that ∆ k ≥ 0 for every k .

Original: This is clear for k = 0 from (39).
Revised: This is clear for k = 0 from (39).

Original: Let us show that ∆ k +1 ≥ 0 for k ≥ 0 .
Revised: Let us show that ∆ k +1 ≥ 0 for k ≥ 0 .

Original: Using Young’s inequality and (2), we obtain the inequality
Revised: Using Young’s inequality and (2), we obtain the inequality

Original: [Equation]
Revised: [Equation]

Original: which holds for all B > 0 .
Revised: which holds for all B > 0 .

Original: Selecting B = 1 L xy η y , the blue term is zeroed out, and using the primal and dual stepsizes (37), we get
Revised: Selecting B = 1 L xy η y , the blue term is zeroed out, and using the primal and dual stepsizes (37), we get

Original: [Equation]
Revised: [Equation]

Original: Establishing technical bounds.
Revised: Establishing technical bounds.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma
Revised: Combining Lemma

Original: [Equation]
Revised: [Equation]

Original: which provides a bound on ∥ x k +1 − x ⋆ ∥
Revised: which provides a bound on ∥ x k +1 − x ⋆ ∥

Original: [Equation]
Revised: [Equation]

Original: and Lemma
Revised: and Lemma

Original: [Equation]
Revised: [Equation]

Original: which provides a bound on ∥
Revised: which provides a bound on ∥

Original: y k +1 − y ⋆ ∥ 2 , we obtain
Revised: y k +1 − y ⋆ ∥ 2 , we obtain

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This is the same inequality as (31) with the exception that y k was replaced by ¯ y k .
Revised: This is the same inequality as (31) with the exception that y k was replaced by ¯ y k .

Original: Using Lemma 5 to bound the inner product in (43), we get
Revised: Using Lemma 5 to bound the inner product in (43), we get

Original: [Equation]
Revised: [Equation]

Original: where in the last step we have made the choice C def
Revised: where in the last step we have made the choice C def

Original: = ( θη x L xy ) − 1 .
Revised: = ( θη x L xy ) − 1 .

Original: Showing that ∆ k +1 ≤ θ ∆ k for k ≥ 1 .
Revised: Showing that ∆ k +1 ≤ θ ∆ k for k ≥ 1 .

Original: By combining (38) and (42), and applying inequality (44), for k ≥ 1 we get ∆ k
Revised: By combining (38) and (42), and applying inequality (44), for k ≥ 1 we get ∆ k

Original: [Equation]
Revised: [Equation]

Original: where in step (A) we annihilated the red and blue terms as their sum is zero, in step (B) we used the fact that L 2 xy η x η y ≤ 1 , which follows from the stepsize choice, and in (C) we used the inequalities 1 ≤ θ (1 + 2 µ x η x ) , 1 ≤ θ (1 + 2 µ y η y ) , and θ ≤ 1 , which follow from (36).
Revised: where in step (A) we annihilated the red and blue terms as their sum is zero, in step (B) we used the fact that L 2 xy η x η y ≤ 1 , which follows from the stepsize choice, and in (C) we used the inequalities 1 ≤ θ (1 + 2 µ x η x ) , 1 ≤ θ (1 + 2 µ y η y ) , and θ ≤ 1 , which follow from (36).

Original: Showing that ∆ k +1 ≤ θ ∆ k for k = 0 .
Revised: Showing that ∆ k +1 ≤ θ ∆ k for k = 0 .

Original: We start with inequality (43) for k = 0 :
Revised: We start with inequality (43) for k = 0 :

Original: [Equation]
Revised: [Equation]

Original: According to (38) and (36), we have ∆
Revised: According to (38) and (36), we have ∆

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we used the inequalities 1 ≤ θ (1 + 2 µ x η x ) and 1 ≤ θ (1 + 2 µ y η y ) , which follow from (36).
Revised: where in the last inequality we used the inequalities 1 ≤ θ (1 + 2 µ x η x ) and 1 ≤ θ (1 + 2 µ y η y ) , which follow from (36).

Original: The informal result (13) mentioned in Section C.5 is a simple corollary of the above theorem.
Revised: The informal result (13) mentioned in Section C.5 is a simple corollary of the above theorem.

Original: Indeed, using the definition of θ , we can obtain:
Revised: Indeed, using the definition of θ , we can obtain:

Original: [Equation]
Revised: [Equation]

Original: It remains to remark that
Revised: It remains to remark that

Original: [Equation]
Revised: [Equation]

Original: which is the result from (13).
Revised: which is the result from (13).

Original: D Analysis of the Accelerated Primal-Dual Algorithm (APDA; Algorithm 1)
Revised: D Analysis of the Accelerated Primal-Dual Algorithm (APDA; Algorithm 1)

Original: In this section we perform convergence analysis for our new method APDA (Algorithm 1).
Revised: In this section we perform convergence analysis for our new method APDA (Algorithm 1).

Original: We start by establishing three lemmas, followed by the proof of the main theorem.
Revised: We start by establishing three lemmas, followed by the proof of the main theorem.

Original: D.1 Three lemmas
Revised: D.1 Three lemmas

Original: The first result is a variant of Lemma 4, offering a bound on the distance of the primal iterates from the primal optimal solution.
Revised: The first result is a variant of Lemma 4, offering a bound on the distance of the primal iterates from the primal optimal solution.

Original: Compared to Lemma 4, this result is strengthened by the additional assumption of L x -smoothness of G (Assumption 2).
Revised: Compared to Lemma 4, this result is strengthened by the additional assumption of L x -smoothness of G (Assumption 2).

Original: Lemma 6.
Revised: Lemma 6.

Original: Let Assumptions 1 and 2 hold and choose any η x , η y > 0 .
Revised: Let Assumptions 1 and 2 hold and choose any η x , η y > 0 .

Original: Then the iterates of APDA (Algorithm 1) for all k ≥ 0 satisfy
Revised: Then the iterates of APDA (Algorithm 1) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By writing x k +1 as x k + ( x k +1 − x k ) , and using line 3 of Algorithm 1, which reads x k +1 = x k − η x (cid:0) ∇ G ( x k +1 ) + K ⊤ ¯ y k (cid:1) , we get
Revised: By writing x k +1 as x k + ( x k +1 − x k ) , and using line 3 of Algorithm 1, which reads x k +1 = x k − η x (cid:0) ∇ G ( x k +1 ) + K ⊤ ¯ y k (cid:1) , we get

Original: [Equation]
Revised: [Equation]

Original: We now split the inner product into two parts by applying the optimality condition (see (8))
Revised: We now split the inner product into two parts by applying the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original: obtaining
Revised: obtaining

Original:  η x ∥ x k +1 − x ⋆ ∥
Revised:  η x ∥ x k +1 − x ⋆ ∥

Original: [Equation]
Revised: [Equation]

Original: Since G is µ x -strongly-convex and L x -smooth, we can lower-bound [Nesterov, 2004] the inner product appearing in (47) as follows:
Revised: Since G is µ x -strongly-convex and L x -smooth, we can lower-bound [Nesterov, 2004] the inner product appearing in (47) as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, by plugging (48) into (47), we get
Revised: Finally, by plugging (48) into (47), we get

Original: [Equation]
Revised: [Equation]

Original: The second result is a technical lemma borrowed from [Kovalev et al., 2021a] for lower-bounding the term ∥ K ⊤
Revised: The second result is a technical lemma borrowed from [Kovalev et al., 2021a] for lower-bounding the term ∥ K ⊤

Original: y − K ⊤
Revised: y − K ⊤

Original: y ⋆ ∥ .
Revised: y ⋆ ∥ .

Original: Lemma 7 (See Kovalev et al.
Revised: Lemma 7 (See Kovalev et al.

Original: [2021a]) .
Revised: [2021a]) .

Original: Let Assumption 4 hold and let ( x ⋆ , y ⋆ ) be a solution of (7) .
Revised: Let Assumption 4 hold and let ( x ⋆ , y ⋆ ) be a solution of (7) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The third result offers a bound on the distance between the dual iterates and the dual optimal solution.
Revised: The third result offers a bound on the distance between the dual iterates and the dual optimal solution.

Original: When analyzing the Proximal Point method and the Chambolle-Pock method, in this step we relied on
Revised: When analyzing the Proximal Point method and the Chambolle-Pock method, in this step we relied on

Original: Lemma 3, in the proof of which we required F ⋆ to be µ x -strongly convex.
Revised: Lemma 3, in the proof of which we required F ⋆ to be µ x -strongly convex.

Original: However, for APDA we explicitly wish to avoid using this assumption.
Revised: However, for APDA we explicitly wish to avoid using this assumption.

Original: As we shall see, in order to obtain linear convergence, it is enough will invoke Assumption 4.
Revised: As we shall see, in order to obtain linear convergence, it is enough will invoke Assumption 4.

Original: The next lemma is an analogue of Lemma 3 without the need to assume µ x -strong-convexity of F ⋆ .
Revised: The next lemma is an analogue of Lemma 3 without the need to assume µ x -strong-convexity of F ⋆ .

Original: Lemma 8.
Revised: Lemma 8.

Original: Let Assumptions 2, 3, and 4 be satisfied, and choose
Revised: Let Assumptions 2, 3, and 4 be satisfied, and choose

Original: [Equation]
Revised: [Equation]

Original: Then the iterates of APDA (Algorithm 1) for all k ≥ 0 satisfy
Revised: Then the iterates of APDA (Algorithm 1) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By writing y k +1 as y k
Revised: By writing y k +1 as y k

Original: + ( y k +1 − y k )
Revised: + ( y k +1 − y k )

Original: [Equation]
Revised: [Equation]

Original: and using line 4 from Algorithm
Revised: and using line 4 from Algorithm

Original: [Equation]
Revised: [Equation]

Original: which reads
Revised: which reads

Original: [Equation]
Revised: [Equation]

Original: we
Revised: we

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We split the inner product appearing in A k into two parts by applying the optimality condition (see (8))
Revised: We split the inner product appearing in A k into two parts by applying the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original: obtaining
Revised: obtaining

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Applying the parallelogram identity 14 to B k
Revised: Applying the parallelogram identity 14 to B k

Original: [Equation]
Revised: [Equation]

Original: and using the optimality condition (see (8))
Revised: and using the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Plugging (51) and (53) back into (50), and using convexity of F ⋆ , we get
Revised: Plugging (51) and (53) back into (50), and using convexity of F ⋆ , we get

Original: [Equation]
Revised: [Equation]

Original: It now only remains to plug the bounds ∥ K ⊤ y k − K ⊤ y ⋆ ∥ 2 ≥ µ 2 xy ∥
Revised: It now only remains to plug the bounds ∥ K ⊤ y k − K ⊤ y ⋆ ∥ 2 ≥ µ 2 xy ∥

Original: y k − y ⋆ ∥ 2 (see Lemma 7) and ∥
Revised: y k − y ⋆ ∥ 2 (see Lemma 7) and ∥

Original: K ⊤ y k − K ⊤
Revised: K ⊤ y k − K ⊤

Original: y k +1 ∥ 2 ≤ L 2 xy ∥
Revised: y k +1 ∥ 2 ≤ L 2 xy ∥

Original: y k − y k +1 ∥ 2 into (54)
Revised: y k − y k +1 ∥ 2 into (54)

Original: [Equation]
Revised: [Equation]

Original: and apply the restriction (49) on β y .
Revised: and apply the restriction (49) on β y .

Original: D.2 Main result
Revised: D.2 Main result

Original: We are now ready to state the formal version of Theorem 1.
Revised: We are now ready to state the formal version of Theorem 1.

Original: Theorem 7 (Covergence of APDA; formal) .
Revised: Theorem 7 (Covergence of APDA; formal) .

Original: Let Assumptions 1, 2, 3, and 4 hold and choose the various parameters of the method as follows:
Revised: Let Assumptions 1, 2, 3, and 4 hold and choose the various parameters of the method as follows:

Original: [Equation]
Revised: [Equation]

Original: Then for the Lyapunov function defined for k ≥ 0 via
Revised: Then for the Lyapunov function defined for k ≥ 0 via

Original: [Equation]
Revised: [Equation]

Original: and for k = 0 via
Revised: and for k = 0 via

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: We will proceed in several steps
Revised: We will proceed in several steps

Original: [Equation]
Revised: [Equation]

Original: Showing that Ξ k ≥ 0 for all k ≥ 0 .
Revised: Showing that Ξ k ≥ 0 for all k ≥ 0 .

Original: First, we will show that Ξ k ≥ 0 for every k .
Revised: First, we will show that Ξ k ≥ 0 for every k .

Original: This is clear for k = 0 .
Revised: This is clear for k = 0 .

Original: Let us show that Ξ k +1 ≥ 0 for every k ≥ 0 .
Revised: Let us show that Ξ k +1 ≥ 0 for every k ≥ 0 .

Original: Using Young’s inequality with any parameter
Revised: Using Young’s inequality with any parameter

Original: B > 0 , and the definition of L xy from (2), we obtain Ξ k +
Revised: B > 0 , and the definition of L xy from (2), we obtain Ξ k +

Original: [Equation]
Revised: [Equation]

Original: Using (61) with B = 1 

Revised: Using (61) with B = 1

Original: L xy η y , and noticing that 2 L 2 xy η x η y = 

Revised: L xy η y , and noticing that 2 L 2 xy η x η y =

Original: (this follows from (56) and (57)), we get
Revised: (this follows from (56) and (57)), we get

Original: [Equation]
Revised: [Equation]

Original: Establishing technical bounds.
Revised: Establishing technical bounds.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: By adding the inequalities from Lemmas 6 and
Revised: By adding the inequalities from Lemmas 6 and

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: where in the last step we used the bound
Revised: where in the last step we used the bound

Original: [Equation]
Revised: [Equation]

Original: which follows from the restriction β y ≤ 1 L x ; see (55).
Revised: which follows from the restriction β y ≤ 1 L x ; see (55).

Original: Using line 5 from Algorithm
Revised: Using line 5 from Algorithm

Original: [Equation]
Revised: [Equation]

Original: which says
Revised: which says

Original: [Equation]
Revised: [Equation]

Original: applying Cauchy-Schwarz inequality, and subsequently using Young’s inequality with constant
Revised: applying Cauchy-Schwarz inequality, and subsequently using Young’s inequality with constant

Original: C > 0 , the inner product from (63) can for k ≥ 1 be further bounded as follows
Revised: C > 0 , the inner product from (63) can for k ≥ 1 be further bounded as follows

Original: [Equation]
Revised: [Equation]

Original: Showing that Ξ k +1 ≤ θ Ξ k for all k ≥ 1 .
Revised: Showing that Ξ k +1 ≤ θ Ξ k for all k ≥ 1 .

Original: Plugging the bounds (62) and (65) into the Lyapunov function (59), for any k ≥ 1 we get Ξ k +
Revised: Plugging the bounds (62) and (65) into the Lyapunov function (59), for any k ≥ 1 we get Ξ k +

Original: [Equation]
Revised: [Equation]

Original: Showing that Ξ k +1 ≤ θ Ξ k for k = 0 .
Revised: Showing that Ξ k +1 ≤ θ Ξ k for k = 0 .

Original: This can be done using similar arguments as those used in the proof of Theorem
Revised: This can be done using similar arguments as those used in the proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: D.3 Proof of Theorem 1 (informal)
Revised: D.3 Proof of Theorem 1 (informal)

Original: We now provide the iteration complexity of Algorithm 1 as a corollary of Theorem 7.
Revised: We now provide the iteration complexity of Algorithm 1 as a corollary of Theorem 7.

Original: Note that in view of (60), we get k ≥ O
Revised: Note that in view of (60), we get k ≥ O

Original: (cid:18)1 − θ log ε (cid:19) ⇒ Ξ k ≤ ε Ξ 0 .
Revised: (cid:18)1 − θ log ε (cid:19) ⇒ Ξ k ≤ ε Ξ 0 .

Original: Using the definition of θ given in (58), we haveO
Revised: Using the definition of θ given in (58), we have

Original: [Equation]
Revised: [Equation]

Original: This proves the statement of Theorem 1 mentioned in Section 4 in the main body of the paper.
Revised: This proves the statement of Theorem 1 mentioned in Section 4 in the main body of the paper.

Original: E Analysis
Revised: E Analysis

Original: with Inexact Prox (Algorithm 2)
Revised: with Inexact Prox (Algorithm 2)

Original: In this section we provide convergence analysis for our second new method, APDA with Inexact Prox
Revised: In this section we provide convergence analysis for our second new method, APDA with Inexact Prox

Original: (Algorithm 2).
Revised: (Algorithm 2).

Original: We start with statements of three lemmas, followed by the proof of the main theorem.
Revised: We start with statements of three lemmas, followed by the proof of the main theorem.

Original: E.1 Three Lemmas
Revised: E.1 Three Lemmas

Original: The first result is essentially a modification of Lemma 6, providing a bound on the distance of the primal iterates from the primal optimal solution.
Revised: The first result is essentially a modification of Lemma 6, providing a bound on the distance of the primal iterates from the primal optimal solution.

Original: Compare to Lemma 6, these changes consist in using the definition of function Ψ k
Revised: Compare to Lemma 6, these changes consist in using the definition of function Ψ k

Original: (see the problem (15)) to prove this key fact.
Revised: (see the problem (15)) to prove this key fact.

Original: Lemma 9.
Revised: Lemma 9.

Original: Let w ⋆k be an exact solution to the problem (15) .
Revised: Let w ⋆k be an exact solution to the problem (15) .

Original: Then under Assumptions 1, 2, we have
Revised: Then under Assumptions 1, 2, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In view of line 4 of Algorithm
Revised: In view of line 4 of Algorithm

Original: [Equation]
Revised: [Equation]

Original: which reads
Revised: which reads

Original: [Equation]
Revised: [Equation]

Original: and writing x k +1 as x k + ( x k +1 − x k ) , we get
Revised: and writing x k +1 as x k + ( x k +1 − x k ) , we get

Original: [Equation]
Revised: [Equation]

Original: Using the identity − 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a + b ∥ 2 to rewrite the second inner product from the previous equation
Revised: Using the identity − 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a + b ∥ 2 to rewrite the second inner product from the previous equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: ) where in the last equation we applied the optimality condition (see (8))
Revised: ) where in the last equation we applied the optimality condition (see (8))

Original: [Equation]
Revised: [Equation]

Original: Due to the fact that G is µ x -strongly-convex and L x -smooth, we can estimate the inner product 2 ⟨∇ G (ˆ x k ) − ∇ G ( x ⋆ ) , ˆ x k − x ⋆ ⟩ from below similarly as in the proof of Lemma 6:
Revised: Due to the fact that G is µ x -strongly-convex and L x -smooth, we can estimate the inner product 2 ⟨∇ G (ˆ x k ) − ∇ G ( x ⋆ ) , ˆ x k − x ⋆ ⟩ from below similarly as in the proof of Lemma 6:

Original: [Equation]
Revised: [Equation]

Original: Now, by plugging (68) into (67), we obtain
Revised: Now, by plugging (68) into (67), we obtain

Original: [Equation]
Revised: [Equation]

Original: Applying the inequality ∥ a − b ∥ 2 ≥
Revised: Applying the inequality ∥ a − b ∥ 2 ≥

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: c ∥ 2 to ∥ ˆ x k − x ⋆ ∥ 2 and taking c = x k +1 , we obtain
Revised: c ∥ 2 to ∥ ˆ x k − x ⋆ ∥ 2 and taking c = x k +1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we used the bound ∥ a − b ∥ 2 ≥ 12 ∥ a − c ∥ 2 − ∥
Revised: where in the last inequality we used the bound ∥ a − b ∥ 2 ≥ 12 ∥ a − c ∥ 2 − ∥

Original: b − c ∥ 2 to estimate 1 η x ∥ x k − ˆ x k ∥ 2 .
Revised: b − c ∥ 2 to estimate 1 η x ∥ x k − ˆ x k ∥ 2 .

Original: From line 3 of Algorithm 2, according to the definition of function Ψ k ( x ) , we get ∇ Ψ k (ˆ x k ) =
Revised: From line 3 of Algorithm 2, according to the definition of function Ψ k ( x ) , we get ∇ Ψ k (ˆ x k ) =

Original: [Equation]
Revised: [Equation]

Original: Finally, substituting (70) into (69), we gain
Revised: Finally, substituting (70) into (69), we gain

Original: [Equation]
Revised: [Equation]

Original: where we also use 1 η x -strong convexity of function Ψ k ( x ) (see problem (15)).
Revised: where we also use 1 η x -strong convexity of function Ψ k ( x ) (see problem (15)).

Original: The second result offers a bound on the distance between the dual itarates and the dual optimal solution as Lemma 8.
Revised: The second result offers a bound on the distance between the dual itarates and the dual optimal solution as Lemma 8.

Original: Since the line 4 of Algorithm 1 coincides with the line 5 of Algorithm 2, the statement and proof of the following lemma coincides with Lemma 8 and its proof with the only change of replacing x k +1 with ˆ x k .
Revised: Since the line 4 of Algorithm 1 coincides with the line 5 of Algorithm 2, the statement and proof of the following lemma coincides with Lemma 8 and its proof with the only change of replacing x k +1 with ˆ x k .

Original: Lemma 10.
Revised: Lemma 10.

Original: Then under Assumptions 2, 3, and 4 be satisfied
Revised: Then under Assumptions 2, 3, and 4 be satisfied

Original: [Equation]
Revised: [Equation]

Original: and choose
Revised: and choose

Original: [Equation]
Revised: [Equation]

Original: Then the iterates of APDA with Inexact Prox
Revised: Then the iterates of APDA with Inexact Prox

Original: (Algorithm 2) for all k ≥ 0 satisfy
Revised: (Algorithm 2) for all k ≥ 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is identical to the proof of Lemma 8, with the sole distinction that x k +1 should be replaced everywhere by ˆ x k .
Revised: The proof is identical to the proof of Lemma 8, with the sole distinction that x k +1 should be replaced everywhere by ˆ x k .

Original: The third result is a technical lemma, which offers a bound on an inner product .
Revised: The third result is a technical lemma, which offers a bound on an inner product .

Original: Lemma 11.
Revised: Lemma 11.

Original: Let w ⋆k be an exact solution to the problem (15) .
Revised: Let w ⋆k be an exact solution to the problem (15) .

Original: Then the following inequality holds
Revised: Then the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: the definition of L xy as the norm of K (see (2))
Revised: the definition of L xy as the norm of K (see (2))

Original: [Equation]
Revised: [Equation]

Original: and applying Young’s inequality
Revised: and applying Young’s inequality

Original: [Equation]
Revised: [Equation]

Original: we can bound the inner product by
Revised: we can bound the inner product by

Original: [Equation]
Revised: [Equation]

Original: for any C > 0 .
Revised: for any C > 0 .

Original: Taking C =
Revised: Taking C =

Original: L xy η x
Revised: L xy η x

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where in the last equation we use (70) for k − 1 , which reads ∇ Ψ k (ˆ x k − 1 )
Revised: where in the last equation we use (70) for k − 1 , which reads ∇ Ψ k (ˆ x k − 1 )

Original: = 1 η x (cid:0) ˆ x k − 1 − x k (cid:1) .
Revised: = 1 η x (cid:0) ˆ x k − 1 − x k (cid:1) .

Original: E.2 Detailed theorem
Revised: E.2 Detailed theorem

Original: Theorem 8 (Convergence of APDA with Inexact Prox; formal) .
Revised: Theorem 8 (Convergence of APDA with Inexact Prox; formal) .

Original: Let Assumptions 1, 2, 3 and 4 hold and select the various parameters of the method as follows:
Revised: Let Assumptions 1, 2, 3 and 4 hold and select the various parameters of the method as follows:

Original: [Equation]
Revised: [Equation]

Original: Then for the Lyapunov function defined for k ≥ 0 via
Revised: Then for the Lyapunov function defined for k ≥ 0 via

Original: [Equation]
Revised: [Equation]

Original: and for k = 0 via
Revised: and for k = 0 via

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We denote V k +1 as follows:
Revised: We denote V k +1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Combining Lemmas 9, 10 and (72), we obtain
Revised: Combining Lemmas 9, 10 and (72), we obtain

Original: V k
Revised: V k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where in last inequality the line 6 from Algorithm 2 is used.
Revised: where in last inequality the line 6 from Algorithm 2 is used.

Original: Next, according to Lemma 11, we gain
Revised: Next, according to Lemma 11, we gain

Original: V k +
Revised: V k +

Original: [Equation]
Revised: [Equation]

Original: Using definition of function Ψ k and its 1 η x -strong convexity in the following form
Revised: Using definition of function Ψ k and its 1 η x -strong convexity in the following form

Original: [Equation]
Revised: [Equation]

Original: and assuming that 32 L 2 xy η x η y ≤ 1 and 0 < θ ≤ 1 , we get
Revised: and assuming that 32 L 2 xy η x η y ≤ 1 and 0 < θ ≤ 1 , we get

Original: V k +
Revised: V k +

Original: [Equation]
Revised: [Equation]

Original: (1 − µ 2 xy β y η y ) 1 η y ∥
Revised: (1 − µ 2 xy β y η y ) 1 η y ∥

Original: y k − y ⋆ ∥
Revised: y k − y ⋆ ∥

Original: 2 + θ 2 η y ∥
Revised: 2 + θ 2 η y ∥

Original: y k − y k − 1 ∥+(2 η x + µ x η 2 x ) ∥∇ Ψ k
Revised: y k − y k − 1 ∥+(2 η x + µ x η 2 x ) ∥∇ Ψ k

Original: (ˆ x k ) ∥ 2 − 1 4 η x ∥ x k − w ⋆k ∥ 2 + θη x 4 ∥∇ Ψ k
Revised: (ˆ x k ) ∥ 2 − 1 4 η x ∥ x k − w ⋆k ∥ 2 + θη x 4 ∥∇ Ψ k

Original: (ˆ x k ) ∥ 

Revised: (ˆ x k ) ∥

Original: + θη x 8 ∥∇ Ψ k − 1 η x (ˆ x k − 1 ) ∥+2 ⟨ K ⊤ y k +

Revised: + θη x 8 ∥∇ Ψ k − 1 η x (ˆ x k − 1 ) ∥+2 ⟨ K ⊤ y k +

Original: − K ⊤ y k , ˆ x k − x ⋆ ⟩ − 2 θ ⟨ K ⊤ y k − K ⊤ y k − 1 , ˆ x k − 1 − x ⋆ ⟩ 0 <θ 
Revised: − K ⊤ y k , ˆ x k − x ⋆ ⟩ − 2 θ ⟨ K ⊤ y k − K ⊤ y k − 1 , ˆ x k − 1 − x ⋆ ⟩ 0 <θ ≤ 1 ≤ η x

Original: [Equation]
Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original: ⊤ y k
Revised: x ⋆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⊤ y k

Original: [Equation]
Revised: [Equation]

Original: ˆ x k
Revised: y η

Original: [Equation]
Revised: [Equation]

Original: x ⋆ ⟩

Original: [Equation]
Revised: [Equation]

Original: 2 θ ⟨ K ⊤ y k
Revised: y [Equation] +

Original: [Equation]
Revised: [Equation]

Original: K ⊤ y k
Revised: η

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: η x

Original: [Equation]
Revised: [Equation]

Original:  k
Revised: +2 ⟨ K ⊤ y k + [Equation] K ⊤ y k , ˆ x k − x ⋆ ⟩ − 2 θ ⟨ K ⊤ y k − K ⊤ y k − 1 , ˆ x k

Original: [Equation]
Revised: [Equation]

Revised: 1 − x ⋆ ⟩ .

Original: To estimate ∥∇ Ψ k
Revised: To estimate ∥∇ Ψ k

Original: (ˆ x k ) ∥ 2 we apply Lemma 17 for the problem (15) as follows (see (18)):
Revised: (ˆ x k ) ∥ 2 we apply Lemma 17 for the problem (15) as follows (see (18)):

Original: [Equation]
Revised: [Equation]

Original: According to (18), (73) and (74), we obtain
Revised: According to (18), (73) and (74), we obtain

Original: [Equation]
Revised: [Equation]

Original: k +
Revised: k +

Original: [Equation]
Revised: [Equation]

Original: Showing that ∆ k +1 ≤ θ ∆ k for k ≥ 1 .
Revised: Showing that ∆ k +1 ≤ θ ∆ k for k ≥ 1 .

Original: Using the definition of ∆ k +1 (see (77)), we have ∆ k +
Revised: Using the definition of ∆ k +1 (see (77)), we have ∆ k +

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we take θ = max (cid:110)
Revised: where in the last inequality we take θ = max (cid:110)

Original: [Equation]
Revised: [Equation]

Original: see (76)).
Revised: see (76)).

Original: Showing that ∆ k +1 ≤ θ ∆ k for k = 0 .
Revised: Showing that ∆ k +1 ≤ θ ∆ k for k = 0 .

Original: We start with inequality (81) for k = 0 :
Revised: We start with inequality (81) for k = 0 :

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we take β y ≤ 1 L x
Revised: where in the last inequality we take β y ≤ 1 L x

Original: (see (72)).
Revised: (see (72)).

Original: Now, we apply Lemma 1 to the problem (15) for k = 

Revised: Now, we apply Lemma 1 to the problem (15) for k =

Original: (see (18)):
Revised: (see (18)):

Original: [Equation]
Revised: [Equation]

Original: Substituting (83) into (82), we can get
Revised: Substituting (83) into (82), we can get

Original: V≤
Revised: V≤

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we take T according to (73).
Revised: where in the last inequality we take T according to (73).

Original: According to (77) and (78), we have
Revised: According to (77) and (78), we have

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we used the inequalities 1 ≤ θ (cid:0) 1 + µ x η x 2 (cid:1) and 1 − µ 2 xy β y η y ≤ θ , which follow from (76).
Revised: where in the last inequality we used the inequalities 1 ≤ θ (cid:0) 1 + µ x η x 2 (cid:1) and 1 − µ 2 xy β y η y ≤ θ , which follow from (76).

Original: Showing that ∆ k +1 ≥ 0 for k ≥ 0 .
Revised: Showing that ∆ k +1 ≥ 0 for k ≥ 0 .

Original: Finally, we need to show that ∆ k ≥ 0 for every k .
Revised: Finally, we need to show that ∆ k ≥ 0 for every k .

Original: This is obvious for k = 0 from .
Revised: This is obvious for k = 0 from .

Original: Using the Cauchy-Schwarz inequality, the definition of L xy as the norm of
Revised: Using the Cauchy-Schwarz inequality, the definition of L xy as the norm of

Original: K (see (2)), and applying Young’s inequality, we get ∆ k
Revised: K (see (2)), and applying Young’s inequality, we get ∆ k

Original: + (77) =
Revised: + (77) =

Original: [Equation]
Revised: [Equation]

Original: for any B > 0 .
Revised: for any B > 0 .

Original: According to the definition of function Ψ k (see 15), selecting B = 1 2 L xy η y , we get ∆ k + ≥ (cid:16)
Revised: According to the definition of function Ψ k (see 15), selecting B = 1 2 L xy η y , we get ∆ k + ≥ (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Choosing stepsizes η x , η y according to (74) and (75), we can derive the following inequality:
Revised: Choosing stepsizes η x , η y according to (74) and (75), we can derive the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Now, plugging (86) and (18) into (85), we obtain
Revised: Now, plugging (86) and (18) into (85), we obtain

Original: [Equation]
Revised: [Equation]

Original: where in last inequality we use (73) to evaluate two last terms from below by zero.
Revised: where in last inequality we use (73) to evaluate two last terms from below by zero.

Original: E.3 Proof of Theorem
Revised: E.3 Proof of Theorem

Original: To prove the informal result (19)
Revised: To prove the informal result (19)

Original: [Equation]
Revised: [Equation]

Original: we simply apply the above theorem.
Revised: we simply apply the above theorem.

Original: Using the definition of θ , the theorem implies that
Revised: Using the definition of θ , the theorem implies that

Original: [Equation]
Revised: [Equation]

Original: It remains to remark that the number of outer iterations to find ε -solution is equal to
Revised: It remains to remark that the number of outer iterations to find ε -solution is equal to

Original: [Equation]
Revised: [Equation]

Original: L 2 xy µ 2 xy (cid:41) logε (cid:33) =
Revised: L 2 xy µ 2 xy (cid:41) logε (cid:33) =

Original: O (cid:32) max (cid:40) 

Revised: O (cid:32) max (cid:40)

Original: + (cid:115)
Revised: + (cid:115)

Original: L x µ x
Revised: L x µ x

Original: L xy µ xy ,
Revised: L xy µ xy ,

Original: L 2 xy µ 2 xy (cid:41) logε (cid:33) .
Revised: L 2 xy µ 2 xy (cid:41) logε (cid:33) .

Original: In other words, the number of computations of prox is equal to ♯ prox =
Revised: In other words, the number of computations of prox is equal to ♯ prox =

Original: O (cid:32) max (cid:40) 

Revised: O (cid:32) max (cid:40)

Original: + (cid:115)
Revised: + (cid:115)

Original: L x µ x
Revised: L x µ x

Original: L xy µ xy ,
Revised: L xy µ xy ,

Original: L 2 xy µ 2 xy (cid:41) logε (cid:33) , and the number of gradient evaluations is ♯ ∇ G = k · T = O (cid:32) max (cid:40) 

Revised: L 2 xy µ 2 xy (cid:41) logε (cid:33) , and the number of gradient evaluations is ♯ ∇ G = k · T = O (cid:32) max (cid:40)

Original: + (cid:115)
Revised: + (cid:115)

Original: L x µ x
Revised: L x µ x

Original: L xy µ xy ,
Revised: L xy µ xy ,

Original: L 2 xy µ 2 xy (cid:41) logε (cid:33) · O (cid:32)(cid:18)
Revised: L 2 xy µ 2 xy (cid:41) logε (cid:33) · O (cid:32)(cid:18)

Original: L x µ x (cid:19) 1 / α (cid:33) =
Revised: L x µ x (cid:19) 1 / α (cid:33) =

Original: O (cid:32) max (cid:40)(cid:18)
Revised: O (cid:32) max (cid:40)(cid:18)

Original: L x µ x (cid:19) 1 / α + (cid:18)
Revised: L x µ x (cid:19) 1 / α + (cid:18)

Original: L x µ x (cid:19) 2+ α 2 α L xy µ xy , (cid:18)
Revised: L x µ x (cid:19) 2+ α 2 α L xy µ xy , (cid:18)

Original: L x µ x (cid:19) 1 / α L 2 xy µ 2 xy (cid:41) logε (cid:33) .
Revised: L x µ x (cid:19) 1 / α L 2 xy µ 2 xy (cid:41) logε (cid:33) .

Original: 1: Input : Initial point ( x 0 , y 0 )
Revised: 1: Input : Initial point ( x 0 , y 0 )

Original: ∈ R d x × R d y , ¯ y 

Revised: ∈ R d x × R d y , ¯ y

Original: = y 0 ; Step sizes η x , η y , β y > 0 , θ ∈ [0 , 1] ;
Revised: = y 0 ; Step sizes η x , η y , β y > 0 , θ ∈ [0 , 1] ;

Original: Number of inner iterations T ; Number of iterations of Accelerated Gossip N ; 2: for k = 0 , 1 , 2 , . . .
Revised: Number of inner iterations T ; Number of iterations of Accelerated Gossip N ; 2: for k = 0 , 1 , 2 , . . .

Original: do 3:
Revised: do 3:

Original: Find ˆ x k as a final point of T iteration of method M for following problem:
Revised: Find ˆ x k as a final point of T iteration of method M for following problem:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: 6: ¯ y k
Revised: 6: ¯ y k

Original: +

Revised: +

Original: =
Revised: =

Original: y k
Revised: y k

Original: +1 + θ (cid:0) y k
Revised: +1 + θ (cid:0) y k

Original: +1 − y k (cid:1) 7: end for 8: procedure AG( W, x , N ) =
Revised: +1 − y k (cid:1) 7: end for 8: procedure AG( W, x , N ) =

Original: Accelerated Gossip 9:
Revised: Accelerated Gossip 9:

Original: Set a 0 = 1 , a 1 = c 2 , x 0 = x , x
Revised: Set a 0 = 1 , a 1 = c 2 , x 0 = x , x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: ( I − c
Revised: ( I − c

Original: [Equation]
Revised: [Equation]

Original: W ) x 10: for i = 1 , . . .
Revised: W ) x 10: for i = 1 , . . .

Original: , N − 1 do 11: a
Revised: , N − 1 do 11: a

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( I − c
Revised: ( I − c

Original: [Equation]
Revised: [Equation]

Original: W ) x i − x
Revised: W ) x i − x

Original: [Equation]
Revised: [Equation]

Original: 12: end for 13: return x − x N a N 14: end procedure
Revised: 12: end for 13: return x − x N a N 14: end procedure

Original: F Analysis of the Accelerated Primal-Dual Algorithm with Inexact Prox and Accelerated Gossip (Algorithm 5) F.1 Gossip matrices
Revised: F Analysis of the Accelerated Primal-Dual Algorithm with Inexact Prox and Accelerated Gossip (Algorithm 5) F.1 Gossip matrices

Original: In Algorithm 5 we work with a
Revised: In Algorithm 5 we work with a

Original: more
Revised: more

Original: general
Revised: general

Original: notionof a
Revised: notion

Original:  gossip matrix (beyondgraph
Revised: of a gossip matrix (beyond

Original:  Laplacians
Revised: graph Laplacians

Original: ), defined next.
Revised: ), defined next.

Original: Assumption 6 (see [Scaman et al., 2017]) .
Revised: Assumption 6 (see [Scaman et al., 2017]) .

Original: Let G =
Revised: Let G =

Original: ( V , E ) be a connected communication network.
Revised: ( V , E ) be a connected communication network.

Original: The communication process is represented via multiplication by a matrix ˆ W ∈ R n × n which satisfies the following conditions:
Revised: The communication process is represented via multiplication by a matrix ˆ W ∈ R n × n which satisfies the following conditions:

Original: • ˆ W is symmetric, • ˆ W is positive semi-definite, • the kernel of ˆ W satisfies Ker ˆ W = span { (1 , . . .
Revised: • ˆ W is symmetric, • ˆ W is positive semi-definite, • the kernel of ˆ W satisfies Ker ˆ W = span { (1 , . . .

Original: ⊤ ∈ R n } , and • ˆ W i,j ̸ = 

Revised: ⊤ ∈ R n } , and • ˆ W i,j ̸ =

Original: if and only if i = j or ( i, j ) ∈ E .
Revised: if and only if i = j or ( i, j ) ∈ E .

Original: It is easy to see that σ ( ˆ W ) ⊂
Revised: It is easy to see that σ ( ˆ W ) ⊂

Original: σ ( W ) , where σ ( · ) denotes the spectrum, and
Revised: σ ( W ) , where σ ( · ) denotes the spectrum, and

Original: W =
Revised: W =

Original: ˆ W ⊗ I dn .
Revised: ˆ W ⊗ I dn .

Original: F.2 Theorem 3 follows from Theorem
Revised: F.2 Theorem 3 follows from Theorem

Original: Our main result for Algorithm 5, i.e., Theorem 3, follows from Theorem 2.
Revised: Our main result for Algorithm 5, i.e., Theorem 3, follows from Theorem 2.

Original: Below we state it more formally.
Revised: Below we state it more formally.

Original: Theorem 9 (Formal version of Theorem 3; convergence for Algorithm 5) .
Revised: Theorem 9 (Formal version of Theorem 3; convergence for Algorithm 5) .

Original: Let us invoke the same assumptions as those made in Theorem 2.
Revised: Let us invoke the same assumptions as those made in Theorem 2.

Original: Additionally, let Assumption 6 hold.
Revised: Additionally, let Assumption 6 hold.

Original: Assume that the auxiliary problem (87) is solved by one of the methods from Lemma 1.
Revised: Assume that the auxiliary problem (87) is solved by one of the methods from Lemma 1.

Original: Set the parameters N , c 1 , c 2 , c 3 to
Revised: Set the parameters N , c 1 , c 2 , c 3 to

Original: [Equation]
Revised: [Equation]

Original: and let
Revised: and let

Original: [Equation]
Revised: [Equation]

Original: β

Original: [Equation]
Revised: [Equation]

Original: = min(cid:26)
Revised: min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:27) , η x = 2 √ L

Original: [Equation]
Revised: [Equation]

Original: µ x λλ
Revised: η

Original: [Equation]
Revised: [Equation]

Original: y = √

Original: L x µ x √ 2 λ

Original: [Equation]
Revised: [Equation]

Original: λ 2 ,
Revised: √ 2 λ

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Then for the Lyapunov function ∆ k from Theorem 8, we have
Revised: Then for the Lyapunov function ∆ k from Theorem 8, we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, for every ε > 0 , Algorithm 5 finds ( x k , y k ) for which ∆ k ≤ ε ∆ 0 in at most
Revised: Moreover, for every ε > 0 , Algorithm 5 finds ( x k , y k ) for which ∆ k ≤ ε ∆ 0 in at most

Original: O (cid:16) κ 

Revised: O (cid:16) κ

Original: + α 2 α log ( 1 / ε ) (cid:17) gradient computations and at most O (cid:0) √ κχ log ( 1 / ε ) (cid:1) communication rounds.
Revised: + α 2 α log ( 1 / ε ) (cid:17) gradient computations and at most O (cid:0) √ κχ log ( 1 / ε ) (cid:1) communication rounds.

Original: Proof.
Revised: Proof.

Original: The main idea of the proof is this methods is to show two following things: i) Theoremholds true with some replacements, i.e. L xy → λ 1 , µ xy → λ 2 , where λ 1 , λ 2 is upper bound and lower bound of the spectrum of a matrix, which is defined below; ii) Estimate λ 1 and λ 2 .
Revised: The main idea of the proof is this methods is to show two following things: i) Theoremholds true with some replacements, i.e. L xy → λ 1 , µ xy → λ 2 , where λ 1 , λ 2 is upper bound and lower bound of the spectrum of a matrix, which is defined below; ii) Estimate λ 1 and λ 2 .

Original: The proof of Theorem 9 is similar to the proof of Corollary 1 from [Kovalev et al., 2020].
Revised: The proof of Theorem 9 is similar to the proof of Corollary 1 from [Kovalev et al., 2020].

Original: According to the proof from [Kovalev et al., 2020], we need to replace
Revised: According to the proof from [Kovalev et al., 2020], we need to replace

Original: [Equation]
Revised: [Equation]

Original: G Proof of Lemma
Revised: G Proof of Lemma

Original: Proof.
Revised: Proof.

Original: We consider the three options separately.
Revised: We consider the three options separately.

Original: We will use f instead of Ψ and x instead of w in the proof.
Revised: We will use f instead of Ψ and x instead of w in the proof.

Original: Let T = 2 K .
Revised: Let T = 2 K .

Original: (i) K iterations of GD followed by K iterations GD.
Revised: (i) K iterations of GD followed by K iterations GD.

Original: We consider the GD method with stepsize γ = 1 L , which performs the iterations
Revised: We consider the GD method with stepsize γ = 1 L , which performs the iterations

Original: [Equation]
Revised: [Equation]

Original: where L is the smoothness constant of f .
Revised: where L is the smoothness constant of f .

Original: We assume that f is convex.
Revised: We assume that f is convex.

Original: • Gradient decreases monotonically.
Revised: • Gradient decreases monotonically.

Original: According to the chain of inequalities
Revised: According to the chain of inequalities

Original: [Equation]
Revised: [Equation]

Original: where the first inequality follows from convexity and L -smoothness, the gradient norm decreases monotonically.
Revised: where the first inequality follows from convexity and L -smoothness, the gradient norm decreases monotonically.

Original: • Bound on best gradient norm.
Revised: • Bound on best gradient norm.

Original: Further, using L -smoothness of f , we obtain
Revised: Further, using L -smoothness of f , we obtain

Original: [Equation]
Revised: [Equation]

Original: Summing up from k = K to k = 2 K , we get
Revised: Summing up from k = K to k = 2 K , we get

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Bound on function suboptimality.
Revised: Bound on function suboptimality.

Original: It is known that for convex L -smooth functions, Gradient Descent (GD) with constant stepsize γ = 1 L , i
Revised: It is known that for convex L -smooth functions, Gradient Descent (GD) with constant stepsize γ = 1 L , i

Original: [Equation]
Revised: [Equation]

Original: e., the method
Revised: e., the method

Original: [Equation]
Revised: [Equation]

Original: satisfies [Nesterov, 2004]
Revised: satisfies [Nesterov, 2004]

Original: [Equation]
Revised: [Equation]

Original: • Bound on last gradient norm.
Revised: • Bound on last gradient norm.

Original: By combining (88), (90) and (89), we get
Revised: By combining (88), (90) and (89), we get

Original: [Equation]
Revised: [Equation]

Original: (ii) K iterations of FGD followed by K iterations GD.
Revised: (ii) K iterations of FGD followed by K iterations GD.

Original: A better rate can be obtained if we replace the first half of the iterative process with the Fast Gradient Descent (FGD) method [Nesterov, 2004].
Revised: A better rate can be obtained if we replace the first half of the iterative process with the Fast Gradient Descent (FGD) method [Nesterov, 2004].

Original: Bound on function suboptimality.
Revised: Bound on function suboptimality.

Original: If we employ the Fast Gradient Descent (FGD) method during the first K iterations, we get (see, for example, [Nesterov, 2004]):
Revised: If we employ the Fast Gradient Descent (FGD) method during the first K iterations, we get (see, for example, [Nesterov, 2004]):

Original: [Equation]
Revised: [Equation]

Original: • Bound on last gradient norm.
Revised: • Bound on last gradient norm.

Original: Since in the last K iterations we use GD, inequalities (88) and (89) still apply.
Revised: Since in the last K iterations we use GD, inequalities (88) and (89) still apply.

Original: It remains to combine (88), (89) and (91):
Revised: It remains to combine (88), (89) and (91):

Original: [Equation]
Revised: [Equation]

Original: (iii) K iterations of FGD followed by K iterations FSFOM.
Revised: (iii) K iterations of FGD followed by K iterations FSFOM.

Original: A better rate can be obtained if we further replace the second half of the iterative process with the FSFOM method of Kim and Fessler [2021].
Revised: A better rate can be obtained if we further replace the second half of the iterative process with the FSFOM method of Kim and Fessler [2021].

Original: Bound on the gradient norm of FSFOM.
Revised: Bound on the gradient norm of FSFOM.

Original: The FSFOM method satisfies the following inequality (see Theorem 6.1 from [Kim and Fessler, 2021]):
Revised: The FSFOM method satisfies the following inequality (see Theorem 6.1 from [Kim and Fessler, 2021]):

Original: [Equation]
Revised: [Equation]

Original: • Bound on last gradient norm.
Revised: • Bound on last gradient norm.

Original: By combining (92) and (91), we obtain
Revised: By combining (92) and (91), we obtain

Original: [Equation]
Revised: [Equation]

Original: Results from parts (i), (ii) and (iii) proved above can be written in a unified form as
Revised: Results from parts (i), (ii) and (iii) proved above can be written in a unified form as

Original: [Equation]
Revised: [Equation]

Original: where T = 2 K , and A is a constant which depends on the combination of the methods used during the first K iterations and the last K iterations.
Revised: where T = 2 K , and A is a constant which depends on the combination of the methods used during the first K iterations and the last K iterations.