Original: Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling
Revised: Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling

Original: Abstract
Revised: Abstract

Original: In this paper we study the convex-concave saddle-point problem min x max y f
Revised: In this paper we study the convex-concave saddle-point problem min x max y f

Original: ( x )+ y T
Revised: ( x )+ y T

Original: A x − g
Revised: A x − g

Original: ( y ) , where f ( x ) and g ( y ) are smooth and convex functions.
Revised: ( y ) , where f ( x ) and g ( y ) are smooth and convex functions.

Original: We propose an Accelerated Primal-Dual Gradient Method (APDG) for solving this problem, achieving (i) an optimal linear convergence rate in the strongly-convex-stronglyconcave regime, matching the lower complexity bound (Zhang et al., 2021) and (ii) an accelerated linear convergence rate in the case when only one of the functions f ( x ) and g ( y ) is strongly convex or even none of them are.
Revised: We propose an Accelerated Primal-Dual Gradient Method (APDG) for solving this problem, achieving (i) an optimal linear convergence rate in the strongly-convex-stronglyconcave regime, matching the lower complexity bound (Zhang et al., 2021) and (ii) an accelerated linear convergence rate in the case when only one of the functions f ( x ) and g ( y ) is strongly convex or even none of them are.

Original: Finally, we obtain a linearly convergent algorithm for the general smooth and convex-concave saddle point problem min x max y F ( x, y ) without the requirement of strong convexity or strong concavity.
Revised: Finally, we obtain a linearly convergent algorithm for the general smooth and convex-concave saddle point problem min x max y F ( x, y ) without the requirement of strong convexity or strong concavity.

Original:  Introduction
Revised:  Introduction

Original: In this paper we revisit the well studied smooth convex-concave saddle point problem with a bilinear coupling function, which takes the form
Revised: In this paper we revisit the well studied smooth convex-concave saddle point problem with a bilinear coupling function, which takes the form

Original: [Equation]
Revised: [Equation]

Original: where f ( x ): R d x → R and g ( y ): R d y → R are smooth and convex functions, and A ∈ R d y × d x is a coupling matrix.
Revised: where f ( x ): R d x → R and g ( y ): R d y → R are smooth and convex functions, and A ∈ R d y × d x is a coupling matrix.

Original: Problem (1) has a large number of application, some of which we now briefly introduce.
Revised: Problem (1) has a large number of application, some of which we now briefly introduce.

Original: 1.1 Empirical risk minimization
Revised: 1.1 Empirical risk minimization

Original: A classical application is the regularized empirical risk minimization (ERM) with linear predictors, which is a classical supervised learning problem.
Revised: A classical application is the regularized empirical risk minimization (ERM) with linear predictors, which is a classical supervised learning problem.

Original: Given a data matrix A
Revised: Given a data matrix A

Original: =
Revised: =

Original: [ a 1 , . .
Revised: [ a 1 , . .

Original: . , a n ] ⊤ ∈ R n × d , where a i ∈ R d is the feature vector of the i -th data entry
Revised: . , a n ] ⊤ ∈ R n × d , where a i ∈ R d is the feature vector of the i -th data entry

Original: , our goal is to find a solution of
Revised: , our goal is to find a solution of

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) : R d → R is a convex regularizer, ℓ ( y ) :
Revised: where f ( x ) : R d → R is a convex regularizer, ℓ ( y ) :

Original: R n → R is a convex loss function, and x ∈ R
Revised: R n → R is a convex loss function, and x ∈ R

Original: d is a linear predictor.
Revised: d is a linear predictor.

Original: Alterantively, one can solve the following equivalent saddle-point reformulation of problem (2):
Revised: Alterantively, one can solve the following equivalent saddle-point reformulation of problem (2):

Original: [Equation]
Revised: [Equation]

Original: The saddle-point reformulation is often preferable.
Revised: The saddle-point reformulation is often preferable.

Original: For example, when such a formulation admits a finite sum structure (Zhang and Lin, 2015; Wang and Xiao, 2017), this may reduce the communication complexity in the distributed setting (Xiao et al., 2019), and one may also better exploit the udnerlying sparsity structure (Lei et al., 2017).
Revised: For example, when such a formulation admits a finite sum structure (Zhang and Lin, 2015; Wang and Xiao, 2017), this may reduce the communication complexity in the distributed setting (Xiao et al., 2019), and one may also better exploit the udnerlying sparsity structure (Lei et al., 2017).

Original: 1.2 Reinforcement learning
Revised: 1.2 Reinforcement learning

Original: In reinforcement learning (RL) we are given a sequence { ( s t , a t , r t , s t +1 ) } nt =1 generated by a policy π , where s t is the state at time step t , a t is the action taken at time step t by policy π and r t is the reward after taking action a t .
Revised: In reinforcement learning (RL) we are given a sequence { ( s t , a t , r t , s t +1 ) } nt =1 generated by a policy π , where s t is the state at time step t , a t is the action taken at time step t by policy π and r t is the reward after taking action a t .

Original: A key step in many RL algorithms is to estimate the value function of a given policy π , which is defined as
Revised: A key step in many RL algorithms is to estimate the value function of a given policy π , which is defined as

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ (0 , 1) is a discount factor.
Revised: where γ ∈ (0 , 1) is a discount factor.

Original: A common approach to this problem is to use a linear approximation V π ( s ) =
Revised: A common approach to this problem is to use a linear approximation V π ( s ) =

Original: ϕ ( s ) ⊤ x , where ϕ ( s ) is a feature vector of a state s .
Revised: ϕ ( s ) ⊤ x , where ϕ ( s ) is a feature vector of a state s .

Original: The model parameter x is often estimated by minimizing the mean squared projected Bellman error
Revised: The model parameter x is often estimated by minimizing the mean squared projected Bellman error

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: C = (cid:80) nt =1 ϕ ( s t ) ϕ
Revised: C = (cid:80) nt =1 ϕ ( s t ) ϕ

Original: ( s t ) ⊤ , b = (cid:80) nt =1 r t ϕ ( s t ) and A = C − γ (cid:80) nt =1 ϕ ( s t ) ϕ
Revised: ( s t ) ⊤ , b = (cid:80) nt =1 r t ϕ ( s t ) and A = C − γ (cid:80) nt =1 ϕ ( s t ) ϕ

Original: ( s t +1 ) ⊤ .
Revised: ( s t +1 ) ⊤ .

Original: One can observe that it is hard to apply gradient-based methods to problem (5) because this would require one to compute an inverse of the matrix C .
Revised: One can observe that it is hard to apply gradient-based methods to problem (5) because this would require one to compute an inverse of the matrix C .

Original: In order to tackle this issue, one can solve an equivalent saddle-point reformulation proposed by Du et al.
Revised: In order to tackle this issue, one can solve an equivalent saddle-point reformulation proposed by Du et al.

Original: (2017) instead.
Revised: (2017) instead.

Original: This reformulation is given by
Revised: This reformulation is given by

Original: [Equation]
Revised: [Equation]

Original: and is an instance of problem (1).
Revised: and is an instance of problem (1).

Original: Solving this reformulation with gradient methods does not require matrix inversion.
Revised: Solving this reformulation with gradient methods does not require matrix inversion.

Original: 1.3 Minimization under affine constraints
Revised: 1.3 Minimization under affine constraints

Original: Next, consider the problem of convex minimization under affine constraints,
Revised: Next, consider the problem of convex minimization under affine constraints,

Original: [Equation]
Revised: [Equation]

Original: where b ∈ range A .
Revised: where b ∈ range A .

Original: This problem covers a wide range of applications, including inverse problems in imaging (Chambolle and Pock, 2016), sketched learning-type applications (Keriven et al., 2018), network flow optimization (Zargham et al., 2013) and optimal transport (Peyré et al., 2019).
Revised: This problem covers a wide range of applications, including inverse problems in imaging (Chambolle and Pock, 2016), sketched learning-type applications (Keriven et al., 2018), network flow optimization (Zargham et al., 2013) and optimal transport (Peyré et al., 2019).

Original: Another important application of problem (7) is decentralized distributed optimization (Kovalev et al., 2020; Scaman et al., 2017; Li et al., 2020; Nedic et al., 2017; Arjevani et al., 2020; Ye et al., 2020).
Revised: Another important application of problem (7) is decentralized distributed optimization (Kovalev et al., 2020; Scaman et al., 2017; Li et al., 2020; Nedic et al., 2017; Arjevani et al., 2020; Ye et al., 2020).

Original: In this setting, the distributed minimization problem is often reformulated as
Revised: In this setting, the distributed minimization problem is often reformulated as

Original: [Equation]
Revised: [Equation]

Original: where f i ( x i ) is a function stored locally by a computing node i ∈ { 1 , . .
Revised: where f i ( x i ) is a function stored locally by a computing node i ∈ { 1 , . .

Original: . , n } and W ∈ R n × n is the Laplacian matrix of a graph representing the communication network.
Revised: . , n } and W ∈ R n × n is the Laplacian matrix of a graph representing the communication network.

Original: The constraint enforces consensus among the nodes: x 1 = . . .
Revised: The constraint enforces consensus among the nodes: x 1 = . . .

Original: = x n .
Revised: = x n .

Original: One can observe that problem (7) is equivalent to the saddle-point formulation
Revised: One can observe that problem (7) is equivalent to the saddle-point formulation

Original: [Equation]
Revised: [Equation]

Original: which is another instance of problem (1).
Revised: which is another instance of problem (1).

Original: State-of-the-art methods often focus on this formulation instead of directly solving (7).
Revised: State-of-the-art methods often focus on this formulation instead of directly solving (7).

Original: In particular, Salim et al.
Revised: In particular, Salim et al.

Original: (2021) and Kovalev et al.
Revised: (2021) and Kovalev et al.

Original: (2020) obtained optimal algorithms for solving (7) and (8) using this saddle-point approach.
Revised: (2020) obtained optimal algorithms for solving (7) and (8) using this saddle-point approach.

Original: [Table]
Revised: [Table]

Original: 1.4 Bilinear min-max problems
Revised: 1.4 Bilinear min-max problems

Original: Unconstrained bilinear saddle-point problems of the form
Revised: Unconstrained bilinear saddle-point problems of the form

Original: [Equation]
Revised: [Equation]

Original: (10) are another special case of problem (1)
Revised: (10) are another special case of problem (1)

Original: [Table]
Revised: [Table]

Original: one where both f ( x ) and g ( y ) are linear functions.
Revised: one where both f ( x ) and g ( y ) are linear functions.

Original: While such problems do not usually play an important role in practice
Revised: While such problems do not usually play an important role in practice

Original: [Table]
Revised: [Table]

Original: they are often a good testing ground for theoretical purposes (Gidel et al., 2019;
Revised: they are often a good testing ground for theoretical purposes (Gidel et al., 2019;

Original: Azizian et al., 2020; Zhang et al., 2021a; Mokhtari et al., 2020; Daskalakis et al., 2018; Liang and Stokes, 2019)
Revised: Azizian et al., 2020; Zhang et al., 2021a; Mokhtari et al., 2020; Daskalakis et al., 2018; Liang and Stokes, 2019)

Original: [Table]
Revised: [Table]

Original:  Literature Review and Contributions
Revised:  Literature Review and Contributions

Original: In this work we are interested in algorithms able to solve problem (1) with a linear iteration complexity.
Revised: In this work we are interested in algorithms able to solve problem (1) with a linear iteration complexity.

Original: That is, we are interested in methods that can provably find an ϵ -accurate solution of problem (1) in a number of iterations proportional to log 1 ϵ (see Definitions 2 and 3).
Revised: That is, we are interested in methods that can provably find an ϵ -accurate solution of problem (1) in a number of iterations proportional to log 1 ϵ (see Definitions 2 and 3).

Original: This is typically achieved when functions f ( x ) and g ( x ) are assumed to be strongly convex (see Definition 1).
Revised: This is typically achieved when functions f ( x ) and g ( x ) are assumed to be strongly convex (see Definition 1).

Original: An example of this is the celebrated extragradient method of Korpelevich (1976).
Revised: An example of this is the celebrated extragradient method of Korpelevich (1976).

Original: Recent work has shown that linear iteration complexity can be achieved also in the less restrictive case when only one of the functions f ( x ) and g ( x ) is strongly convex.
Revised: Recent work has shown that linear iteration complexity can be achieved also in the less restrictive case when only one of the functions f ( x ) and g ( x ) is strongly convex.

Original: This was first shown by Du and Hu (2019), and later improved on by Zhang et al.
Revised: This was first shown by Du and Hu (2019), and later improved on by Zhang et al.

Original: (2021a).
Revised: (2021a).

Original: However, and this is the starting point of our research, to the best of our knowledge, there are no algorithms with linear iteration complexity in the case when neither f ( x ) nor g ( x ) is strongly convex.
Revised: However, and this is the starting point of our research, to the best of our knowledge, there are no algorithms with linear iteration complexity in the case when neither f ( x ) nor g ( x ) is strongly convex.

Original: 2.1 Acceleration
Revised: 2.1 Acceleration

Original: Loosely speaking, we say that an algorithm is non-accelerated if its iteration complexity is proportional to at least the first power of the condition numbers associated with the problem, such as L x / µ x and L y / µ y , where L x and L y are smoothness constants, and µ x and µ y are strong convexity constants (see Assumption 1 and Assumption 2).
Revised: Loosely speaking, we say that an algorithm is non-accelerated if its iteration complexity is proportional to at least the first power of the condition numbers associated with the problem, such as L x / µ x and L y / µ y , where L x and L y are smoothness constants, and µ x and µ y are strong convexity constants (see Assumption 1 and Assumption 2).

Original: In contrast, the iteration complexity of an accelerated algorithm is proportional to the square root of such condition numbers, e.g., (cid:112)
Revised: In contrast, the iteration complexity of an accelerated algorithm is proportional to the square root of such condition numbers, e.g., (cid:112)

Original: L x / µ x and (cid:112) L y / µ y .
Revised: L x / µ x and (cid:112) L y / µ y .

Original: There were several recent attempts to design accelerated algorithms for solving problem (1) (Xie et al., 2021; Wang and Li, 2020;
Revised: There were several recent attempts to design accelerated algorithms for solving problem (1) (Xie et al., 2021; Wang and Li, 2020;

Original: Alkousa et al., 2020).
Revised: Alkousa et al., 2020).

Original: These attempts rely on stacking multiple algorithms on top of each other , and result in complicated methods.
Revised: These attempts rely on stacking multiple algorithms on top of each other , and result in complicated methods.

Original: For example, Lin et al.
Revised: For example, Lin et al.

Original: (2020) use a non-accelerated algorithm as a sub-routine for the inexact accelerated proximal-point method.
Revised: (2020) use a non-accelerated algorithm as a sub-routine for the inexact accelerated proximal-point method.

Original: This approach allows them to obtain accelerated algorithms for solving problem (1) in a straightforward and tractable way.
Revised: This approach allows them to obtain accelerated algorithms for solving problem (1) in a straightforward and tractable way.

Original: However, this approach has significant drawbacks: the algorithms obtained this way have (i) additional logarithmic factors in their iteration complexity, and (ii) a complex nested structure with the requirement to manually set inner loop sizes, which is a byproduct of the design process based on combining multiple algorithms.
Revised: However, this approach has significant drawbacks: the algorithms obtained this way have (i) additional logarithmic factors in their iteration complexity, and (ii) a complex nested structure with the requirement to manually set inner loop sizes, which is a byproduct of the design process based on combining multiple algorithms.

Original: This drawback limits the performance of the resulting algorithms in theory, and requires additional fine tuning in practice.
Revised: This drawback limits the performance of the resulting algorithms in theory, and requires additional fine tuning in practice.

Original: A philosophically different approach to designing such algorithms—one that we adopt in this work—is to attempt to provide a direct acceleration of a suitable algorithm for solving problem (1), similarly to what Nesterov (1983) did for convex minimization problems.
Revised: A philosophically different approach to designing such algorithms—one that we adopt in this work—is to attempt to provide a direct acceleration of a suitable algorithm for solving problem (1), similarly to what Nesterov (1983) did for convex minimization problems.

Original: While this technically more demanding, algorithms obtained this way typically don’t have the aforementioned drawbacks.
Revised: While this technically more demanding, algorithms obtained this way typically don’t have the aforementioned drawbacks.

Original: Hence, we follow the latter approach in this work.
Revised: Hence, we follow the latter approach in this work.

Original: 2.2 Main contributions
Revised: 2.2 Main contributions

Original: In this work we propose an Accelerated Primal-Dual Gradient Method (APDG; Algorithm 1) for solving problem (1) and provide a theoretical analysis of its convergence properties (Theorem 1).
Revised: In this work we propose an Accelerated Primal-Dual Gradient Method (APDG; Algorithm 1) for solving problem (1) and provide a theoretical analysis of its convergence properties (Theorem 1).

Original: In particular, we prove the following results.
Revised: In particular, we prove the following results.

Original: (i) When both functions f ( x ) and g ( y ) are strongly convex, Algorithm 1 achieves the optimal linear convergence rate, matching the lower bound obtained by Zhang et al.
Revised: (i) When both functions f ( x ) and g ( y ) are strongly convex, Algorithm 1 achieves the optimal linear convergence rate, matching the lower bound obtained by Zhang et al.

Original: (2021b).
Revised: (2021b).

Original: To the best of our knowledge, Algorithm 1 is the first optimal algorithm in this regime.
Revised: To the best of our knowledge, Algorithm 1 is the first optimal algorithm in this regime.

Original: (ii)
Revised: (ii)

Original: We establish linear convergence of Algorithm 1 in the case when only one of the functions f ( x ) or g ( y ) is strongly convex, and A is a full row or full column rank matrix, respectively.
Revised: We establish linear convergence of Algorithm 1 in the case when only one of the functions f ( x ) or g ( y ) is strongly convex, and A is a full row or full column rank matrix, respectively.

Original: This improves upon the results provided by Du and Hu (2019); Zhang et al.
Revised: This improves upon the results provided by Du and Hu (2019); Zhang et al.

Original: (2021a).
Revised: (2021a).

Original: (iii)
Revised: (iii)

Original: We establish linear convergence of the Algorithm 1 in the case when neither of the functions f ( x ) nor g ( y ) is strongly convex, and the matrix A is square and full rank.
Revised: We establish linear convergence of the Algorithm 1 in the case when neither of the functions f ( x ) nor g ( y ) is strongly convex, and the matrix A is square and full rank.

Original: To the best of our knowledge, Algorithm 1 is the first algorithm achieving linear convergence in this setting.
Revised: To the best of our knowledge, Algorithm 1 is the first algorithm achieving linear convergence in this setting.

Original: Table 1 provides a brief comparison of the complexity of Algorithm 1 (Theorem 1) with the current state of the art.
Revised: Table 1 provides a brief comparison of the complexity of Algorithm 1 (Theorem 1) with the current state of the art.

Original: Please refer to section 5 for a detailed discussion of this result and comparison with related work.
Revised: Please refer to section 5 for a detailed discussion of this result and comparison with related work.

Original: 2.3 General min-max problem and additional contributions
Revised: 2.3 General min-max problem and additional contributions

Original: In our work we also consider the saddle-point problem
Revised: In our work we also consider the saddle-point problem

Original: [Equation]
Revised: [Equation]

Original: 11) where F ( x, y ): R d x × R d y
Revised: 11) where F ( x, y ): R d x × R d y

Original: → R is a smooth function, which is convex in x and concave in y .
Revised: → R is a smooth function, which is convex in x and concave in y .

Original: One can observe that the main problem (1) is a special case of this more general problem (11).
Revised: One can observe that the main problem (1) is a special case of this more general problem (11).

Original: As an additional contribution, we propose a Gradient Descent-Ascent Method with Extrapolation (GDAE) for solving the general convex-concave saddle-point problem (11), and provide a theoretical analysis of its convergence properties.
Revised: As an additional contribution, we propose a Gradient Descent-Ascent Method with Extrapolation (GDAE) for solving the general convex-concave saddle-point problem (11), and provide a theoretical analysis of its convergence properties.

Original: (i) When the function F ( x, y ) is strongly convex in x and strongly concave in y , GDAE achieves a linear convergence rate, which recovers the convergence result of Cohen et al.
Revised: (i) When the function F ( x, y ) is strongly convex in x and strongly concave in y , GDAE achieves a linear convergence rate, which recovers the convergence result of Cohen et al.

Original: (ii) Under certain assumptions on the way the variables x and y are coupled by the function F ( x, y ) , we establish linear convergence of GDAE in the case when the function F ( x, y ) is strongly-convex-concave, convex-strongly-concave, or even just convex-concave.
Revised: (ii) Under certain assumptions on the way the variables x and y are coupled by the function F ( x, y ) , we establish linear convergence of GDAE in the case when the function F ( x, y ) is strongly-convex-concave, convex-strongly-concave, or even just convex-concave.

Original: To the best of our knowledge, GDAE is the first algorithm achieving linear convergence under such assumptions.
Revised: To the best of our knowledge, GDAE is the first algorithm achieving linear convergence under such assumptions.

Original: Please refer to the Appendix for a detailed description of these results and related work.
Revised: Please refer to the Appendix for a detailed description of these results and related work.

Original:  Basic Definitions and Assumptions
Revised:  Basic Definitions and Assumptions

Original: We start by formalizing the notions of smoothness and strong convexity of a function.
Revised: We start by formalizing the notions of smoothness and strong convexity of a function.

Original: Definition 1.
Revised: Definition 1.

Original: Function h
Revised: Function h

Original: ( z ): R d → R is L -smooth and µ -strongly convex for L ≥ µ ≥ 0 , if for all z 1 , z 2 ∈ R d the following inequality holds:
Revised: ( z ): R d → R is L -smooth and µ -strongly convex for L ≥ µ ≥ 0 , if for all z 1 , z 2 ∈ R d the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Above, D h ( z 1 , z 2 ) =
Revised: Above, D h ( z 1 , z 2 ) =

Original: h ( z 1 ) − h ( z 2 ) − ⟨∇ h ( z 2 ) , z 1 − z 2 ⟩ is the Bregman divergence associated with the function
Revised: h ( z 1 ) − h ( z 2 ) − ⟨∇ h ( z 2 ) , z 1 − z 2 ⟩ is the Bregman divergence associated with the function

Original: h
Revised: h

Original: ( z ) .
Revised: ( z ) .

Original: We are now ready to state the main assumptions that we impose on problem (1).
Revised: We are now ready to state the main assumptions that we impose on problem (1).

Original: We start with
Revised: We start with

Original: Assumptions 1 and 2 that formalize the strong-convexity and smoothness properties of functions f ( x ) and g ( y ) .
Revised: Assumptions 1 and 2 that formalize the strong-convexity and smoothness properties of functions f ( x ) and g ( y ) .

Original: Assumption 1.
Revised: Assumption 1.

Original: Function f ( x ) is L x -smooth and µ x -strongly convex for L x ≥ µ x ≥ 0 .
Revised: Function f ( x ) is L x -smooth and µ x -strongly convex for L x ≥ µ x ≥ 0 .

Original: Assumption 2.
Revised: Assumption 2.

Original: Function g ( y ) is L y -smooth and µ y -strongly convex for L y ≥ µ y ≥ 0 .
Revised: Function g ( y ) is L y -smooth and µ y -strongly convex for L y ≥ µ y ≥ 0 .

Original: Note, that µ x and µ y are allowed to be zero.
Revised: Note, that µ x and µ y are allowed to be zero.

Original: That is, both f ( x ) and g ( y ) are allowed to be non-strongly convex.
Revised: That is, both f ( x ) and g ( y ) are allowed to be non-strongly convex.

Original: The following assumption formalizes the spectral properties of matrix A .
Revised: The following assumption formalizes the spectral properties of matrix A .

Original: Assumption 3.
Revised: Assumption 3.

Original: There exist constants L xy > µ xy , µ yx ≥ 

Revised: There exist constants L xy > µ xy , µ yx ≥

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: where λ min ( · ) , λ +min ( · ) and λ max ( · ) denote the smallest, smallest positive and largest eigenvalue of a matrix, respectively, and range denotes the range space of a matrix.
Revised: where λ min ( · ) , λ +min ( · ) and λ max ( · ) denote the smallest, smallest positive and largest eigenvalue of a matrix, respectively, and range denotes the range space of a matrix.

Original: By S ⊂ R d x × R d y we denote the solution set of problem (1).
Revised: By S ⊂ R d x × R d y we denote the solution set of problem (1).

Original: Note that ( x ∗ , y ∗ ) ∈ S
Revised: Note that ( x ∗ , y ∗ ) ∈ S

Original: if and only if ( x ∗ , y ∗ ) satisfies the first-order optimality conditions
Revised: if and only if ( x ∗ , y ∗ ) satisfies the first-order optimality conditions

Original: [Equation]
Revised: [Equation]

Original: Our main goal is to propose an algorithm for finding a solution to problem (1).
Revised: Our main goal is to propose an algorithm for finding a solution to problem (1).

Original: Numerical iterative algorithms typically find an approximate solution of a given problem.
Revised: Numerical iterative algorithms typically find an approximate solution of a given problem.

Original: We formalize this through the following definition.
Revised: We formalize this through the following definition.

Original: Definition 2.
Revised: Definition 2.

Original: Let the solution set S be nonempty.
Revised: Let the solution set S be nonempty.

Original: We call a pair of vectors (
Revised: We call a pair of vectors (

Original: x, y ) ∈
Revised: x, y ) ∈

Original: R d x × R d y an ϵ -accurate solution of problem (1) for a given accuracy ϵ > 0 if it satisfies
Revised: R d x × R d y an ϵ -accurate solution of problem (1) for a given accuracy ϵ > 0 if it satisfies

Original: [Equation]
Revised: [Equation]

Original: We also want to propose an efficient algorithm for solving problem (1).
Revised: We also want to propose an efficient algorithm for solving problem (1).

Original: That is, we want to propose an algorithm with the the lowest possible iteration complexity , which we define next.
Revised: That is, we want to propose an algorithm with the the lowest possible iteration complexity , which we define next.

Original: Definition 3.
Revised: Definition 3.

Original: The iteration complexity of an algorithm for solving problem (1) is the number of iterations the algorithm requires to find an ϵ -accurate solution of this problem.
Revised: The iteration complexity of an algorithm for solving problem (1) is the number of iterations the algorithm requires to find an ϵ -accurate solution of this problem.

Original: At each iteration the algorithm is allowed to perform O (1) computations of the gradients ∇ f ( x ) and ∇ g ( y ) and matrix-vector multiplications with matrices A and A ⊤ .
Revised: At each iteration the algorithm is allowed to perform O (1) computations of the gradients ∇ f ( x ) and ∇ g ( y ) and matrix-vector multiplications with matrices A and A ⊤ .

Original:  Accelerated Primal-Dual Gradient Method
Revised:  Accelerated Primal-Dual Gradient Method

Original: 1: Input: x 0 ∈ range A ⊤ , y 0 ∈ range A , η x , η y , α x , α y , β x , β y > 0 , τ x , τ y , σ x , σ y ∈ (0 , 1] , θ ∈ (0 , 1) 2:
Revised: 1: Input: x 0 ∈ range A ⊤ , y 0 ∈ range A , η x , η y , α x , α y , β x , β y > 0 , τ x , τ y , σ x , σ y ∈ (0 , 1] , θ ∈ (0 , 1) 2:

Original: 0 f = x do
Revised: 0 f = x do

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: 12: end for
Revised: 12: end for

Original: In this section we present the Accelerated Primal-Dual Gradient Method (APDG; Algorithm 1) for solving problem (1).
Revised: In this section we present the Accelerated Primal-Dual Gradient Method (APDG; Algorithm 1) for solving problem (1).

Original: First, we prove an outline of the key ideas used in the development of this algorithm.
Revised: First, we prove an outline of the key ideas used in the development of this algorithm.

Original: 4.1 Algorithm development strategy
Revised: 4.1 Algorithm development strategy

Original: First, we observe that problem (1) is equivalent to the problem of finding a zero of a sum of two monotone operators, G 1 , G 2 : R d x × R d y → R d x × R d y , defined as
Revised: First, we observe that problem (1) is equivalent to the problem of finding a zero of a sum of two monotone operators, G 1 , G 2 : R d x × R d y → R d x × R d y , defined as

Original: [Equation]
Revised: [Equation]

Original: Indeed, G 1 ( x ∗ , y ∗ )
Revised: Indeed, G 1 ( x ∗ , y ∗ )

Original: + G 2 ( x ∗ , y ∗ )
Revised: + G 2 ( x ∗ , y ∗ )

Original: = 0 is just another way to write the optimality conditions (13).
Revised: = 0 is just another way to write the optimality conditions (13).

Original: The Forward Backward algorithm.
Revised: The Forward Backward algorithm.

Original: A natural way to tackle this problem is via Forward Backward algorithm
Revised: A natural way to tackle this problem is via Forward Backward algorithm

Original: (Bauschke and Combettes, 2011), the iterates of which have the form
Revised: (Bauschke and Combettes, 2011), the iterates of which have the form

Original: [Equation]
Revised: [Equation]

Original: where the operator J G 2 is the inverse of the operator
Revised: where the operator J G 2 is the inverse of the operator

Original: I + G 2 , and I is the identity operator.
Revised: I + G 2 , and I is the identity operator.

Original: Note that
Revised: Note that

Original: J G 2 can be written as J G 2 : ( x, y )
Revised: J G 2 can be written as J G 2 : ( x, y )

Original: (cid:55)→ ( x + , y + ) , where ( x + , y + ) ∈ R
Revised: (cid:55)→ ( x + , y + ) , where ( x + , y + ) ∈ R

Original: d x × R d y is a solution of the linear system
Revised: d x × R d y is a solution of the linear system

Original: [Equation]
Revised: [Equation]

Original: Linear extrapolation step.
Revised: Linear extrapolation step.

Original: Next, notice that the computation of operator J G 2 requires solving the linear system (18).
Revised: Next, notice that the computation of operator J G 2 requires solving the linear system (18).

Original: This is expensive 5 and has to be done at each iteration of the Forward Backward algorithm.
Revised: This is expensive 5 and has to be done at each iteration of the Forward Backward algorithm.

Original: Let us instead consider the related problem
Revised: Let us instead consider the related problem

Original: [Equation]
Revised: [Equation]

Original: where y m ∈ R
Revised: where y m ∈ R

Original: d y is a newly introduced variable.
Revised: d y is a newly introduced variable.

Original: It’s easy to observe that (19) is equivalent to (18)
Revised: It’s easy to observe that (19) is equivalent to (18)

Original: when
Revised: when

Original: y m = y + .
Revised: y m = y + .

Original: Next, notice that choosing y m =
Revised: Next, notice that choosing y m =

Original: y makes (19) easy to solve.
Revised: y makes (19) easy to solve.

Original: However, it turns out that the convergence analysis of an algorithm with this approximation may be challenging (Zhang et al., 2021a), especially if we want to combine it with other techniques, such as acceleration.
Revised: However, it turns out that the convergence analysis of an algorithm with this approximation may be challenging (Zhang et al., 2021a), especially if we want to combine it with other techniques, such as acceleration.

Original: Our key idea is to propose a better alternative: the linear extrapolation step
Revised: Our key idea is to propose a better alternative: the linear extrapolation step

Original: [Equation]
Revised: [Equation]

Original: where y − ∈ R d y corresponds to y obtained from the previous iteration of the Forward Backward algorithm, and θ ∈ (0 , 1] is an extrapolation parameter.
Revised: where y − ∈ R d y corresponds to y obtained from the previous iteration of the Forward Backward algorithm, and θ ∈ (0 , 1] is an extrapolation parameter.

Original: The linear extrapolation step was introduced by Chambolle and Pock (2011) in the analysis of the Primal-Dual Hybrid Gradient algorithm 6 .
Revised: The linear extrapolation step was introduced by Chambolle and Pock (2011) in the analysis of the Primal-Dual Hybrid Gradient algorithm 6 .

Original: Nesterov acceleration.
Revised: Nesterov acceleration.

Original: Next, we note that operator G 1 is equal to the gradient of the (potential) function ( x, y )
Revised: Next, we note that operator G 1 is equal to the gradient of the (potential) function ( x, y )

Original: (cid:55)→ f ( x ) + g ( y ) function.
Revised: (cid:55)→ f ( x ) + g ( y ) function.

Original: This function is smooth and convex due to Assumptionsand 2.
Revised: This function is smooth and convex due to Assumptionsand 2.

Original: This allows us to incorporate the Nesterov acceleration mechanism in the Forward Backward algorithm.
Revised: This allows us to incorporate the Nesterov acceleration mechanism in the Forward Backward algorithm.

Original: Nesterov acceleration is known to be a powerful tool which allows to improve convergence properties of gradient methods (Nesterov, 1983, 2003).
Revised: Nesterov acceleration is known to be a powerful tool which allows to improve convergence properties of gradient methods (Nesterov, 1983, 2003).

Original: 4.2 Convergence of the algorithm
Revised: 4.2 Convergence of the algorithm

Original: We are now ready to study the convergence properties of Algorithm 1.
Revised: We are now ready to study the convergence properties of Algorithm 1.

Original: We are interested in the case when the following condition holds:
Revised: We are interested in the case when the following condition holds:

Original: [Equation]
Revised: [Equation]

Original: In this case one can show that the solution set S of problem (1) is nonempty.
Revised: In this case one can show that the solution set S of problem (1) is nonempty.

Original: Moreover, strong duality holds in this case, as captured by the following lemma.
Revised: Moreover, strong duality holds in this case, as captured by the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let Assumptions 1, 2 and 3 and condition (21) hold.
Revised: Let Assumptions 1, 2 and 3 and condition (21) hold.

Original: Let p be the optimal value of the primal problem
Revised: Let p be the optimal value of the primal problem

Original: [Equation]
Revised: [Equation]

Original: and let d be the optimal value of the dual problem
Revised: and let d be the optimal value of the dual problem

Original: [Equation]
Revised: [Equation]

Original: Then p =
Revised: Then p =

Original: d is finite and ( x ∗ , y ∗ ) ∈ S
Revised: d is finite and ( x ∗ , y ∗ ) ∈ S

Original: if and only if x ∗ is a solution of the primal problem (22) and y ∗ is a solution of the dual problem (23) .
Revised: if and only if x ∗ is a solution of the primal problem (22) and y ∗ is a solution of the dual problem (23) .

Original: Under the aforementioned conditions, Algorithm 1 achieves linear convergence.
Revised: Under the aforementioned conditions, Algorithm 1 achieves linear convergence.

Original: That is, the iteration complexity is proportional to log 1 ϵ .
Revised: That is, the iteration complexity is proportional to log 1 ϵ .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Assumptions 1, 2 and 3 and condition (21) hold.
Revised: Let Assumptions 1, 2 and 3 and condition (21) hold.

Original: Then there exist parameters of
Revised: Then there exist parameters of

Original: Algorithm 1 such that its iteration complexity for finding an ϵ -accurate solution of problem (1) is
Revised: Algorithm 1 such that its iteration complexity for finding an ϵ -accurate solution of problem (1) is

Original: [Equation]
Revised: [Equation]

Original: where T a , T b , T c , T d are defined as
Revised: where T a , T b , T c , T d are defined as

Original: [Equation]
Revised: [Equation]

Original: and C > 0 is some constant, which does not depend on ϵ , but possibly depends on
Revised: and C > 0 is some constant, which does not depend on ϵ , but possibly depends on

Original: L x , µ x , L y , µ y , L xy , µ xy , µ yx .
Revised: L x , µ x , L y , µ y , L xy , µ xy , µ yx .

Original:  Discussion of Theorem 1 and Related Work
Revised:  Discussion of Theorem 1 and Related Work

Original: In this section we comment on the iteration complexity result for Algorithm 1 provided in Theorem 1.
Revised: In this section we comment on the iteration complexity result for Algorithm 1 provided in Theorem 1.

Original: We consider important and illustrative special cases of this complexity result and draw connections with the existing results in the literature.
Revised: We consider important and illustrative special cases of this complexity result and draw connections with the existing results in the literature.

Original: 5.1 Strongly convex and strongly concave case
Revised: 5.1 Strongly convex and strongly concave case

Original: In this case µ x , µ y > 0 .
Revised: In this case µ x , µ y > 0 .

Original: We can always assume µ xy = µ yx = 0 in Assumption 3.
Revised: We can always assume µ xy = µ yx = 0 in Assumption 3.

Original: Then, Algorithmhas iteration complexity given by
Revised: Then, Algorithmhas iteration complexity given by

Original: [Equation]
Revised: [Equation]

Original: This improves the current state-of-the-art results
Revised: This improves the current state-of-the-art results

Original: [Equation]
Revised: [Equation]

Original: due to Xie et al.
Revised: due to Xie et al.

Original: (2021), and
Revised: (2021), and

Original: [Equation]
Revised: [Equation]

Original: due to Wang and Li (2020), where ˜ O ( · ) hides additional logarithmic factors, and L = max { L x , L y , L xy } .
Revised: due to Wang and Li (2020), where ˜ O ( · ) hides additional logarithmic factors, and L = max { L x , L y , L xy } .

Original: Moreover, our result (25) matches the lower complexity bound provided by
Revised: Moreover, our result (25) matches the lower complexity bound provided by

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021b).
Revised: (2021b).

Original: Hence, Algorithm 1 is optimal in this regime.
Revised: Hence, Algorithm 1 is optimal in this regime.

Original: Apart from our work, algorithms that achieve optimal complexity (25) were developed in three independent works by Thekumparampil et al.
Revised: Apart from our work, algorithms that achieve optimal complexity (25) were developed in three independent works by Thekumparampil et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: ; Du et al. (2022).
Revised: ; Du et al. (2022).

Original: However, to the best of our knowledge these works were published or appeared on arXiv in 2022, while our work appeared on arXiv in 2021.
Revised: However, to the best of our knowledge these works were published or appeared on arXiv in 2022, while our work appeared on arXiv in 2021.

Original: Hence, Algorithm 1 is the first algorithm which achieves the lower complexity bound (25) for smooth and strongly-convex-strongly-concave saddle-point problems with bilinear coupling.
Revised: Hence, Algorithm 1 is the first algorithm which achieves the lower complexity bound (25) for smooth and strongly-convex-strongly-concave saddle-point problems with bilinear coupling.

Original: 5.2 Affinely-constrainted minimization case
Revised: 5.2 Affinely-constrainted minimization case

Original: In this case µ x > 0 and µ y = 0 .
Revised: In this case µ x > 0 and µ y = 0 .

Original: Firstly, we consider the case when L y = 0 , i.e., g ( y ) is a linear function.
Revised: Firstly, we consider the case when L y = 0 , i.e., g ( y ) is a linear function.

Original: Then, problem (1) is equivalent to the smooth and strongly-convex affinely-constrained minimization problem (7).
Revised: Then, problem (1) is equivalent to the smooth and strongly-convex affinely-constrained minimization problem (7).

Original: Algorithm 1 enjoys the linear convergence rate
Revised: Algorithm 1 enjoys the linear convergence rate

Original: [Equation]
Revised: [Equation]

Original: where µ xy = λ
Revised: where µ xy = λ

Original: +min ( AA ⊤ ) > 0 due to Assumption 3.
Revised: +min ( AA ⊤ ) > 0 due to Assumption 3.

Original: This result recovers the complexity of the
Revised: This result recovers the complexity of the

Original: APAPC algorithm (Kovalev et al., 2020).
Revised: APAPC algorithm (Kovalev et al., 2020).

Original: It is possible to incorporate the Chebyshev acceleration mechanism (Arioli and Scott, 2014) into Algorithm 1 for solving problem (7) to obtain the improved complexity
Revised: It is possible to incorporate the Chebyshev acceleration mechanism (Arioli and Scott, 2014) into Algorithm 1 for solving problem (7) to obtain the improved complexity

Original: [Equation]
Revised: [Equation]

Original: This matches the complexity of the OPAPC algorithm of Kovalev et al.
Revised: This matches the complexity of the OPAPC algorithm of Kovalev et al.

Original: ; Salim et al.
Revised: ; Salim et al.

Original: (2021), which was shown to be optimal (Salim et al., 2021; Scaman et al., 2017).
Revised: (2021), which was shown to be optimal (Salim et al., 2021; Scaman et al., 2017).

Original: 5.3 Strongly convex and concave case
Revised: 5.3 Strongly convex and concave case

Original: We also allow L y > 0 , i.e., function g ( y ) is a general, not necessarily linear, smooth and convex function.
Revised: We also allow L y > 0 , i.e., function g ( y ) is a general, not necessarily linear, smooth and convex function.

Original: It is often possible that µ xy > 0 due to Assumption 3; for instance, when A is a full row rank matrix.
Revised: It is often possible that µ xy > 0 due to Assumption 3; for instance, when A is a full row rank matrix.

Original: Then, Algorithm 1 enjoys the following linear iteration complexity:
Revised: Then, Algorithm 1 enjoys the following linear iteration complexity:

Original: [Equation]
Revised: [Equation]

Original: This case was previously studied by Du and Hu (2019); Du et al.
Revised: This case was previously studied by Du and Hu (2019); Du et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021a).
Revised: (2021a).

Original: Du and Hu (2019) provided an analysis for an algorithm called Sim-GDA, and established its iteration complexity
Revised: Du and Hu (2019) provided an analysis for an algorithm called Sim-GDA, and established its iteration complexity

Original: [Equation]
Revised: [Equation]

Original: This result is substantially worse than our complexity (30); possibly due to a suboptimal analysis.
Revised: This result is substantially worse than our complexity (30); possibly due to a suboptimal analysis.

Original: Subsequently, Zhang et al.
Revised: Subsequently, Zhang et al.

Original: (2021a) provided an improved analysis for the Sim-GDA algorithm, obtaining the complexity
Revised: (2021a) provided an improved analysis for the Sim-GDA algorithm, obtaining the complexity

Original: [Equation]
Revised: [Equation]

Original: They also studied the Alt-GDA method, obtaining the complexity
Revised: They also studied the Alt-GDA method, obtaining the complexity

Original: [Equation]
Revised: [Equation]

Original: where L = max { L x , L y , L xy }
Revised: where L = max { L x , L y , L xy }

Original: However, these results are local, i.e., they are valid only if the initial iterates of these algorithms are close enough to the solution of problem (1).
Revised: However, these results are local, i.e., they are valid only if the initial iterates of these algorithms are close enough to the solution of problem (1).

Original: Moreover, these results are still worse than our rate (30) because Sim-GDA and Alt-GDA do not utilize the Nesterov acceleration mechanism, while our Algorithm 1 does.
Revised: Moreover, these results are still worse than our rate (30) because Sim-GDA and Alt-GDA do not utilize the Nesterov acceleration mechanism, while our Algorithm 1 does.

Original: 5.4 Bilinear case
Revised: 5.4 Bilinear case

Original: In this case µ x = µ y =
Revised: In this case µ x = µ y =

Original: L x = L y = 0 .
Revised: L x = L y = 0 .

Original: That is, functions f ( x ) and g ( y ) are linear.
Revised: That is, functions f ( x ) and g ( y ) are linear.

Original: Then, problem (1) turns into the bilinear min-max problem (10), and µ 2 xy = µ 2 yx = λ +min ( A ⊤ A ) > 0 due to Assumption 3.
Revised: Then, problem (1) turns into the bilinear min-max problem (10), and µ 2 xy = µ 2 yx = λ +min ( A ⊤ A ) > 0 due to Assumption 3.

Original: The iteration complexity of Algorithm 1 becomes
Revised: The iteration complexity of Algorithm 1 becomes

Original: [Equation]
Revised: [Equation]

Original: This recovers the results of Daskalakis et al.
Revised: This recovers the results of Daskalakis et al.

Original: Liang and Stokes (2019); Gidel et al.
Revised: Liang and Stokes (2019); Gidel et al.

Original: (2018, 2019); Mishchenko et al.
Revised: (2018, 2019); Mishchenko et al.

Original: (2020); Mokhtari et al.
Revised: (2020); Mokhtari et al.

Original: (2020) for the bilinear min-max problem (10).
Revised: (2020) for the bilinear min-max problem (10).

Original: However, this result is worse than the complexity lower bound
Revised: However, this result is worse than the complexity lower bound

Original: [Equation]
Revised: [Equation]

Original: obtained in the work of Ibrahim et al.
Revised: obtained in the work of Ibrahim et al.

Original: (2020), which was reached by Azizian et al.
Revised: (2020), which was reached by Azizian et al.

Original: Du et al.
Revised: Du et al.

Original: 5.5 Convex-concave case
Revised: 5.5 Convex-concave case

Original: In this case µ y = µ x = 0 .
Revised: In this case µ y = µ x = 0 .

Original: It is often possible that µ xy = µ yx > 0 due to Assumption 3, for example, when A is a square and full rank matrix.
Revised: It is often possible that µ xy = µ yx > 0 due to Assumption 3, for example, when A is a square and full rank matrix.

Original: Then, the iteration complexity of Algorithm 1 becomes
Revised: Then, the iteration complexity of Algorithm 1 becomes

Original: [Equation]
Revised: [Equation]

Original: which is still linear.
Revised: which is still linear.

Original: This complexity result generalizes the result (34) for bilinear min-max problems as it allows for general, not necessarily linear, convex and smooth functions f ( x ) and g ( x ) .
Revised: This complexity result generalizes the result (34) for bilinear min-max problems as it allows for general, not necessarily linear, convex and smooth functions f ( x ) and g ( x ) .

Original: To the best of our knowledge, Algorithm 1 is the first algorithm which can achieve linear convergence for smooth and non-strongly convex non-strongly concave min-max problems with bilinear coupling.
Revised: To the best of our knowledge, Algorithm 1 is the first algorithm which can achieve linear convergence for smooth and non-strongly convex non-strongly concave min-max problems with bilinear coupling.

Original: Acknowledgements
Revised: Acknowledgements

Original: The work of Alexander Gasnikov was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of the Russian Academy of Sciences dated November2021 No. 70-2021-00142. A A Novel Gradient Method for General Convex-Concave Saddle-Point Problems Algorithm
Revised: The work of Alexander Gasnikov was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of the Russian Academy of Sciences dated November

Original: Input: x 0 ∈ R d x , y 0 ∈ R d y , η x , η y > 0 , θ ∈ (0 , 1) x − 1 = xy − 1 = yfor k

Original: = 0 , 1 , 2 , . . .

Original: do

Original: [Equation]

Original: ) end for

Original: In this section we present a new method—Gradient Descent-Ascent Method with Extrapolation (GDAE; Algorithm 2)—for solving problem (11).

Original: A.1 Assumptions and definitions

Original: First, we state the main assumptions that we impose on problem (11).

Original: Assumption 4.

Original: Function F ( x, y ) is L x -smooth and µ x -strongly convex in x and L y -smooth and µ y -strongly concave in y , where L x ≥ µ x ≥ 0 , L y ≥ µ y ≥ 0 .

Original: Assumption 4 generalizes the smoothness and strong convexity Assumptions 1 and 2 imposed on problem (1).

Original: Assumption 5.

Original: There exists a constant L xy > 0 such that for all x, x 1 , x 2 ∈ R d x and y, y 1 , y 2 ∈ R d y , the following inequalities hold:

Original: [Equation]

Original: Assumption 6.

Original: There exist constants µ xy , µ yx ≥ 0 such that for all x, x 1 , x 2 ∈ R d x and y, y 1 , y 2 ∈

Original: R d y , the following inequalities hold:

Original: [Equation]

Original: Assumptions 5 and 6 combined form a generalized version of Assumption 3 for problem (11).

Original: Indeed, if one assumes that (37) and (38) hold for problem (1), then the following inequalities hold

Original: [Equation]

Original: which can be seen as a simplified version of Assumption 3.

Original: Next, we recall several basic definitions.

Original: Similarly to section 3, by S ⊂ R d x × R d y we denote the solution set of problem (11).

Original: Note that ( x ∗ , y ∗ ) ∈ S

Original: if and only if ( x ∗ , y ∗ ) satisfies the optimality conditions

Original: [Equation]

Original: We also use notions of iteration complexity for achieving an ϵ -accurate solution analogous to

Original: Definitions 2 and 3, respectively.

Original: A.2 Algorithm development

Original: We now present the main ingredients and intuition behind the development of our method.

Original: Implicit gradient descent-ascent.

Original: First, we recall the iterations of the Forward-Backward algorithm (17), which can be written in the form

Original: [Equation]

Original: where η x , η y > 0 are stepsizes.

Original: Iterations (41) can also be written in terms of the gradients ∇ x F ( x, y ) and ∇ y F ( x, y ) ,

Original: [Equation]

Original: which makes the method applicable to the general problem (11).

Original: Iterations (42) were the foundation for the development of Algorithm 1 for solving problem (1), with strong convergence properties established by Theorem 1.

Original: Hence, we expect that this approach would work for solving the more general problem (11).

Original: However, (42) is an implicit algorithm and can’t be applied in its current state.

Original: Gradient extrapolation.

Original: In analogy to the development of Algorithm 1, we want to find a good approximation of the implicit iterations (42).

Original: A naive solution would be using the approximation

Original: [Equation]

Original: where y m ≈ y + .

Original: Similarly to Section 4.1, we could use y m = y , which would lead to the Alt-GDA algorithm

Original: (Zhang et al., 2021a), or y m

Original: =

Original: y

Original: + θ ( y − y − ) , which is a linear extrapolation step (Chambolle and Pock, 2011).

Original: The linear extrapolation step with θ = 1 is based on the “assumption” that y + ≈ y m =

Original: y + ( y − y − ) , or equivalently, y + − y

Original: ≈

Original: y − y − .

Original: We can use a similar intuition for the gradients ∇ x F ( x, y ) rather than the iterates y .

Original: In particular, we “assume” that

Original: [Equation]

Original: or equivalently,

Original: [Equation]

Original: This intuition leads to the following novel update rule, which we call gradient extrapolation step :

Original: [Equation]

Original: Above, θ ∈ (0 , 1] is the extrapolation parameter.

Original: We use this gradient extrapolation step together with the update rule for y from (42) in the design of our Algorithm 2.

Original: A.3 Convergence of Algorithm 2 and related work

Original: We now present Theorem 2, which establishes linear convergence rate for Algorithm 2 under

Original: Assumptions 4, 5 and 6.

Original: Theorem 2.

Original: Let Assumptions 4, 5 and 6 and condition (21) hold.

Original: Then there exist parameters of

Original: Algorithm 2 such that the iteration complexity for finding an ϵ -accurate solution of problem (11) is

Original: [Equation]

Original: where T a , T b , T c , T d are defined as

Original: [Equation]

Original: = max

Original: [Equation]

Original: and C > 0 is some constant, which does not depend on ϵ , but possibly depends on

Original: L x , µ x , L y , µ y , L xy , µ xy , µ yx .

Original: Consider the case when µ x , µ y > 0 .

Original: In this case the iteration complexity of Algorithm 2 becomes

Original: [Equation]

Original: This recovers the result of Cohen et al.

Original: Moreover, when µ x = µ y , this result recovers the complexity of solving problem (11) by a number of known algorithms, including the extragradient method (Korpelevich, 1976), optimistic gradient method (Daskalakis et al., 2018; Gidel et al., 2018), and the dual extrapolation method (Nesterov and Scrimali, 2006).

Original: Finally, consider then opposite case when at least one of the constants µ x and µ y is zero.

Original: To the best of our knowledge, there are no algorithms that can achieve a linear convergence.

Original: However,

Original: Algorithm 2 can still achieve linear iteration complexity provided that condition (21) is satisfied.

Original: B Proof of Lemma

Original: Part 1.

Original: Let us first show that primal problem (22) has at least a single solution x

Original: ∗ ∈ R d x .

Original: Condition (21) implies that max { µ x , µ yx } > 0 .

Original: If µ x > 0 then function P ( x ) is obviously strongly convex and primal problem indeed has a solution.

Original: Consider the opposite case µ x = 0 .

Original: Then µ yx >due to condition (21).

Original: Assumption 3 and µ yx > 0 imply that ∇ f ( x ) ∈ range A ⊤ for all x ∈ R d x .

Original: Hence,

Original: [Equation]

Original: Using the definition of P ( x ) we get

Original: [Equation]

Original: for all x ∈ R d x , x ′ ∈ ker A .

Original: From this one can conclude that

Original: [Equation]

Original: for any vector x 0 ∈ R d x .

Original: From the definition of P ( x ) it follows that P ( x ) is µ yx -strongly convex on any affine space x 0 + range A ⊤ for arbitrary x 0 ∈ R d x .

Original: Hence, problem min x ∈ x 0 +range A ⊤ P ( x ) has a unique solution and primal problem min x ∈ R dx P ( x ) has at least a single solution x ∗ .

Original: Part 2.

Original: Let us show that there exists y ∗ ∈

Original: R d y such that ( x ∗ , y ∗ ) ∈ S , i.e., ( x ∗ , y ∗ ) satisfy optimality conditions (13).

Original: Let us show that −∇ f ( x ∗ )

Original: ∈

Original: A ⊤ ∂g ∗ ( A x ∗ ) .

Original: We use condition (21) which implies max { µ y , µ xy } > 0 .

Original: If µ y > 0 , then function g ∗ ( y ) is smooth and our statement is trivial.

Original: Consider the opposite case µ y = 0 .

Original: Then µ xy > 0 due to condition (21).

Original: Assumption 3 and µ xy > 0 imply that ∇ g ( y ) ∈ range A for all y ∈ R d y .

Original: Hence, dom g ∗ ( · ) ⊂ range A .

Original: Let h ( x ) =

Original: g ∗ ( A x ) .

Original: From standard theory it follows that −∇ f ( x ∗ ) ∈

Original: ∂h ( x ∗ ) or h ( x ) ≥ h ( x ∗ ) − ⟨∇ f ( x ∗ ) , x − x ∗ ⟩ for all x ∈ R d x ,

Original: From this one can conclude that

Original: [Equation]

Original: which implies ∇ f ( x ∗ ) ∈ (ker A )

Original: ⊥ =

Original: range A ⊤ .

Original: Hence, there exists vector y ∗ ∈

Original: R d

Original: y such that −∇ f ( x ∗ )

Original: =

Original: A ⊤ y ∗ .

Original: Now, we can write h ( x

Original: [Equation]

Original: ≥ h ( x

Original: [Equation]

Original: y

Original: [Equation]

Original: , x

Original: [Equation]

Original: x ∗ ⟩ for all x ∈ R d x , which is equivalent to

Original: [Equation]

Original: The latter can be written as

Original: [Equation]

Original: But dom g ∗ ( · ) ⊂ range A , which means that g ∗ ( y ) =

Original: + ∞ for all y / ∈ range A .

Original: This implies

Original: [Equation]

Original: which is a definition of y ∗ ∈ ∂g ∗ ( A x ∗ ) .

Original: An equivalent for this is ∇ g ( y ∗ ) =

Original: A x ∗ , which together with −∇ f ( x ∗ ) =

Original: A ⊤ y ∗ form optimality condition (13).

Original: Part 3.

Original: We showed that there exists a pair of vectors ( x ∗ , y ∗ ) ∈ R d x × R d y which is a saddle point of the function F ( x, y ) in problem (1).

Original: Hence, strong duality holds and proof of the rest of Lemmais trivial.

Original: C Proof of Theorem

Original: Lemma 2.

Original: There exists a solution ( x ∗ , y ∗ ) ∈ S of the problem (1) such that for all k = 0 , 1 , 2 , . . .

Original: the iterates of Algorithm 1 satisfy

Original: [Equation]

Original: Proof.

Original: The proof of this lemma is a trivial extension of the derivations from the proof of Lemma

Original: [Equation]

Original: Lemma 3.

Original: Let τ x be defined as

Original: [Equation]

Original: Let α x be defined as

Original: [Equation]

Original: Let β x be defined as

Original: [Equation]

Original: Then, the following inequality holds:

Original: [Equation]

Original: Proof.

Original: Using Line 8 of the Algorithm 1 we get

Original: [Equation]

Original: Using the parallelogram rule we get

Original: [Equation]

Original: Using the optimality condition ∇ g ( y ∗ )

Original: =

Original: A x ∗

Original: [Equation]

Original: which follows from (13)

Original: [Equation]

Original: and the parallelogram rule we get

Original: [Equation]

Original: Using Assumption 3, equation 47 and L y -smoothness of g we get

Original: [Equation]

Original: Using the optimality condition ∇ f ( x ∗ ) + A ⊤ y ∗ =

Original: [Equation]

Original: which follows from (13)

Original: [Equation]

Original: we get

Original: [Equation]

Original: Using µ y -strong convexity of f and Lines 6 and 10 of the Algorithm 1 we get

Original: [Equation]

Original: , x k +1 − x ∗ ⟩ .

Original: Using convexity of D f ( x, x ∗ ) with respect to x

Original: [Equation]

Original: which follows from the convexity of f

Original: [Equation]

Original: we get

Original: [Equation]

Original: Using L x -smoothness of D f ( x, x ∗ ) with respect to x , which follows from the L x -smoothness of f , we get

Original: [Equation]

Original: Using Line 10 of the Algorithm 1 we get

Original: [Equation]

Original: Using the definition of τ x

Original: [Equation]

Original: α x and β x we get

Original: [Equation]

Original: Lemma

Original: [Equation]

Original: Let τ y be defined as

Original: [Equation]

Original: Let α y be defined as

Original: [Equation]

Original: Let β y be defined as

Original: [Equation]

Original: Then, the following inequality holds:

Original: [Equation]

Original: Proof.

Original: The proof is similar to the proof of the previous lemma.

Original: Lemma 5.

Original: Let η x be defined as

Original: [Equation]

Original: and let η y be defined as

Original: [Equation]

Original: where δ > 0 is a parameter.

Original: Let θ be defined as

Original: [Equation]

Original: where ρ a ( δ, σ x , σ y ) , ρ b ( δ, σ x , σ y ) , ρ c ( δ, σ x , σ y ) , ρ d ( δ, σ x , σ y ) are defined as

Original: [Equation]

Original: ρ

Original: [Equation]

Original: =

Original: [Equation]

Original: Let Ψ k be the following Lyapunov function:

Original: [Equation]

Original: Then, the following inequalities hold

Original: [Equation]

Original: Proof.

Original: After adding up (51) and (55) we get

Original: [Equation]

Original: where (LHS) is given as

Original: [Equation]

Original: f , x

Original: [Equation]

Original: y

Original: [Equation]

Original: Using Line 5 of the Algorithm 1 and Assumption 3 we get

Original: [Equation]

Original: [Equation]

Original: Using the definition of η x and η y and the fact that θ < 1 we get (LHS) ≤

Original: [Equation]

Original: Using the definition of β x and β y we get

Original: [Equation]

Original: ≤

Original: [Equation]

Original: 2 xy 2 L 2 xy (cid:41)(cid:33)η y ∥

Original: y k − y ∗ ∥+ θ 4 η

Original: [Equation]

Original: ∥

Original: [Equation]

Original: +

Original: [Equation]

Original: x

Original: [Equation]

Original: σ y −(cid:19)

Original: D

Original: [Equation]

Original: ( y

Original: [Equation]

Original: , y ∗ )

Original: +

Original: [Equation]

Original: ⟨ y k +1 − y k , A ( x k +1 − x ∗ ) ⟩ −

Original: [Equation]

Original: θ ⟨ y k − y k − 1 , A ( x k − x ∗ ) ⟩ ≤ (cid:32) 1 − max (cid:40)

Original: η x µ x , min (cid:40)

Original: η x µ 2 yx 2 L y , µ 2 yx 2 L 2 xy (cid:41)(cid:41)(cid:33)η x ∥ x k − x ∗ ∥+ (cid:32) 1 − max (cid:40)

Original: η y µ y , min (cid:40)

Original: η y µ 2 xy 2 L x , µ 2 xy 2 L 2 xy (cid:41)(cid:41)(cid:33)η y ∥

Original: y k − y ∗ ∥+ θ

Original: [Equation]

Original: k − 1 ∥

Original: [Equation]

Original: −

Original: [Equation]

Original: η y ∥

Original: y k +1 − y k

Original: ∥ 2 + (cid:18)

Original: [Equation]

Original: σ x −(cid:19)

Original: D f ( x kf , x ∗ )

Original: + (cid:18)

Original: [Equation]

Original: σ y −(cid:19)

Original: D g

Original: ( y kf , y ∗ )

Original: + 2 ⟨ y k +1 − y k , A ( x k +1 − x ∗ ) ⟩ − 2 θ ⟨ y k − y k − 1 , A

Original: ( x k − x ∗ ) ⟩ .

Original: Using the definition of θ we get (LHS) ≤ θ (cid:18)

Original: [Equation]

Original: −

Original: [Equation]

Original: After rearranging and using the definition of Ψ k we get Ψ

Original: [Equation]

Original: ≤

Original: [Equation]

Original: Ψ k .

Original: Finally, using the definition of Ψ k , η x and η y we get

Original: [Equation]

Original: Proof of Theorem 1.

Original: From (64) and (65) we can conclude that

Original: [Equation]

Original: This implies the following inequality

Original: [Equation]

Original: Hence, we can conclude that

Original: [Equation]

Original: as long as the number of iterations k satisfies

Original: [Equation]

Original: where C = Ψ 0 max { 4 η x / 3 , η y } , which does not depend on ϵ .

Original: From (58) we obtain

Original: [Equation]

Original: (cid:26) ρ a ( δ, σ x , σ y ) , ρ b ( δ, σ x , σ y ) , ρ c ( δ, σ x , σ y ) , ρ d ( δ, σ x , σ y ) (cid:27) .

Original: We can now try to approximately optimize parameters δ > 0 and σ x , σ y ∈ (0 , 1] to obtain the smallest possible values of ρ a ( δ, σ x , σ y ) − 1 , ρ b ( δ, σ x , σ y ) − 1 , ρ c ( δ, σ x , σ y ) − 1 , ρ d ( δ, σ x , σ y ) − 1 .

Original: This can be done in a closed form and the result is the following:

Original: [Equation]

Original: x

Original: [Equation]

Original: L y

Original: [Equation]

Original: L

Original: [Equation]

Original: √

Original: [Equation]

Original: y (cid:41) for

Original: [Equation]

Original: (cid:114) µ y µ x

Original: [Equation]

Original: σ x = (cid:114) µ x 2 L x , σ y = (cid:114) µ x 2 L x ,

Original: [Equation]

Original: (cid:40)(cid:112) L

Original: [Equation]

Original: L y µ xy , L xy µ xy (cid:115)

Original: L x µ x ,

Original: L 2 xy µ 2 xy (cid:41) for δ = (cid:115) µ 2 xy

Original: [Equation]

Original: µ x L x

Original: [Equation]

Original: σ x = (cid:114) µ x 2 L x

Original: [Equation]

Original: σ y = min (cid:40) 1 , (cid:115) µ 2 xy 4 L x L y (cid:41) ,

Original: [Equation]

Original: (cid:40)(cid:112)

Original: L x L y µ yx , L xy µ yx (cid:115)

Original: L y µ y ,

Original: L 2 xy µ 2 yx (cid:41) for δ = (cid:115) 2 µ y L y µ 2 yx , σ x

Original: [Equation]

Original: Note, that we set µ y = 0 in the bound for ρ − 1 b , µ x = 0 in the bound for ρ − 1 c and µ x = µ y = 0 in the bound for ρ − 1 d .

Original: This is a valid move, because any convex function is 0 -strongly convex by the definition of strong convexity.

Original: D Proof of Theorem

Original: Lemma 6.

Original: Problem (11) has a unique solution ( x ∗ , y ∗ ) .

Original: Proof.

Original: Consider operator T : R d x × R d y → R d x × R d y defined as T : ( x, y )

Original: (cid:55)→ ( x − t x ∇ x F ( x, y ) , y − t y ∇ y F ( x, y )) for some fixed t x , t y > 0 .

Original: It is obvious that ( x, y ) is a fixed point of operator T if and only if ( x, y ) is a solution to problem (11).

Original: If one can show that this operator is contractive, then it has a unique fixed point due to Banach fixed-point theorem.

Original: The proof of the fact that T is contractive is similar to the proof of the rest of Theorem 2.

Original: Lemma 7.

Original: Let η x be defined as

Original: [Equation]

Original: and let η y be defined as

Original: [Equation]

Original: where δ > 0 is a parameter.

Original: Let θ be defined as

Original: [Equation]

Original: where ρ a ( δ ) , ρ b ( δ ) , ρ c ( δ ) , ρ d ( δ ) are defined as

Original: [Equation]

Original: Let Ψ k be the following Lyapunov function:

Original: [Equation]

Original: Then, the following inequalities hold

Original: [Equation]

Original: Proof.

Original: Using Line 5 of the Algorithm 2 we get.

Original: [Equation]

Original: x k −

Original: [Equation]

Original: y k

Original: [Equation]

Original: [Equation]

Original: x k

Original: [Equation]

Original: [Equation]

Original: ∗ ⟩

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: Using the Assumption 4 we get

Original: [Equation]

Original: Using the Assumption 5 we get

Original: [Equation]

Original: Similarly

Original: [Equation]

Original: we can obtain the following upper-bound on 1 η y ∥

Original: y k +1 − y ∗ ∥ 2 :

Original: [Equation]

Original: x k +1 , y ∗ )) .

Original: Summing up the upper-bounds on 1 η x ∥ x k +1 − x ∗ ∥ 2 and 1 η y ∥

Original: y k +1 − y ∗ ∥ 2 gives

Original: [Equation]

Original: where (LHS) is defined as

Original: [Equation]

Original: The Assumption 4 states

Original: [Equation]

Original: that function F ( x, y ) is L x

Original: -smooth in x and L y -smooth in y .

Original: Hence, using the optimality conditions (40) we get

Original: [Equation]

Original: where δ x

Original: [Equation]

Original: δ y ∈ (0 , 1] are some parameters, that will be defined later.

Original: Using the Assumption 6 we get

Original: [Equation]

Original: LHS) ≤

Original: [Equation]

Original: Using Lines 5 and 6 of the Algorithm 2and the Lipschitzness property of ∇ x F ( x, y ) and ∇ y F ( x, y ) we get

Original: [Equation]

Original: Now

Original: [Equation]

Original: we set δ x = min { 1 , c x η x L x } , δ y = min { 1 , c y η y L y } , where c x , c y > 0 will be defined later

Original: [Equation]

Original: and obtain

Original: [Equation]

Original: [Equation]

Original: Using the definition of η x and η y we get

Original: [Equation]

Original: ≤

Original: [Equation]

Original: Now

Original: [Equation]

Original: we choose c x = c y = 14 and get

Original: [Equation]

Original: Using the definition of η x and η y we get (LHS) ≤ (cid:32)η x − µ x − δ y µ 2 yx 2 L y (cid:33)

Original: [-∥ x k − x ∗ ∥ 
-]
Revised: [-∥ x k − x ∗ ∥ 
-]

Original: + (cid:32)η y − µ y − δ x µ 2 xy 2 L x (cid:33) ∥

Original: y k − y ∗ ∥− 2 θ ⟨∇ x F

Original: [-( x k − 1 , y k ) − ∇ x F ( x k − 1 , y k − 1 ) , x k − x ∗ ⟩ + 5 θ 2 
-]
Revised: [-( x k − 1 , y k ) − ∇ x F ( x k − 1 , y k − 1 ) , x k − x ∗ ⟩ + 5 θ 2 
-]

Original: η y ∥

Original: y k − y k − 1 ∥ 2 − 16 η y ∥

Original: y k +1 − y k

Original: ∥ 2 .

Original: Using the definition of δ x and δ y we get (LHS) ≤ (cid:32)

Original: 1 − max (cid:40)

Original: η

Original: [Equation]

Original: , min (cid:40) η

Original: [Equation]

Original: µ

Original: [Equation]

Original: L y , η

Original: [Equation]

Original: η

Original: [Equation]

Original: µ

Original: [Equation]

Original: yx (cid:41)(cid:41)(cid:33)η x

Original: [Equation]

Original: x

Original: [Equation]

Original: − x

Original: [Equation]

Original: ∥+ (cid:32) 1 − max (cid:40)

Original: η

Original: [Equation]

Original: µ y , min (cid:40) η y µ 2 xy 2 L x , η y η x µ 2 xy (cid:41)(cid:41)(cid:33)η y ∥

Original: y k − y ∗ ∥− 2 θ ⟨∇ x F

Original: [-( x k − 1 , y k ) − ∇ x F ( x k − 1 , y k − 1 ) , x k − x ∗ ⟩ + 5 θ 2 
-]
Revised: [-( x k − 1 , y k ) − ∇ x F ( x k − 1 , y k − 1 ) , x k − x ∗ ⟩ + 5 θ 2 
-]

Original: η y ∥ y k − y k − 1 ∥ 2 − 16 η y ∥ y k +1 − y k ∥ 2 .
Revised: 2021 No. 70-2021-00142.

Original: Using the definition of η x , η y and θ we get

Original: [Equation]

Original: After rearranging and using the definition of Ψ k we get

Original: [Equation]

Original: Finally, using the definition of Ψ k

Original: [Equation]

Original: η x and η y we get

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Proof of Theorem 2.

Original: From (74) and (75) we can conclude that

Original: [Equation]

Original: This implies the following inequality

Original: [Equation]

Original: Hence, we can conclude that

Original: [Equation]

Original: as long as the number of iterations k satisfies

Original: [Equation]

Original: where C = Ψ 0 max { 4 η x / 3 , η y } , which does not depend on ϵ .

Original: From (68) we obtain

Original: [Equation]

Original: Now, we find the parameter δ to obtain the following upper bounds on ρ a ( δ ) , ρ b ( δ ) , ρ c ( δ ) , ρ d ( δ ) :

Original: [Equation]

Original: δ = (cid:114)

Original: [Equation]

Original: [Equation]

Original: max

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: µ

Original: [Equation]

Original: (cid:41) for δ

Original: [Equation]

Original: min

Original: [Equation]

Original: (cid:112) µ

Original: [Equation]

Original: L

Original: [Equation]

Original: µ yx

Original: [Equation]

Original: (cid:114) L

Original: [Equation]

Original: L

Original: [Equation]

Original: (cid:41)

Original: [Equation]

Original: [Equation]

Original: max

Original: [Equation]

Original: L

Original: [Equation]

Original: µ

Original: [Equation]

Original: yx , 128 L 2 xy µ 2 xy , 128 L 2 xy µ 2 yx (cid:41) for δ

Original: [Equation]