Original: Convergence Analysis of Linear Coupling with Inexact Proximal Operator
Revised: Convergence Analysis of Linear Coupling with Inexact Proximal Operator

Original: Abstract
Revised: Abstract

Original: Linear coupling is recently proposed to accelerate first-order algorithms by linking gradient descent and mirror descent together, which is able to achieve the accelerated convergence rate for first-order algorithms.
Revised: Linear coupling is recently proposed to accelerate ﬁrst-order algorithms by linking gradient descent and mirror descent together, which is able to achieve an accelerated convergence rate.

Original: This work focuses on the convergence analysis of linear coupling for convex composite minimization when the proximal operator cannot be exactly computed.
Revised: This work focuses on the convergence analysis of linear coupling for convex composite minimization when a proximal operator cannot be exactly computed.

Original: It is of particular interest to study the convergence of linear coupling because it not only achieves the accelerated convergence rate for first-order algorithm but also works for generic norms.
Revised: It is of particular interest to study the convergence of linear coupling because it not only achieves the accelerated convergence rate for ﬁrst-order algorithms but also works for generic norms.

Original: We present convergence analysis of linear coupling by allowing the proximal operator to be computed up to a certain precision.
Revised: We present convergence analysis of linear coupling by allowing the proximal operator to be computed up to a certain precision.

Original: Our analysis illustrates that the accelerated convergence rate of linear coupling with  inexact proximal operator can be preserved if the error sequence of  inexact proximal operator decreases in a sufficiently fast rate.
Revised: Our analysis illustrates that the accelerated convergence rate of linear coupling with an inexact proximal operator can be preserved if the error sequence of the inexact proximal operator decreases in a sufﬁciently fast rate.

Original: More importantly, our analysis leads to better bounds than existing works on inexact proximal operator.
Revised: More importantly, our analysis leads to better bounds than existing works with inexact proximal operators.

Original: Experiment results on several real-world datasets verify our theoretical results.
Revised: Experiment results on several real-world datasets verify our theoretical results.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: In this work, we considerthe convex composite minimization of the form 
Revised: In this work, we consider convex composite minimization problems in the form of

Original: [Equation]
Revised: [Equation]

Original: wherefunction g is convex or µ -strongly convex, and Lsmooth, and h is convex but possibly non-smoothfunction [Nesterov, 2005].
Revised: where g is convex or µ -strongly convex, and L -smooth, and h is convex but possibly non-smooth [Nesterov, 2005].

Original: Various machine learning problems can be formulated in the form of (1), where g defines a convex loss function for training examples, and h regularizes the model to promote a specified structure [Bach et al., 2011, Sra et al., 2012, Jenatton et al., 2011].
Revised: Various machine learning problems can be formulated in the form of (1), where g deﬁnes a convex loss function for training examples, and h regularizes the model to promote a speciﬁed structure [Bach et al., 2011, Sra et al., 2012, Jenatton et al., 2011].

Original: For instance, it is well-known that h ( x ) = ∥ x ∥ 1 can be used to induce  sparse structure for x [Tibshirani, 1996].
Revised: For instance, it is well-known that h ( x ) = (cid:107) x (cid:107) 1 can be used to induce a sparse structure for x [Tibshirani, 1996].

Original: In addition, constrained optimization problem can also be formulated in the form of (1) through reformulation.
Revised: In addition, constrained optimization problems can also be formulated in the form of (1) through reformulation.

Original: Specifically, for any convex optimization problem with constraint x ∈ C , it can be reformulated as (1) by defining h ( x ) def = I C ( x ) as the indicator function of the convex set C , where I C ( x )
Revised: Speciﬁcally, for any convex optimization problem with constraint x ∈ C , it can be reformulated as (1) by deﬁning h ( x ) def = I C ( x ) as the indicator function of the convex set C , where I C ( x )

Original: = 

Revised: =

Original: if x ∈ C or I C ( x ) = ∞ otherwise.
Revised: if x ∈ C or I C ( x ) = ∞ otherwise.

Original: To solve (1), first-order algorithms are the most popular choice due to their simplicity and generality [Sra et al., 2012, Nesterov, 2013].
Revised: To solve (1), ﬁrst-order algorithms are the most popular choice due to their simplicity and generality [Sra et al., 2012, Nesterov, 2013].

Original: First-order algorithms generally assume that the first-order gradient of a smooth function can be queried by a black-box in constant time.
Revised: First-order algorithms generally assume that the ﬁrst-order gradient of a smooth function can be queried by a black-box in constant time.

Original: Therefore, the complexity for solving a smooth and convex function f is measured by the number of times that first-order gradient is queried to produce a sequence { x k } Tk =1 s.t. f
Revised: Therefore, the complexity for solving a smooth and convex function f is measured by the number of times that ﬁrst-order gradients are queried to produce a sequence { x k } Tk =1 s.t. f

Original: ( x T ) − f ( x ⋆ ) ≤ ϵ , where x ⋆ is the optimal solution.
Revised: ( x T ) − f ( x (cid:63) ) ≤ (cid:15) , where x (cid:63) is the optimal solution.

Original: The gradient descent algorithm has O ( L/ϵ ) iteration complexity for L -smooth convex minimization problems, which improves to O ( L/µ log(1 /ϵ ))
Revised: A gradient descent algorithm has O ( L/(cid:15) ) iteration complexity for L -smooth convex minimization problems, which can be improved to O ( L/µ log(1 /(cid:15) ))

Original: when the function is µ -strongly convex and L -smooth [Nesterov, 2013].
Revised: when the function is µstrongly convex and L -smooth [Nesterov, 2013].

Original: However, these complexities are not optimal, which leaves big space for improvement [Nemirovsky and Yudin, 1983].
Revised: However, these complexities are not optimal, which leaves big space for improvement [Nemirovsky and Yudin, 1983].

Original: In the seminal work [Nesterov, 1983], Nesterov laid the foundation of accelerated gradient methods (AGDs) for convex and L -smooth functions (i.e. h ( x ) = [-
-] in (1)).
Revised: In the seminal work [Nesterov, 1983], Nesterov laid the foundation of accelerated gradient methods (AGDs) for convex and L -smooth functions (i.e. h ( x ) = [-
-]0 in (1)).

Original: Specifically, the method proposed in [Nesterov, 1983] successfully improves the complexities of gradient descent  to O ( (cid:112) L/ϵ ) and O ( (cid:112) L/µ log(1 /ϵ )) for generallyand µ -strongly convex and smooth functions, respectively, which are accelerated and optimal for first-order algorithms [Nemirovsky and Yudin, 1983, Nesterov, 2013].
Revised: Speciﬁcally, the method proposed in [Nesterov, 1983] successfully improves the complexities of gradient descent methods to [Equation] and O ( (cid:112) L/µ log(1 /(cid:15) )) for generally µ -strongly convex and smooth functions, respectively, which are accelerated and optimal for ﬁrst-order algorithms [Nemirovsky and Yudin, 1983, Nesterov, 2013].

Original: Since then, accelerated first-order algorithms with the optimal convergence rate have been further developed to solve convex composite minimization problems [Tseng, 2008, Lin et al., 2017].
Revised: Since then, accelerated ﬁrst-order algorithms with the optimal convergence rate have been further developed to solve convex composite minimization problems [Tseng, 2008, Lin et al., 2017].

Original: However, the interpretation of the proof of acceleration and intuitions behind the convergence analysis in Nesterov’s
Revised: However, the interpretation of the proof of acceleration and intuitions behind the convergence analysis in Nesterov’s

Original: AGDs are not clear.
Revised: AGDs are not clear.

Original: Many efforts have been devoted to present  clear interpretation for Nesterov’s AGDs, or alternatively to develop new accelerated and interpretable variants with the optimal convergence rate [Bubeck et al., 2015, Su et al., 2014, Krichene et al., 2015, Diakonikolas andOrecchia, 2018].
Revised: Many efforts have been devoted to present a clear interpretation for Nesterov’s AGDs, or alternatively to develop new accelerated and interpretable variants with the optimal convergence rate [Bubeck et al., 2015, Su et al., 2014, Krichene et al., 2015, Diakonikolas and

Original:  Recently, Allen-Zhu and Orecchia [Allen Zhu and Orecchia, 2017] proved that the optimal convergence rate can be achieved for smooth and convex optimization problems by linearly coupling two fundamental first-order algorithms (namely gradient descent and mirror descent).
Revised: Orecchia, 2018]. Recently, Allen Zhu and Orecchia [2017] proved that the optimal convergence rate can be achieved for smooth and convex optimization problems by linearly coupling two fundamental ﬁrst-order algorithms (namely gradient descent and mirror descent).

Original: Later, the extension of linear coupling to generally convex composite minimization was presented in [Rodomanov, 2016].
Revised: Later, Rodomanov [2016] extended linear coupling to generally convex composite minimization problems.

Original: As a variant of Nesterov’s AGDs, linear coupling (LC) elegantly extends AGDs to non-Euclidean norms, which is important for many applications.
Revised: As a variant of Nesterov’s AGDs, linear coupling (LC) elegantly extends AGDs to non-Euclidean norms, which is important for many applications.

Original: In [Allen Zhu and
Revised: In [Allen Zhu and

Original: Orecchia, 2017, Appendix A.1],Allen-Zhu presented several concrete examples  to illustrate the importance of allowing non-Euclidean norms in first-order algorithms.
Revised: Orecchia, 2017, Appendix A.1], several concrete examples have been presented to illustrate the importance of allowing non-Euclidean norms in ﬁrst-order algorithms.

Original: For example, one prefers to use the ℓ 1 norm instead of ℓ 2 norm gradient descent for the saddle point problem min x ∈ ∆ n max y ∈ ∆ m y ⊤ Ax , where ∆ n and ∆ m denotes the unit simplex in R n and R m , respectively, andA ∈
Revised: For example, one prefers to use the (cid:96) 1 norm instead of the (cid:96) 2 norm gradient descent for the saddle point problem min x ∈ ∆ n max y ∈ ∆ m y (cid:62) Ax , where ∆ n and ∆ m denotes the unit simplex in R n and R m , respectively, and

Original:  R m × n with all the entries are between [ − 1 , 1] .
Revised: A ∈ R m × n with all the entries being in [ − 1 , 1] .

Original: In this problem, there are two reasons for choosing ℓ 1 norm gradient descent.
Revised: In this problem, there are two reasons for choosing the (cid:96) 1 norm gradient descent as follows.

Original: 1) In order to apply ℓ 2 norm gradient descent, one needs to further assume the square ℓ 2 norm of each row of A is upper bounded by 1 .
Revised: 1) In order to apply the (cid:96) 2 norm gradient descent, one needs to further assume the square (cid:96) 2 norm of each row of A is upper bounded by 1 .

Original: Obviously, it is a stronger condition and harder to satisfy than that of ℓ 1 norm.
Revised: Obviously, it is a stronger condition and harder to satisfy than that of the (cid:96) 1 norm.

Original: 2) More importantly, even the stronger condition is satisfied, the ℓ 2 norm also leads to a larger value for L , thus the ℓ 1 norm gradient descent has faster convergence [Nesterov, 2005].
Revised: 2) More importantly, even the stronger condition is satisﬁed, the (cid:96) 2 norm also leads to a larger value for L , thus the (cid:96) 1 norm gradient descent has faster convergence [Nesterov

Original: Another example is near-linear time maximum flow in which one need to apply ℓ ∞ gradient descent [Kelner et al.
Revised: Another example is near-linear time maximum ﬂow in which one need to apply (cid:96) ∞ gradient descent [Kelner et al., 2014].

Original: For convex composite optimization problems (1), the proximal gradient descent [Parikh and Boyd, 2014] performs the following updates:
Revised: For convex composite optimization problems (1), the proximal gradient descent [Parikh and Boyd, 2014] performs the following updates:

Original: [Equation]
Revised: [Equation]

Original: where prox ηh ( · ) is the proximal operator [Combettes and
Revised: where prox ηh ( · ) is the proximal operator [Combettes and

Original: Pesquet, 2011] of ηh ( x ) defined for any scalar η > 0 as the unique solution of
Revised: Pesquet, 2011] of ηh ( x ) deﬁned for any scalar η > 0 as the unique solution of

Original: [Equation]
Revised: [Equation]

Original: If h is considerably simple (e.g., h ( x ) = ∥ x ∥ 1 ), there is an analytical solution for x k +1 [Combettes and Pesquet, 2011].
Revised: If h is considerably simple (e.g., h ( x ) = (cid:107) x (cid:107) 1 ), there is an analytical solution for x k +1 [Combettes and Pesquet, 2011].

Original: However, in more general cases, it is challenging to obtain the exact solution of the proximal operator possibly due to  two reasons.
Revised: However, in more general cases, it is challenging to obtain the exact solution of the proximal operator possibly due to the following two reasons.

Original:  First, the proximal operator does not admit an analytical solution.
Revised: • First, the proximal operator does not admit an analytical solution.

Original: For example, there is no closed form solution for the proximal operator if h is the isotropic total variation regularization [Beck and Teboulle, 2009].
Revised: For example, there is no closed form solution for the proximal operator if h is the isotropic total variation regularization [Beck and Teboulle, 2009].

Original: In this case, the proximal operator can only be solved by employing some optimization algorithms up to a certain precision.
Revised: In this case, the proximal operator can only be solved by employing some optimization algorithm up to a certain precision.

Original: More details of this example can be found inour Experiments .
Revised: More details of this example can be found in Experiments .

Original: • Second, it might be computationally expensive to obtain the exact solution.
Revised: Second, it may be computationally expensive to obtain the exact solution.

Original: For instance, for optimization problems with an ℓ 1 norm ball constraint (e.g., h ( x ) =
Revised: For instance, for optimization problems with an (cid:96) 1 norm ball constraint (e.g., h ( x ) =

Original: I ∥ x ∥ 1 ≤ r ( x ) , r ∈ R + ), the complexity of exactly performing the proximal operator is O
Revised: I (cid:107) x (cid:107) 1 ≤ r ( x ) , r ∈ R + ), the complexity of exactly performing the proximal operator is O

Original: ( d log d ) [Duchi et al., 2008, Liu and Ye, 2009].
Revised: ( d log d ) [Duchi et al., 2008, Liu and Ye, 2009].

Original: Therefore, it is highly demanding in computation for high-dimensional learning problems as d is considerably large.
Revised: Therefore, it is highly demanding in computation for high-dimensional cases as d is considerably large.

Original: Indeed, we empirically compare the efficiency of exact and inexact proximal operator of ℓ 1 norm ball constraint.
Revised: We empirically compare the efﬁciency of exact and inexact proximal operator of the (cid:96) 1 norm ball constraint.

Original: The result suggests that  inexact proximal operator outperforms the exact counterpart by carefully controlling the error sequences in this application.
Revised: The result suggests that the inexact proximal operator outperforms the exact counterpart by carefully controlling the error sequences in this application.

Revised: In [Allen Zhu and Orecchia, 2017], the objective is assumed to be convex and smooth, i.e., h ( x ) = 0 .

Revised: Thus, the analysis of linear coupling does not involve the computation of proximal operator.

Revised: However, it is well-known that many machine learning problems can be formulated as a convex composite minimization problem due to the non-smooth regularization.

Original: Therefore, it is of particular interest to characterize the convergence behavior of linear coupling, when the proximal operator is not exactly solved.
Revised: Therefore, it is of particular interest to characterize the convergence behavior of linear coupling, when the proximal operator is not exactly solved.

Original: However, existing analyses [Schmidt et al., 2011] on inexact proximal operator only cover the ℓ 2 norm, thus it does not applicable to linear coupling due to non-Euclidean norm.
Revised: However, existing analyses [Schmidt et al., 2011] on inexact proximal operators only cover the (cid:96) 2 norm case, thus they are not applicable to linear coupling due to non-Euclidean norms.

Original: To this end, we present a complete study for the convergence rate of linear coupling with inexact proximal operator.
Revised: To this end, we present a complete study for the convergence rate of linear coupling with inexact proximal operators.

Original: Compared with existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019], our focus is the convergence analysis of linear coupling with inexact proximal operator, which presents new challenges due to the generic Bregman divergence (refer to Definition 3).
Revised: Compared with existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019], our focus is the convergence analysis of linear coupling with inexact proximal operators, which presents new challenges due to the generic Bregman divergence (refer to Deﬁnition 3).

Original: In particular, the key step for analyzing inexact proximal operator is to bound the subgradient of inexact solution.
Revised: In particular, the key step for analyzing inexact proximal operators is to bound the subgradient of inexact solution.

Original: It is an easy task in the case of squared Euclidean distance as the bounding problem has an analytical solution [Schmidt et al., 2011].
Revised: It is an easy task in the case of squared Euclidean distance as the bounding problem has an analytical solution [Schmidt et al., 2011].

Original: In contrast, it does not admit an analytical form in our case due to the generic Bregman divergence (refer to Definition 1), e.g., Kullback-Leibler divergence.
Revised: In contrast, it does not admit an analytical form in our case due to the generic Bregman divergence, e.g., the KullbackLeibler divergence.

Original: To address it, we present a relaxation method for the bounding problem so that the subgradient still can be bounded (see Lemma 1).
Revised: To address it, we present a relaxation method for the bounding problem so that the subgradient still can be bounded (see Lemma 1).

Original: More importantly, our analysis leads to tighter bounds (refer to Remarks 2 and 5 for details) than previous works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].
Revised: More importantly, our analysis leads to tighter bounds (refer to Remarks 2 andfor details) than previous works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].

Original:  NOTATION AND PRELIMINARIES
Revised:  NOTATION AND PRELIMINARIES

Original: Throughout this paper, vectors and matrices are denoted by lower-case and upper-case boldface characters (e.g., x and
Revised: Throughout this paper, vectors and matrices are denoted by lower-case and upper-case boldface characters (e.g., x and

Original: X ), respectively.
Revised: X ), respectively.

Original: Let 0 be a vector or matrix with all its entries equal to 0 .
Revised: Let 0 be a vector or matrix with all its entries equal to 0 .

Original: For both x , y ∈ R d , their inner product is denoted by ⟨ x , y ⟩ = (cid:80) di
Revised: For both x , y ∈ R d , their inner product is denoted by (cid:104) x , y (cid:105) = (cid:80) di

Original: =1 x i y i .
Revised: =1 x i y i .

Original: Let ∥ · ∥ be a generic norm and its dual norm is denoted by ∥ · ∥ ∗ that is defined as ∥ y ∥ ∗ = sup x (cid:8) ⟨ x , y ⟩ | ∥ x ∥ ≤ 1 } .
Revised: Let (cid:107) · (cid:107) be a generic norm and its dual norm is denoted by (cid:107) · (cid:107) ∗ that is deﬁned as (cid:107) y (cid:107) ∗ = sup x (cid:8) (cid:104) x , y (cid:105) | (cid:107) x (cid:107) ≤ 1 } .

Original: Let ∥ · ∥ 1 and ∥ · ∥denote the ℓ 1 and ℓ 2 norm, respectively.
Revised: Let (cid:107) · (cid:107) 1 and (cid:107) · (cid:107)denote the (cid:96) 1 and the (cid:96) 2 norm, respectively.

Original: Definition 1.
Revised: Deﬁnition 1.

Original: A function f is L -smooth w.r.t.
Revised: A function f is L -smooth w.r.t.

Original: ∥ · ∥ , if 
Revised: (cid:107) · (cid:107) , if f ( y ) ≤ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105)

Original: [Equation]
Revised: + L 2 (cid:107) y − x (cid:107) 2 , ∀ x , y .

Original: Definition 2.
Revised: Deﬁnition 2.

Original: A function f is µ -strongly convex w.r.t.
Revised: A function f is µ -strongly convex w.r.t.

Original: ∥ · ∥ , if 
Revised: (cid:107) · (cid:107) , if f ( y ) ≥ f

Original: [Equation]
Revised: ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + µ 2 (cid:107) y − x (cid:107) 2 , ∀ x , y .

Original: Definition 3.
Revised: Deﬁnition 3.

Original: Let ψ : Q → R be a strictly convex and continuously differentiable function.
Revised: Let ψ : Q → R be a strictly convex and continuously differentiable function.

Original: Then, the Bregman divergence is
Revised: Then, the Bregman divergence is

Revised: V ψ ( y , x ) def =

Original: [Equation]
Revised: [Equation]

Original: The Definition 3 implies V ψ ( x , x ) = 0 , and V ψ ( y , x ) ≥ ρ 2 ∥ x − y ∥ 2 if ψ is ρ -strongly convex w.r.t.
Revised: ( y [Equation] − ψ ( x ) − (cid:104)∇ ψ ( x ) , y − x (cid:105) , ∀ x , y ∈ Q . Deﬁnition 3 implies V ψ ( x , x ) = 0 , and V ψ ( y , x ) ≥ ρ 2 (cid:107) x − y (cid:107) 2 if ψ is ρ -strongly convex w.r.t.

Original: ∥ · ∥ .
Revised: (cid:107) · (cid:107) .

Original: The Bregman divergence includes many well-known examples.
Revised: The Bregman divergence includes many well-known examples.

Original: If ψ ( x ) def = 12 ∥ x ∥ 22 , then V ψ ( y , x ) is the squared Euclidean distance V ψ ( y , x )
Revised: If ψ ( x ) def = 12 (cid:107) x (cid:107) 22 , then V ψ ( y , x ) is the squared Euclidean distance V ψ ( y , x )

Original: = [-
-]
Revised: = [-
-]12 (cid:107) x

Original: ∥ x − y ∥ 22 .
Revised: − y (cid:107) 22 .

Original: If Q def = (cid:8) x ∈ R d + | (cid:80) i x i =1 (cid:9) and ψ ( x )
Revised: If Q def = (cid:8) x ∈ R d + | (cid:80) i x i = 1 (cid:9) and ψ ( x )

Original: def = (cid:80) i x i log x i , then V ψ ( y , x ) becomes the Kullback-Leibler divergence V ψ ( y , x )
Revised: def = (cid:80) i x i log x i , then V ψ ( y , x ) becomes the KullbackLeibler divergence V ψ ( y , x )

Original: = (cid:80) i
Revised: = (cid:80) i

Original: y i log( y i x i ) between two probability distributions x and y .
Revised: y i log( y i x i ) between two probability distributions x and y .

Original: In particular, ψ ( x ) in this case is 1 -strongly convex w.r.t.
Revised: In particular, ψ ( x ) in this case is 1 -strongly convex w.r

Original: ∥ · ∥ 1 that leads to V ψ ( y , x ) ≥ ∥ x − y ∥ 21 .
Revised: t. (cid:107) · (cid:107) 1 that leads to V ψ ( y , x ) ≥ (cid:107) x − y (cid:107) 21 .

Original: In this case, one needs to employ optimization methods that are applicable for ℓ 1 norm.
Revised: In this case, one needs to employ optimization methods that are applicable for the

Original:  Therefore, this example illustrates the importance of convergence analysis for linear coupling with inexact proximal operator since it works for a generic norm.
Revised: (cid:96) 1 norm. Therefore, this example illustrates the importance of convergence analysis for linear coupling with inexact proximal operators since it works for a generic norm.

Original: Definition 4.
Revised: Deﬁnition 4.

Original: For function f ( x ) , its convex conjugate is defined as
Revised: For function f ( x ) , its convex conjugate is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  LINEAR COUPLING WITH INEXACT PROXIMAL OPERATOR
Revised:  LINEAR COUPLING WITH INEXACT PROXIMAL OPERATOR

Original: For smooth and convex functions, the accelerated convergence rate can be obtained by two non-accelerated algorithms: linearly coupling gradient descent and mirror descent [Allen Zhu and Orecchia, 2017].
Revised: For smooth and convex functions, the accelerated convergence rate can be obtained by two non-accelerated algorithms: linearly coupling gradient descent and mirror descent [Allen Zhu and Orecchia, 2017].

Original: In fact, it can alsobe extended to solve convex composite minimization problem (1) [Rodomanov, 2016]. We assume that g is convex ( µ = 0 ) or µ -strongly convex ( µ > 0 ), and L -smooth w.r.t. ∥ · ∥ .
Revised: In fact, it can also

Original: Here, we present the extension of linear coupling (LC) for (1) when g is either convex ( µ = 0 ) or strongly convex

Original: 1: Input : x 0 , α 0 , µ, L 2: Initialization : y 0 ← x 0 , z 0 ← x3: for k = 0 to T − 1 do 4: if µ = 

Revised: 1: Input : x 0 , α 0 , µ, L 2: Initialization : y 0 ← x 0 , z 0 ← x3: for k = 0 to T − 1 do 4: if µ =

Original: then 5:
Revised: then 5:

Original: Set
Revised: Set

Original: η k +1 ← k +22 L and τ k ← 1 Lη k
Revised: η k +1 ← k +22 L and τ k ← 1 Lη k

Original: +1 6:
Revised: +1 6:

Original: Set w k +1 ← z k 7: else 8:
Revised: Set w k +1 ← z k 7: else 8:

Original: Set
Revised: Set

Original: η k +1 ← 1 Lα k +

Revised: η k +1 ← 1 Lα k +

Original: and τ k ← Lα k +1 − µ L − µ where α k + is obtained via α 2 k +1 =(1 − α k +1 )
Revised: and τ k ← Lα k +1 − µ L − µ where α k + is obtained via α 2 k +1 =(1 − α k +1 )

Original: α 2 k
Revised: α 2 k

Original: + µL α k + 9:
Revised: + µL α k + 9:

Original: Set w k +1 ← τ k α k +

Revised: Set w k +1 ← τ k α k +

Original: z k + (cid:0) [Equation]{+1 − τ k α k +
+}
Revised: z k + (cid:0) {+1 − τ k α k +
+}

Original:  10: end if 11: [Equation]
Revised: (cid:1) y k 10: end if 11: x k +1 ← τ k z k + (1 − τ k ) y k 12:

Original: Find a ξ k +1 -suboptimal solution y k
Revised: Find a ξ k +1 -suboptimal solution y k

Original: +1 for (3) 13:
Revised: +1 for (3) 13:

Original: Find a ξ k +1 -suboptimal solution z k +1 for (4) 14: end for 15: Output : y T  ( µ > 0 )  and summarize the high-level idea in Algorithm 1.
Revised: Find a ξ k +1 -suboptimal solution z k +1 for (4) 14: end for 15: Output : y T be extended to solve convex composite minimization problem (1) [Rodomanov, 2016]. We assume that g is convex ( µ = 0 ) or µ -strongly convex ( µ > 0 ), and L -smooth w.r.t.

Revised: (cid:107) · (cid:107) .

Revised: Here, we present the extension of linear coupling (LC) for (1) when g is either convex ( µ = 0 ) or strongly convex ( µ > 0 ) and summarize the high-level idea in Algorithm 1.

Original: Missing proofs can be found inthe Appendix.
Revised: Missing proofs can be found in Appendix.

Original: Let y k
Revised: Let y k

Original: and z k be the outputs of gradient descent and mirror descent in the ( k − 1) -th iteration, respectively.
Revised: and z k be the outputs of gradient descent and mirror descent in the ( k − 1) -th iteration, respectively.

Original: The key idea of linear coupling is to combine y k and z k together by a linear coupling rate τ k as the starting point for the next iteration such that the accelerated convergence rate can be achieved.
Revised: The key idea of linear coupling is to combine y k and z k together by a linear coupling rate τ k as the starting point for the next iteration such that the accelerated convergence rate can be achieved.

Original: Define x k [Equation] =
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: k z k
Revised: k +

Original: [Equation]
Revised: [Equation]

Original:  k ) y k .
Revised: 1 − τ k ) y k .

Original: In linear coupling, the gradient descent performs the following update:
Revised: In linear coupling, the gradient descent performs the following update:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where (cid:101) Q k
Revised: where (cid:101) Q k

Original: +

Revised: +

Original: ( y ; x k +1 ) is define as
Revised: ( y ; x k +1 ) is deﬁne as

Original: [Equation]
Revised: [Equation]

Original: Define η k +1 , τ k and w k +1 as in Algorithm 1, the mirror descent performs the following update:
Revised: Deﬁne η k +1 , τ k and w k +1 as in Algorithm 1, the mirror descent performs the following update:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Unlike standard gradient descent and mirror descent, the linear coupling takes the gradient at x k +1 instead of y k or z k to obtain y k +1 and z k +1 .
Revised: Unlike standard gradient descent and mirror descent [Equation] the linear coupling takes the gradient at x k +1 instead of y k or z k to obtain y k +1 and z k +1 .

Original: In this way, the gradient descent and mirror descent are coupled together to solve (1), which is able to achieve the optimal convergence rate Allen
Revised: In this way [Equation] the gradient descent and mirror descent are coupled together to solve (1), which is able to achieve the optimal convergence rate [Allen

Original: Zhu and Orecchia [2017].
Revised: Zhu and Orecchia, 2017].

Original: 3.1 THE ξ k +1 -SUBOPTIMAL SOLUTION
Revised: 3.1 THE ξ k +1 -SUBOPTIMAL SOLUTION

Original: The proximal operator is involved in both (3) and (4) due to the non-smooth  h .
Revised: The proximal operator is involved in both (3) and (4) due to the non-smooth function h .

Original: It is worth noting that the proximal distance in both (3) and (4) are defined based a generic norm ∥·∥ instead of ∥·∥ 2 .
Revised: It is worth noting that the proximal distance in both (3) and (4) are deﬁned based a generic norm (cid:107) · (cid:107) instead of (cid:107) · (cid:107) 2 .

Original: Specifically, they are the squared norm ∥ · ∥ 2 and the general Bregman divergence V ψ ( · , · ) , respectively.
Revised: Speciﬁcally, they are the squared norm (cid:107) · (cid:107) 2 and the general Bregman divergence V ψ ( · , · ) , respectively.

Original: In contrast, existing works mainly focus on the case of  squared Euclidean distance ( ℓ 2 norm).
Revised: In contrast, existing works mainly focus on the case of the squared Euclidean distance (the (cid:96) 2 norm).

Original: Therefore, (3) and (4) are more challenging to solve than existing works.
Revised: Therefore, (3) and (4) are more challenging to solve than existing works.

Original: In other words, it is often the case that (3) and (4) can only be solved up to a certain precision.
Revised: In other words, it is often the case that (3) and (4) can only be solved up to a certain precision.

Original: Therefore, it is critical to study the convergence rate of Algorithm 1 by allowing the proximal operator of the form (3) and (4) to be solved approximately.
Revised: Therefore, it is critical to study the convergence rate of Algorithm 1 by allowing the proximal operator of the form (3) and (4) to be solved approximately.

Original: For the case of inexact proximal operator, we introduce ξ k +1 -suboptimal solution to analyze the convergence rate of
Revised: For the case of inexact proximal operators, we introduce ξ k +1 -suboptimal solution to analyze the convergence rate of

Original: Algorithm 1.
Revised: Algorithm 1.

Original: We assume that h is equipped with an oracle such that the proximal operator can be computed up to a certain precision.
Revised: We assume that h is equipped with an oracle such that the proximal operator can be computed up to a certain precision.

Original: Specifically, given a non-negative ξ k +1 , the oracle is able to produce y k +1 and z k +1 such that
Revised: Speciﬁcally, given a non-negative ξ k +1 , the oracle is able to produce y k +1 and z k +1 such that

Original: [Equation]
Revised: [Equation]

Original: If ξ k +1 = 0 , it implies that the proximal operator is exactly solved for both (3) and (4).
Revised: If ξ k +1 = 0 , it implies that the proximal operator is exactly solved for both (3) and (4).

Original: Otherwise, it means that the proximal operator is solved up to a certain precision controlled by ξ k +1 .
Revised: Otherwise, it means that the proximal operator is solved up to a certain precision controlled by ξ k +1 .

Original: Thus, y k +1 and z k +1 are referred to as ξ k +1 -suboptimal solutions to (3) and (4), respectively.
Revised: Thus, y k +1 and z k +1 are referred to as ξ k +1 -suboptimal solutions to (3) and (4), respectively.

Original: We note that the inexactness criteria (5) and (6) are same as that used in [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].
Revised: Note that the inexactness criteria (5) and (6) are same as those used in [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].

Original: Whereas such a criterion has limitations, it remains the most standard one for convergence analysis with inexact proximal operator.
Revised: Whereas such a type of criteria has limitations, it remains the most standard one for convergence analysis with inexact proximal operator.

Original: Our analysis allows  the sub-problems (3) and (4) do not admit closed-form solutions, therefore one needs to compute an approximate solution up to a certain accuracy with some iterative algorithm M .
Revised: Our analysis allows that the sub-problems (3) and (4) do not admit closed-form solutions Therefore one needs to compute an approximate solution up to a certain accuracy with some iterative algorithm M .

Original: Note thatthe sub-problems both  (3) and (4) are strongly convex even the objective f ( x ) is not strongly convex.
Revised: Note that both the subproblems (3) and (4) are strongly convex even the objective f ( x ) is not strongly convex.

Original: The strong convexity allows us to efficiently obtain a ξ k +1 -suboptimal solution via M with a linear convergence rate [Lin et al., 2017, Kulunchakov and
Revised: The strong convexity allows us to efﬁciently obtain a ξ k +1 -suboptimal solution via M with a linear convergence rate [Lin et al., 2017, Kulunchakov and

Original: Mairal, 2019].
Revised: Mairal, 2019].

Original:  CONVERGENCE ANALYSIS
Revised:  CONVERGENCE ANALYSIS

Original: In this section, we present the convergence analysis for Algorithm 1 when apply it to solve (1).
Revised: In this section, we present the convergence analysis for Algorithm 1 when applying it to solve (1).

Original: Specifically, we first present properties of a suboptimal solution in Section 4.1.
Revised: Speciﬁcally, we ﬁrst present properties of a suboptimal solution in Section 4.1.

Original: Then, in Sections 4.2 and 4.3, we present specific convergence results of Algorithm 1 for µ = 0 and µ > 0 , in
Revised: Then, in Sections 4.2 and 4.3, we present speciﬁc convergence results of Algorithm 1 for µ = 0 and µ > 0 , in

Original: Theorems 1 and 2, respectively.
Revised: Theorems 1 and 2, respectively.

Original: For convenience, we also assume the ψ ( x ) is 1 -strongly convex and ρ -smooth w.r.t.
Revised: For convenience, we also assume that ψ ( x ) is 1 -strongly convex and ρ -smooth w.r.t.

Original: ∥ · ∥ . 
Revised: (cid:107) · (cid:107) . By introducing the Bregman divergence, our analysis can include more cases than existing works. In other words, it can recover many general cases. For example, the counterpart considers the squared (cid:96) 2 -norm that can be easily obtained by setting ψ ( x ) = 12 (cid:107) x (cid:107) 22 where ψ ( x ) is 1 -strongly convex and 1 -smooth w.r.t.

Revised: the (cid:96) 2 -norm.

Original: 4.1 PROPERTIES OF SUBOPTIMAL SOLUTION
Revised: 4.1 PROPERTIES OF SUBOPTIMAL SOLUTION

Original: For convergence analysis with inexact proximal operator, the key is to bound the solution of inexact proximal operator and the ξ -subgradient of the inexact solution.
Revised: For convergence analysis with inexact proximal operators, the key is to bound the solution of inexact proximal operator and the ξ -subgradient of the inexact solution.

Original: Thus, we first introduce the definition of ξ -subgradient, which is a generalization of subgradient.
Revised: Thus, we ﬁrst introduce the deﬁnition of ξ -subgradient, which is a generalization of subgradient.

Original: Definition 5.
Revised: Deﬁnition 5.

Original: [Bertsekas et al., 2003]
Revised: [Bertsekas et al., 2003]

Original: For convex function f : R d → R and a non-negative scalar ξ , ∂ ξ f ( x ) is the ξ -subgradient of f at x if it holds that, ∀ v ∈ ∂ ξ f ( x ) ,f ( y ) ≥ f ( x ) + ⟨ v , y − x ⟩ − ξ, ∀ y ∈
Revised: For convex function f : R d → R and a non-negative scalar ξ , ∂ ξ f ( x ) is the ξ -subgradient of f at x if it holds that, ∀ v ∈ ∂ ξ f ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: d . Definition 5 implies that 0 is a ξ -subgradient of f at x if x is a ξ -suboptimal solution of f .
Revised: Deﬁnition 5 implies that 0 is a ξ -subgradient of f at x if x is a ξ -suboptimal solution of f .

Original: In the case of ℓ 2 norm,
Revised: In the case of the (cid:96) 2 norm,

Original: Schmidt et al.
Revised: Schmidt et al.

Original: [2011] showed that v in Definition 5 can be easily bounded as the squared Euclidean distance in (2)  leads to an analytical form for v .
Revised: [2011] showed that v in Deﬁnition 5 can be easily bounded as the squared Euclidean distance in (2) which leads to an analytical form for v .

Original: In contrast, our case is more challenging as it involves ( ∇ ψ ) − 1 which generally does not has an analytical form for generic Bregman divergence.
Revised: In contrast, our case is more challenging as it involves ( ∇ ψ ) − 1 which generally does not has an analytical form for generic Bregman divergence.

Original: To address this problem, we present a relaxation method by exploiting the strongly convexity of ψ so that it still admit an analytical form.
Revised: To address this problem, we present a relaxation method by exploiting the strongly convexity of ψ so that it still admits an analytical form.

Original: We define
Revised: We deﬁne

Revised: Q

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original:  If z k +1 is ξ k +1 -suboptimal to (4), Lemma 1 provides a ξ k +1 -subgradient for Q k +1 ( · ; w k +1 ) at z k +1 .
Revised: ) , z − w [Equation] +1 (cid:105) + h ( z ) . If z k +1 is ξ k +1 -suboptimal to (4), Lemma 1 provides a ξ k +1 -subgradient for Q k +1 ( · ; w k +1 ) at z k +1 .

Original: Lemma 1.
Revised: Lemma 1.

Original: For ∀ k ≥ 0 , if z k +1 is a ξ k +1 -suboptimal solution to (4) in the sense of (6) , then there exists β k +1 with ∥
Revised: For ∀ k ≥ 0 , if z k +1 is a ξ k +1 -suboptimal solution to (4) in the sense of (6) , then there exists β k +1 with (cid:107) β

Original: β k +1 ∥ 2 ∗ ≤ [Equation] k
Revised: k +1 (cid:107) 2 ∗ ≤ 2 ρξ k

Original: [Equation] /η k +1 such that 
Revised: +1 /η k +1 such that ∇ ψ ( w k +1 )

Original: [Equation]{+− ∇ ψ ( z k +1 ) η k + − β k +1 ∈ ∂ ξ k +1 Q k +
 ( z k +1 ; w k +1 ) .+}
Revised: {+− ∇ ψ ( z k +1 ) η k + − β k +1 ∈ ∂ ξ k +1 Q k +
 ( z k +1 ; w k +1 ) .+}

Original: The proof of Lemma 1 is given in Appendix A.1.
Revised: The proof of Lemma 1 is given in Appendix A.1.

Original: By using Lemma 1, the following lemma enables us to bound the intermediate results of mirror descent with  inexact proximal operator.
Revised: By using Lemma 1, the following lemma enables us to bound the intermediate results of mirror descent with the inexact proximal operator.

Original: Lemma 2.
Revised: Lemma 2.

Original: Under the same setting as in Lemma 1, then there exists β k +1 with ∥
Revised: Under the same setting as in Lemma 1, then there exists β k +1 with (cid:107) β

Original: β k +1 ∥ 2 ∗ ≤ 2 ρξ k
Revised: k +1 (cid:107) 2 ∗ ≤ 2 ρξ k

Original: [-+
-] /η
Revised: [-+
-]+1 /η

Original: k +1 such that
Revised: k +1 such that

Original: [Equation]
Revised: [Equation]

Revised: k +1 ( z k +

Revised: [Equation]

Revised: w k +1 )

Revised: + V ψ (cid:0) u , z k +1 (cid:1) /η

Revised: k +1 − ξ k + ≤ (cid:98) Q k +1 ( u ; w k +1 )

Revised: + (cid:10) β k

Revised: +1 , u − z k +1 (cid:11) , ∀ u , k ≥ 0 .

Original: The proof of Lemma 2 is given in Section A.2.
Revised: The proof of Lemma 2 is given in Appendix A.2.

Original: 4.2 CONVERGENCE RATES OF CONVEX g
Revised: 4.2 CONVERGENCE RATES OF CONVEX g

Original: We first focus on the case when g is convex.
Revised: We ﬁrst focus on the case when g is convex.

Original: The next lemma derives a characteristic inequality for a specific Lyapunov function for inexact linear coupling by considering the inexactness of y k +1 and z k +1 .
Revised: The next lemma derives a characteristic inequality for a speciﬁc Lyapunov function for inexact linear coupling by considering the inexactness of y k +1 and z k +1 .

Original: Lemma 3.
Revised: Lemma 3.

Original: Under the same setting as in Lemma 1, for any k ≥ 0 , if g is convex and τ k = 1 /Lη
Revised: Under the same setting as in Lemma 1, for any k ≥ 0 , if g is convex and τ k = 1 /Lη

Original: k +1 ,
Revised: k +1 ,

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma 3 is given in Appendix A.3.
Revised: The proof of Lemma 3 is given in Appendix B.1.

Original: From the
Revised: From the

Original: Lyapunov function, we obtain a general convergence result for linear coupling with inexact proximal operator.
Revised: Lyapunov function, we obtain a general convergence result for linear coupling with inexact proximal operator.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under the same setting as in Lemma 1, if g is convex, η k +1 =
Revised: Under the same setting as in Lemma 1, if g is convex, η k +1 =

Original: ( k + 2) / 2 L and τ k = 1 /Lη
Revised: ( k + 2) / 2 L and τ k = 1 /Lη

Original: k +1 , ∀ k ≥ 0 , then ∀ T ≥ 1 :
Revised: k +1 , ∀ k ≥ 0 , then ∀ T ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: where [Equation] and 
Revised: where (cid:101) E T def = (cid:80) Tk =1 ( k + 2) 2 ξ k and (cid:98) E T def = (cid:16)(cid:80) Tk =1 (cid:112)

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 1 is given in Appendix A.4.
Revised: The proof of Theorem 1 is given in Appendix B.2.

Original: Remark 1.
Revised: Remark 1.

Original: If the proximal operator is exact (i.e., ξ k = 0 , ∀ k ≥ 1 ), it leads to (cid:101) E T = 0 and (cid:98) E T = 0 .
Revised: If the proximal operator is exact (i.e., ξ k = 0 , ∀ k ≥ 1 ), it leads to (cid:101) E T = 0 and (cid:98) E T = 0 .

Original: Then, Theorem 1 recoveries the accelerated complexity O ( (cid:112) L/ϵ ) for convex objectives [Nesterov, 2013].
Revised: Then, Theorem 1 recoveries the accelerated complexity O ( (cid:112) L/(cid:15) ) for convex objectives [Nesterov, 2013].

Original: Corollary 1.
Revised: Corollary 1.

Original: Consider the same setting as Theorem 1, if ξ k ≤ ξ for all k ≥ 1 , then ∀ T ≥ 1 :
Revised: Consider the same setting as Theorem 1, if ξ k ≤ ξ for all k ≥ 1 , then ∀ T ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: where θ 1 =
Revised: where θ 1 =

Original: (6 + 16 ρ ) and θ 2 = 15 + 32 ρ .
Revised: (6 + 16 ρ ) and θ 2 = 15 + 32 ρ .

Original: Remark 2.
Revised: Remark 2.

Original: Under the same setting as Corollary 1, the [Schmidt et al., 2011, Proposition 4] gives
Revised: Under the same setting as Corollary 1, based on [Schmidt et al., 2011, Proposition 4], we have

Original: [Equation]
Revised: [Equation]

Original: The [Lin et al., 2017, Theorem 3] gives
Revised: Based on [Lin et al., 2017, Theorem 3], we have

Original: [Equation]
Revised: [Equation]

Original: The [Kulunchakov and Mairal, 2019, Proposition 4] gives
Revised: Based on [Kulunchakov and Mairal, 2019, Proposition 4], we have

Original: [Equation]
Revised: [Equation]

Original: Comparing (10) with (11) , (12) and (13) , our analysis achieves better bound than existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].
Revised: Comparing (10) with (11) , (12) and (13) , our analysis achieves better bound than existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].

Original: Specifically, our bound on the inexact proximal operator ξ increases as O ( Tξ ) while it is O ( T 2 ξ ) for existing works.
Revised: Speciﬁcally, our bound on the inexact proximal operator ξ increases as O ( Tξ ) while it is O ( T 2 ξ ) for existing works.

Original: Theorem 1 suggests that the inexact proximal operator leads to error accumulation in the convergence result.
Revised: Theorem 1 suggests that the inexact proximal operator leads to error accumulation in the convergence result.

Original: To preserve the accelerated rate O ( 1 T 2 ) , Theorem 1 implies the error sequence should decrease to 0 at a sufficiently fast rate.
Revised: To preserve the accelerated rate O ( 1 T 2 ) , Theorem 1 implies the error sequence should decrease to 0 at a sufﬁciently fast rate.

Original: Corollary 2.
Revised: Corollary 2.

Original: Under the same setting as Theorem 1, for any δ > 0 , if ξ k is chosen as ξ k def = f ( x 0 ) − f ( x ⋆ ) ( k +2) 3+{+
 ++} δ , then ∀ T ≥ 1 ,
Revised: Under the same setting as Theorem 1, for any δ > 0 , if ξ k is chosen as ξ k def = f ( x 0 ) − f ( x (cid:63) ) ( k +2) {+
 ++} δ , then ∀ T ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: To preserve the accelerated rate O (1 /k 2 ) for convex objectives, existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019] requires the error sequences decreases at the rate of O (1 /k 4+ δ ) .
Revised: To preserve the accelerated rate O (1 /k 2 ) for convex objectives, existing works [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019] requires the error sequences decreases at the rate of O (1 /k 4+ δ ) .

Original: In contrast, our analysis suggests O (1 /k 3+ δ ) is sufficient.
Revised: In contrast, our analysis suggests O (1 /k 3+ δ ) is sufﬁcient.

Original: This is consistent with our tighter bound shown in Corollary 1.
Revised: This is consistent with our tighter bound shown in Corollary 1.

Original: As observed in (14), the objective value converges faster for more accurate proximal operator (i.e., a larger value for δ ).
Revised: As observed in (14), the objective value converges faster with more accurate proximal operator (i.e., a larger value for δ ).

Original: However,  larger δ also requires more computation time for each iteration.
Revised: However, a larger δ also requires more computation time for each iteration.

Original: Comparison with Alternating Direction Method of Multipliers (ADMM).
Revised: Comparison with Alternating Direction Method of Multipliers (ADMM)

Original: Besides the accelerated proximal method,the ADMM is another popular method for solving (1) due to it is easy to implement [Boyd et al., 2011].
Revised: Besides the accelerated proximal method, ADMM is another popular method for solving (1) due to its simplicity and applicability to broad applications [Boyd et al., 2011].

Revised: It also allows inexact minimization of subproblem to some extent.

Original: However, it is well-know thatthe ADMM converges as the rate of O (1 /T ) for convex objectives
Revised: However, it is well-known that ADMM converges at the rate of O (1 /T ) for convex objectives

Original: [He and Yuan, 2012].
Revised: [He and Yuan, 2012].

Original: In contrast, the convergence rate of accelerated proximal method is O (1 /T 2 ) , that is more desirable for large-scale machine learning problems.
Revised: In contrast, the convergence rate of accelerated proximal method is O (1 /T 2 ) , which is more desirable for large-scale machine learning problems.

Original: As discussed in [Boyd et al., 2011],the ADMM will converge even the when the subproblems of each iteration are not solved exactly, as long as the approximate solutions satisfy certain suboptimality measures.
Revised: As discussed in [Boyd et al., 2011], ADMM will converge even the when the subproblems of each iteration are not solved exactly, as long as the approximate solutions satisfy certain suboptimality measures.

Original: In other words,the ADMM also suffer from error accumulation if the subproblems cannot be exactly solved, for example, the proximal operator does not admit a closed-form solution.
Revised: In other words, ADMM also suffers from error accumulation if the sub-problems cannot be exactly solved. For example, the proximal operator does not admit a closed-form solution.

Original: We note our inexactness conditions (5) and (6) are absolute criteria that are same as the one used in [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].
Revised: We note our inexactness conditions (5) and (6) are absolute criteria which are same as the those used in [Schmidt et al., 2011, Lin et al., 2017, Kulunchakov and Mairal, 2019].

Original: In contrast, existing works on convergence analysis ofthe ADMM are mainly based on  relative error accuracy [He et al., 2002, 
Revised: In contrast, existing works on convergence analysis of ADMM are mainly based on a relative error accuracy [He et al., 2002, Eckstein and Yao,

Original: Eckstein and Yao, 2018, Alves et al., 2020] that is generally a stronger inexactness condition than ours.
Revised: 2018, Alves et al., 2020] that is generally a stronger inexactness condition than ours.

Original: Ifthe ADMM takes the inexactness criterion as ours, under the same setting as Corollary 1, one can show that the convergence rate of inexact ADMM is
Revised: If ADMM takes the inexactness criterion as ours, under the same setting as Corollary 1, one can show that the convergence rate of inexact ADMM is

Original: [Equation]
Revised: [Equation]

Original: where ν and γ are some constants.
Revised: where ν and γ are some constants.

Original: Comparing it with (10), we observed both ADMM and our result have O ( Tξ ) error accumulation, however, our result achieves  O (1 /T 2 ) convergence rate while it is only O (1 /T ) for ADMM.
Revised: Comparing it with (10), we can observe that both ADMM and our result have O ( Tξ ) error accumulation. However, our result achieves an O (1 /T 2 ) convergence rate while the rate is only O (1 /T ) for ADMM.

Original: 4.3 CONVERGENCE RATES OF STRONGLY CONVEX g
Revised: 4.3 CONVERGENCE RATES OF STRONGLY CONVEX g

Original: In the last section, we present the convergence result of linear coupling with inexact proximal operator for convex g .
Revised: In this section, we present the convergence result of linear coupling with inexact proximal operators for convex g .

Original: Next, we present the convergence result for strongly convex g .
Revised: Next, we present the convergence result for strongly convex g .

Original: As is customary [Nesterov, 2013, Cohen et al., 2018], in this setting we assume that ∥ · ∥ = ∥ · ∥ 2 .
Revised: In this setting we assume (cid:107) · (cid:107) = (cid:107) · (cid:107) 2 that follows the customary [Nesterov, 2013] of convergence analysis of optimization algorithms for strongly convex objectives. It is mainly used to simplify the proof of convergence analysis. Note that by far, whether the same properties obtained by the (cid:96) 2 -norm can be generalized to other norms is still an open question.

Revised: However, even for the (cid:96) 2 -norm, as we shall see in Remark 4, our convergence rate is still better than previously works.

Original: We first introduce the analogue of Lemma 3 for strongly convex g .
Revised: We ﬁrst introduce the analogue of Lemma 3 for strongly convex g .

Original: By considering the inexactness of y k +1 and z k +1 , the next lemma derives a characteristic inequality for a specific
Revised: By considering the inexactness of y k +1 and z k +1 , the next lemma derives a characteristic inequality for a speciﬁc

Original: Lyapunov function for strongly convex objectives.
Revised: Lyapunov function for strongly convex objectives.

Original: Lemma 4.
Revised: Lemma 4.

Original: Under the same setting as Lemma 1, if g is strongly convex, τ k = Lα k +1 − µ L − µ and η k +

Revised: Under the same setting as Lemma 1, if g is strongly convex, τ k = Lα k +1 − µ L − µ and η k +

Original: =
Revised: =

Original: Lα k +1 , then ∀ k ≥ 0 , 
Revised: Lα k +1 , then ∀ k ≥ 0 , f ( y k +1 ) − f ( x (cid:63) ) +

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma 4 is given in Appendix A.8.
Revised: The proof of Lemma 4 is given in Appendix A.8.

Original: From the
Revised: From the

Original: Lyapunov function, we obtain a general convergence result for linear coupling with inexact proximal operator.
Revised: Lyapunov function, we obtain a general convergence result for linear coupling with inexact proximal operators.

Original: We define ∆
Revised: We deﬁne ∆

Original: [Equation]
Revised: [Equation]

Original:  Theorem 2 presents the convergence result of linear coupling with inexact proximal operator for strongly convex g .
Revised: , ∀ k ≥ 0 . Theorem 2 presents the convergence result of linear coupling with inexact proximal operators for strongly convex g .

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the same setting as Lemma 1, if g is strongly convex, τ k = Lα k +1 − µ L − µ and η k +

Revised: Under the same setting as Lemma 1, if g is strongly convex, τ k = Lα k +1 − µ L − µ and η k +

Original: =
Revised: =

Original: Lα k +1 , then ∀T ≥ 1 : ∆ T
Revised: Lα k +1 , then ∀

Original: [Equation]
Revised: [Equation]

Original: T

Original: [Equation]
Revised: [Equation]

Original:  where Γ k def = (cid:81) ki =1 (1 − α k ) .
Revised: ) where Γ k def = (cid:81) ki [Equation] 1 (1 − α k ) .

Original: If α 0 = (cid:112) µL , then ∀ T ≥ 1 : 
Revised: If α 0 = (cid:112) µL , then ∀ T ≥ 1 : f ( y T

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Original: where (cid:101) ∆ 0 def = f ( x 0 ) − f ( x ⋆ ) , (cid:101) R T def = 3 (cid:80) Tk =1 (cid:0) 1 − (cid:112) µL (cid:1) − k ξ k
Revised: ≤ (cid:18) [Equation] − (cid:114) [Equation]

Original: and (cid:98) R T def = 6 ρ (cid:112) µL (cid:16)(cid:80)
Revised: L (cid:19) T [Equation] 17) where (cid:101) ∆

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: Tk =1 (cid:0) 1 − (cid:112) µL (cid:1) − k/ 2 √ ξ k (cid:17) 2 .
Revised: − [Equation] k and [Equation]

Revised: R

Original:  The proof of Theorem 2 is given in Appendix A.9.
Revised: [Equation] def [Equation] The proof of Theorem 2 is given in Appendix A.9.

Original: Remark 4.
Revised: Remark 4.

Original: If the proximal operator is exact (i.e., ξ k = 0 , ∀ k ≥ 1 ), it leads to (cid:101) R T = 0 and (cid:98) R T = 0 .
Revised: If the proximal operator is exact (i.e., ξ k = 0 , ∀ k ≥ 1 ), it leads to (cid:101) R T = 0 and (cid:98) R T = 0 .

Original: Then, (17) recoveries the accelerated complexity O ( (cid:112) L/µ log(1 /ϵ )) for µ -strongly convex objectives
Revised: Then, (17) recoveries the accelerated complexity O ( (cid:112) L/µ log(1 /(cid:15) )) for µ -strongly convex objectives

Original: [Nesterov, 2013].
Revised: [Nesterov, 2013].

Original: Corollary 3.
Revised: Corollary 3.

Original: Consider the same setting as Theorem 2, if ξ k ≤ ξ for all k ≥ 1 , then ∀ T ≥ 1 :
Revised: Consider the same setting as Theorem 2, if ξ k ≤ ξ for all k ≥ 1 , then ∀ T ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: Remark 5.
Revised: Remark 5.

Original: Under the same setting as Corollary 3, the [Schmidt et al., 2011, Proposition 4] gives
Revised: Under the same setting as Corollary 3, based on [Schmidt et al., 2011, Proposition 4], we have

Original: [Equation]
Revised: [Equation]

Original: The [Lin et al., 2017, Theorem 3] gives
Revised: Based on [Lin et al., 2017, Theorem 3], we have

Original: [Equation]
Revised: [Equation]

Original: The [Kulunchakov and Mairal, 2019, Proposition 4] gives [Equation]
Revised: Based on [Kulunchakov and Mairal, 2019, Proposition 4], we have

Original: [Equation]
Revised: [Equation]

Original: Comparing (18) with (19) , (20) and (21) , our analysis achieves better bound than existing works.
Revised: Comparing (18) with (19) , (20) and (21) , our analysis achieves better bound than existing works.

Original: Specifically, our error bound on ξ is O ( (cid:113) Lµ ξ ) , while that are O
Revised: Speciﬁcally, our error bound on ξ is O ( (cid:113) Lµ ξ ) , while that are O

Original: ( L 2 µ 2 ξ ) for [Schmidt et al., 2011] and O ( Lµ ξ ) for [Lin et al., 2017,
Revised: ( L 2 µ 2 ξ ) for [Schmidt et al., 2011] and O ( L µ ξ ) for [Lin et al., 2017,

Original: Kulunchakov and Mairal, 2019], respectively.
Revised: Kulunchakov and Mairal, 2019], respectively.

Original: To preserve accelerated rate for Algorithm 1, Theoremimplies that the error sequence { ξ k } k ≥ 1 needs to decrease to 0 at a linear rate.
Revised: To preserve accelerated rate for Algorithm 1, Theoremimplies that the error sequence { ξ k } k ≥ 1 needs to decrease to 0 at a linear rate.

Original: Corollary 4.
Revised: Corollary 4.

Original: Under the same setting as in Theorem 2, for any ϑ ∈ (0 , (cid:112) µ/L ) , if ξ k is chosen as ξ k ≤
Revised: Under the same setting as in Theorem 2, for any ϑ ∈ (0 , (cid:112) µ/L ) , if ξ k is chosen as ξ k ≤

Original: [Equation]
Revised: [Equation]

Original: ∆ [Equation] (1 − ϑ ) k , then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Similar to the conclusion for convex g , the objective with strongly convex g has a faster convergence speed when the error of inexact proximal operator decreases in a faster rate (i.e., a larger value for ρ ).
Revised: Similar to the conclusion for convex g , the objective with strongly convex g has a faster convergence speed when the error of inexact proximal operators decreases at a faster rate (i.e., a larger value for ρ ).

Original: In this section, we perform two experiments to verify our theoretical results. 5.1 CUR-LIKE FACTORIZATION
Revised: 200 1010 - 1010 - 1010 - k 1/k k 1/kk 1/kk 1/k

Original: We first apply Algorithm 1 to solve the CUR-like factorization optimization problem [Mairal et al., 2011]. For a given
Revised: ADMM k 1/k 3 ADMM k

Original:     200 1010 - 1010 - 1010 - k 1/k k 1/kk 1/kk 1/k
Revised: 1/k    200 1010 - 1010 - 1010k 1/k k 1/kk 1/kk 1/k

Original: ADMM k 1/k 3 ADMM k
Revised: ADMM k 1/k 3 ADMM k

Original: 1/k    200 1010 - 1010 - 1010k 1/k k 1/kk 1/kk 1/k
Revised: 1/k    200 1010 - 1010 - 1010k 1/k k 1/kk 1/kk 1/k

Original: ADMM k 1/k 3 ADMM k 
Revised: ADMM k 1/k 3 ADMM k 1/k

Original: 1/k    200 1010 - 1010 - 1010k 1/k k 1/kk 1/kk 1/k
Revised: In this section, we conduct two experiments to verify our theoretical results. 5.1 CUR-LIKE FACTORIZATION We ﬁrst apply Algorithm 1 to solve the CUR-like factorization optimization problem [Mairal et al., 2011].

Original: ADMM k 1/k 3 ADMM k 1/k matrix D ∈ R m × n , the CUR-like factorization aims to approximate D by a matrix X with sparse rows and sparse columns.
Revised: For a given matrix D ∈ R m × n , the CUR-like factorization aims to approximate D by a matrix X with sparse rows and sparse columns.

Original: [Equation]
Revised: [Equation]

Original: where X i · and X · ,j denote the i -th row and  j -th column of X , respectively.
Revised: where X i · and X · ,j denote the i -th row and the j -th column of X , respectively.

Original: The last two terms impose the ℓ 2 , 1 norms for both row and columns of X , that yields both sparse rows and columns.
Revised: The last two terms impose the (cid:96) 2 , 1 norms for both row and columns of X , that yields both sparse rows and columns.

Original: However, the proximal operator of this regularizer does not admit an analytical solution, even no iterative algorithm  can exactly compute the proximity operator.
Revised: However, the proximal operator of this regularizer does not admit an analytical solution. There is even no iterative algorithm that can exactly compute the proximity operator.

Original: Following [Schmidt et al., 2011], we approximately compute the proximal operator by a block coordinate descent (BCD) algorithm that is presented by Jenatton et al.
Revised: Following [Schmidt et al., 2011], we approximately compute the proximal operator by a block coordinate descent (BCD) algorithm that is presented by Jenatton et al.

Original: [2011] to efficiently obtain an approximate solution of the proximal operator.
Revised: [2011] to efﬁciently obtain an approximate solution of the proximal operator.

Original: The BCD alternates between computing the proximal operator with respect to the rows and to the columns.
Revised: The BCD alternates between computing the proximal operator with respect to the rows and to the columns.

Original: At each iteration, we employ the BCD to solve (3) and (4) until (5) and (6) are satisfied that means both y k and z k are ξ k -suboptimal solutions.
Revised: At each iteration, we employ the BCD to solve (3) and (4) until (5) and (6) are satisﬁed that means both y k and z k are ξ k -suboptimal solutions.

Original: Suggested by Corollary 2, we consider a decreased error sequences { ξ k } k ≥ 1 where ξ k ≤ 1 /k α and the value of α are set to α = 1 , 2 , 3 , 4 in our experiments.
Revised: Suggested by Corollary 2, we consider a decreased error sequences { ξ k } k ≥ 1 where ξ k ≤ 1 /k α and the value of α are set to α = 1 , 2 , 3 , 4 in our experiments.

Original: As discussed before, the accelerated convergence rate of linear coupling can be preserved if α > 3 .
Revised: As discussed before, the accelerated convergence rate of linear coupling can be preserved if α > 3 .

Original: We perform experiments on four data sets 1 : mushroom , secom , a1a and musk .
Revised: We perform experiments on four data sets 1 : mushroom , secom , a1a and musk .

Original: We set λ 1 = 0 .
Revised: We set λ 1 = 0 .

Original: 01 and λ 2 = 0 .for
Revised: 01 and λ 2 = 0 .for

Original: all four datasets.
Revised: all four datasets.

Original: Rather than assume the Lipschitz constant L is known, we estimate it by line search [Nesterov, 2013].
Revised: Rather than assume the Lipschitz constant L is known, we estimate it by line search [Nesterov, 2013].

Original: Specifically, we initialize the value of L as L = 0 .and
Revised: Speciﬁcally, we initialize the value of L as L = 0 .and

Original: double it if the following inequality is not satisfied
Revised: double it if the following inequality is not satisﬁed

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we observed that this strategy always performs better than a fixed but conservative value for L .
Revised: In our experiments, we observed that this strategy always performs better than a ﬁxed but conservative value for L .

Original: To demonstrate the accelerated convergence rate of linear coupling, we compare it with ADMM.
Revised: To demonstrate the accelerated convergence rate of linear coupling, we compare it with ADMM.

Original: It is well-know that the performance of ADMM is highly dependent on the choice of penalty parameter ϱ .
Revised: It is well-know that the performance of ADMM is highly dependent on the choice of penalty parameter (cid:37) .

Original: To show the best performance of ADMM, we perform a grid search to find the best value of ϱ and fixed it for all iterations.
Revised: To show the best performance of ADMM, we perform a grid search to ﬁnd the best value of (cid:37) and ﬁxed it for all iterations.

Original: Fig.
Revised: Fig.

Original: 1 shows the objective function values versus the number of iterations of inexact linear coupling on the three data sets: a1a , secom and mushroom .
Revised: 1 shows the objective function values versus the number of iterations of inexact linear coupling on the three data sets: a1a , secom and mushroom .

Original: For linear coupling, the choice of ξ k ≤ 1 /k 4 achieves the fastest convergence rate according to our analysis (refer to Corollary 2), provides the best empirical performance across all three data sets.
Revised: For linear coupling, the choice of ξ k ≤ 1 /k 4 achieves the fastest convergence rate according to our analysis (refer to Corollary 2), provides the best empirical performance across all three data sets.

Original: However, as the iteration goes, the performance gap between the choices of ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 becomes smaller.
Revised: However, as the iteration goes, the performance gap between the choices of ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 becomes smaller.

Original: This is consistent with our theoretical results that the error sequences only need to decrease faster than O (1 /k 3 ) instead of O (1 /k 4 )
Revised: This is consistent with our theoretical results that the error sequences only need to decrease faster than O (1 /k 3 ) instead of O (1 /k 4 )

Original: [Schmidt et al., 2011, Lin et al., 2017,
Revised: [Schmidt et al., 2011, Lin et al., 2017,

Original: Kulunchakov and Mairal, 2019].
Revised: Kulunchakov and Mairal, 2019].

Original: Furthermore, as observed from Fig.
Revised: Furthermore, as observed from Fig.

Original: 1, the linear coupling clearly outperforms the ADMM on this task.
Revised: 1, the linear coupling clearly outperforms the ADMM on this task.

Original: Here, we show the results of ADMM with ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 .
Revised: Here, we show the results of ADMM with ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 .

Original: In fact, we observed the result of ADMM when α ≥ 2 are very close to each other, that is consistent with the theoreticalresult.
Revised: In fact, we observed the result of ADMM when α ≥ 2 are very close to each other, that is consistent with the theoretical

Revised: 100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k     100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k     100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k result.

Original: Specifically, the error sequences should decrease faster than O (1 /k 2 ) if the convergence rate of the algorithm is O (1 /k ) , for example, non-accelerated proximal method.
Revised: Speciﬁcally, the error sequences should decrease faster than O (1 /k 2 ) if the convergence rate of the algorithm is O (1 /k ) , for example, non-accelerated proximal method.

Original: 5.2 IMAGE DEBLURRING WITH ISOTROPIC TOTAL VARIATION
Revised: 5.2 IMAGE DEBLURRING WITH ISOTROPIC TOTAL VARIATION

Original: For the second experiment, we consider image deblurring with isotropic total variation regularization [Chambolle, 2004, Beck and Teboulle, 2009], which does not admit an analytical solution for the proximal operator [Chambolle and Pock, 2011, Beck and Teboulle, 2009].
Revised: For the second experiment, we consider image deblurring with isotropic total variation regularization [Chambolle, 2004, Beck and Teboulle, 2009], which does not admit an analytical solution for the proximal operator [Chambolle and Pock, 2011, Beck and Teboulle, 2009].

Original: For X ∈ R m × n ,100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k     100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k     100 1010 - 1010 - 101010k 1/k k 1/kk 1/kk 1/k the discrete gradient operator is( ∇ X ) ij =  
Revised: For X ∈ R m × n , the discrete gradient operator is

Original: [Equation]
Revised: [Equation]

Original: if

Original: [Equation]

Original: [Equation]

Original: if i [Equation] = n Then, image deblurring with isotropic total variation regularization can be written as [Chambolle and Pock, 2016,
Revised: Then, image deblurring with isotropic total variation regularization can be written as [Chambolle and Pock, 2016,

Original: Beck and Teboulle, 2009]min
Revised: Beck and Teboulle, 2009]

Original: [Equation]
Revised: [Equation]

Original: where A : R m × n → R m × n is a linear operator representing some blurring processing, B ∈ R m × n is the blurred and noisy image, and λ > 0 denotes a regularization parameter.
Revised: where A : R m × n → R m × n is a linear operator representing some blurring processing, B ∈ R m × n is the blurred and noisy image, and λ > 0 denotes a regularization parameter.

Original: Unlike the ℓ 1 total variation, the proximal operator of isotropic total variation regularization does not admit an analytical solution.
Revised: Unlike the (cid:96) 1 total variation, the proximal operator of isotropic total variation regularization does not admit an analytical solution.

Original: Same as before, we employ the BCD algorithm to compute an approximate solution.
Revised: Same as before, we employ the BCD algorithm to compute an approximate solution.

Original: We conducted this experiment on three images Lena , Boat and Football from M ATLAB image processing toolbox.
Revised: We conducted this experiment on three images Lena , Boat and Football from M ATLAB image processing toolbox.

Original: We first resize each image to 128 × 128 pixels.
Revised: We ﬁrst resize each image to 128 × 128 pixels.

Original: To obtain B , the clean image was first blurred by a 5 × 5 kernel matrix:
Revised: To obtain B , the clean image was ﬁrst blurred by a 5 × 5 kernel matrix:

Original: S = 125 I 5 × 5 , where I 5 × 5 ∈ R 5 × 5 is an identify matrix, followed by additive Gaussian noise with zero mean and standard deviation 10 − 1 .
Revised: S = 125 I 5 × 5 , where I 5 × 5 ∈ R 5 × 5 is an identify matrix, followed by additive Gaussian noise with zero mean and standard deviation 10 − 1 .

Original: The regularization parameter λ was set to 0 .
Revised: The regularization parameter λ was set to 0 .

Original: 1 for all three images.
Revised: 1 for all three images.

Original: Fig.
Revised: Fig.

Original: 2 shows the objective function values versus the number of iterations of inexact linear coupling onthe Lena , Boat and Football .
Revised: 2 shows the objective function values versus the number of iterations of inexact linear coupling on Lena , Boat and Football .

Original: Similar trends as the first experiments are observed for this experiment except the performance between ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 are very close to each other.
Revised: Similar trends as the ﬁrst experiments are observed for this experiment except the performance between ξ k ≤ 1 /k 3 and ξ k ≤ 1 /k 4 are very close to each other.

Original: In this experiment, we did not perform the experiments of ADMM due to it solves the subproblem associated with 12 (cid:13)(cid:13)
Revised: In this experiment, we did not perform the experiments of ADMM due to it solves the subproblem associated with 12 (cid:13)(cid:13)

Original: A ( X ) − B ∥ 2 F is computationally expensive.
Revised: A ( X ) − B (cid:107) 2 F is computationally expensive.

Original: Nevertheless, the experiment in Section 5.1 already clearly shows the advantages of linear coupling over ADMM.
Revised: Nevertheless, the experiment in Section 5.1 already clearly shows the advantages of linear coupling over ADMM.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: Non-smooth regularizations have played irreplaceable roles in machine learning.
Revised: Non-smooth regularizations have played irreplaceable roles in machine learning.

Original: However, many of them do not admit an analytical solution for proximal operator.
Revised: However, many of them do not admit an analytical solution for proximal operator.

Original: As a variant of
Revised: As a variant of

Original: Nesterov’s AGDs, the linear coupling can efficiently solve convex composite minimization with accelerated convergence rates if the proximal operator is exactly computed.
Revised: Nesterov’s AGDs, the linear coupling can efﬁciently solve convex composite minimization with accelerated convergence rates if the proximal operator is exactly computed.

Original: To further understand the performance of linear coupling, we present a complete convergence analysis for it with inexact proximal operator.
Revised: In this work, we present a complete convergence analysis for linear coupling with inexact proximal operators.

Original: Our analysis suggests that inexact linear coupling still achieves the accelerated convergence rate if the error sequence of inexact proximal operator decreases in a sufficiently fast rate.
Revised: Our analysis suggests that inexact linear coupling still achieves the accelerated convergence rate if the error sequence of inexact proximal operator decreases at a sufﬁciently fast rate.

Original: More importantly, our theoretical results is better than previous works.
Revised: More importantly, our theoretical results are better than previous works.

Original: We empirically verify our theoretical analysis by employing linear coupling with inexact operator to solve CUR-like factorization and image deblurring with isotropic total variation on different datasets.
Revised: We empirically verify our theoretical analysis by employing linear coupling with inexact operators to solve CUR-like factorization and image deblurring with isotropic total variation on different datasets.

Original: Empirical results clearly support our theoretical analysis.
Revised: Qiang Zhou is supported by National Science Foundation of China under Grant 62106045 and Southeast University Startup Fund (the Fundamental Research Funds for the Central Universities) under Grant 2242021R10096. Sinno J. Pan is supported by 2020 Microsoft Research Asia collaborative research grant.