RandProx: Primal–Dual Optimization Algorithms with Randomized Proximal Updates
Abstract
Proximal splitting algorithms are well suited for large-scale nonsmooth optimization problems.
We propose a primal–dual algorithm, in which the dual update is randomized, with the proximity operator of one of the function replaced by a stochastic oracle.
A nonsmooth variance-reduction technique is implemented so that the algorithm finds an exact minimizer of the general problem.
We derive linear convergence results in presence of strong convexity.
Several existing randomized algorithms, like Point-SAGA, are recovered as particular cases.
Randomness helps getting faster algorithms; this has long been known for stochastic-gradient-type algorithms and our work shows that this fully applies in the more general primal–dual setting as well.
1. Introduction
Large-dimensional optimization problems arise virtually in all fields, including machine learning, data science, statistics [4, 13, 15, 17, 37, 62, 65, 72, 73].
When a function is smooth, an optimization algorithm typically makes calls to its gradient , whereas for a nonsmooth function, its proximity operator is called instead.
Algorithms making use of proximity operators are called proximal (splitting) algorithms [63].
Over the past 10 years or so, several primal–dual proximal algorithms have been developed a [7, 10, 20, 21, 27, 29, 50, 63].
However, these deterministic algorithms are often too slow.
Stochastic Gradient Descent ( SGD )-type methods [11, 38, 40, 47, 61, 67], with the gradient of smooth functions replaced by stochastic estimates, are the striking demonstration that randomized algorithms can be cheaper than deterministic ones.
But replacing the proximity operator of a function by a stochastic proximity operator estimate is a nearly virgin territory.
This is important, because many functions of practical interest have a costly proximity operator.
We can mention the nuclear norm of matrices, which requires singular value decompositions, indicator functions of sets on which it is difficult to project, or optimal transport costs [64].
In this paper, we propose RandProx (Algorithm 2), a randomized version of the Primal–Dual
Davis–Yin (PDDY) method (Algorithm 1), which a proximal algorithm proposed recently [29, 70].
In RandProx , one proximity operator that appears in the PDDY algorithm is replaced by a stochastic estimate.
RandProx is variance-reduced [38, 40, 42]; that is, by making use of control variates, the algorithm converges to an exact solution, just like its deterministic counterpart.
We prove linear convergence of RandProx in the strongly convex setting, with additional results in the convex setting.
We mention relationships between our results and related works in the literature throughout the paper.
In special cases, RandProx reduces to Point-SAGA [32], the Stochastic Decoupling Method [58],
ProxSkip, SplitSkip and Scaffnew [60], and randomized versions of the PAPC [34], PDHG [16] and
ADMM [12] algorithms.
They are all generalized and unified within our new framework.
Thus, just like Point-SAGA [32] is the proximal counterpart of SAGA [33], our generic algorithm RandProx paves the way to a new world of proximal counterparts of variance-reduced SGD-type algorithms.
1.1. Problem formulation and proximal algorithms
Let X
and U be finite-dimensional real Hilbert spaces.
We consider the convex optimization problem:
[Equation]
where K : X → U is a nonzero linear operator; f is a convex L f -smooth function, for some L f > 0 ; that is, its gradient ∇ f is L f -Lipschitz continuous [6, Definition 1.47]; and g :
X → R ∪ { + ∞} and h : U → R ∪ { + ∞} are proper closed convex functions whose proximity operator is easy to compute.
We also introduce the dual problem to (1):
[Equation]
where K ∗ : U → X is the adjoint operator of K .
For these problems to be well-posed, we suppose that there exists x ⋆ ∈ X such that 0 ∈ ∇ f ( x ⋆ ) + ∂g ( x ⋆ )
+ K ∗ ∂h
( Kx ⋆ ) , where ∂ ( · ) denotes the subdifferential [6].
We will assume strong convexity of some functions: a convex function ϕ is said to be µ ϕ -strongly convex, for some µ ϕ ≥ 0 , if ϕ − µ ϕ 2 ∥ · ∥ 2 is convex.
We recall that for any function ϕ and parameter γ > 0 , the proximity operator of γϕ is [6]: prox γϕ : x ∈ X (cid:55)→ arg min [Equation] (cid:0) [Equation]
[Equation]
+
[Equation]
x ′
[Equation] x ∥ 2 (cid:1) .
This operator has a closed form for many functions of practical interest [36, 63, 66].
In addition, the Moreau identity holds: prox γϕ ∗ ( x ) = x − γ prox ϕ/γ
( x/γ ) ,
where ϕ ∗ : x ∈ X (cid:55)→ sup x ′ ∈X (cid:0) ⟨ x, x ′ ⟩− ϕ ( x ′ ) (cid:1) denotes the conjugate function of ϕ [6].
Thus, one can compute the proximity operator of ϕ from the one of ϕ ∗ , and conversely.
Proximal splitting algorithms, such as the forward–backward and the Douglas–Rachford algorithms [6], are well suited to minimizing the sum, f
+ g or g + h in our notation, of two functions.
However, many problems take the form (1) with K ̸
= Id , where Id denotes the identity, and the proximity operator of h ◦ K is intractable in most cases.
A classical example is the total variation, widely used in image processing [14, 24, 25, 68] or for regularization on graphs [30], where h is some variant of the ℓ 1 norm and K takes differences between adjacent values.
Another example is when h is the indicator function of some nonempty closed convex set Ω ⊂ U ; that is, h ( u )
=
(0 if u ∈ Ω , + ∞ otherwise ) , in which case the problem (1) can be rewritten as
[Equation]
If g = 0 and Ω = { b } for some b ∈ ran( K ) , where ran denotes the range, the problem can be further rewritten as the linearly constrained smooth minimization problem Find x ⋆ ∈ arg min x ∈X f ( x ) s.t.
Kx =
b.
This last problem has applications in decentralized optimization [52, 69, 79], for instance.
Thus, the template problem (1) covers a wide range of optimization problems met in machine learning [4, 65], signal and image processing [17, 20], control [73], and many other fields.
input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ >v 0 :
=
K ∗ ufor t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0) x
t − γ ∇ f ( x t ) − γv t (cid:1) u t
+1 :
= prox τh ∗ (cid:0) u t + τK ˆ x t (cid:1) v t +1 :
=
K ∗ u t + x t +1 :
=
ˆ x t − γ ( v t +1 − v t ) end for input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ > 0 ; ω ≥v 0 :
=
K ∗ ufor t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0) x
t − γ ∇ f ( x t ) − γv t (cid:1) u t
+1 :=
u t + 1 1+ ω R t (cid:0) prox τh ∗ ( u t + τK ˆ x t ) − u t (cid:1) v t
+1 :
=
K ∗ u t + x t +1 :
=
ˆ x t − γ (1 + ω ) ( v t +1 − v t ) end for
2. Proposed algorithm: RandProx
There exist several deterministic algorithms for solving the problem (1); see Condat et al.
[27] for a recent overview.
In this work, we focus on the PDDY algorithm (Algorithm 1) [29, 70].
In particular, our new algorithm RandProx (Algorithm 2) generalizes the PDDY algorithm with a stochastic estimate of the proximity operator of h ∗ .
We recall the general convergence result for the PDDY algorithm [29, Theorem 2]: If γ ∈ (0 , 2 /L f ) , τ > 0 , τγ ∥ K ∥ 2 ≤ 1 , then ( x t ) t ∈ N converges to a primal solution x ⋆ of (1) and ( u t ) t ∈ N converges to a dual solution u ⋆ of (2) .
The PDDY algorithm is similar and closely related to the
PD3O algorithm [81], as discussed in Condat et al.
[29], Salim et al.
We can note that the popular Condat–V˜u algorithm [23, 75] can solve the same problem but has more restrictive conditions on γ and τ .
In the PDDY algorithm, the full gradient ∇ f can be replaced by a stochastic estimator which is typically cheaper to compute [70].
Convergence rates and accelerations of the PDDY algorithm, as well as distributed versions of the algorithm, have been derived in Condat et al.
In particular, if µ f > 0 or µ g > 0 , the primal problem (1) is strongly convex.
In this case, a varying stepsize strategy accelerates the algorithm, with a O (1 /t 2 ) decay of ∥ x t − x ⋆ ∥ 2 , where x ⋆ is the unique solution to (1).
But strong convexity of the primal problem is not sufficient for the PDDY algorithm to converge linearly, and additional assumptions on h and K are needed.
We will prove linear convergence when both the primal and dual problems are strongly convex; this is a natural condition for primal–dual algorithms.
We can note that h is L h -smooth, for some L h > 0 , if and only if h ∗ is µ h ∗ -strongly convex, for some µ h ∗ > 0 , with µ h ∗
= 1 /L h .
In that case, the dual problem (2) is strongly convex.
We propose RandProx (Algorithm 2), a generalization of the PDDY algorithm (Algorithm 1) with a randomized update of the dual variable u .
Let us formalize the random operations using random variables and stochastic processes.
We introduce the underlying probability space ( S , F , P ) .
Given a real Hilbert space H , an H -valued random variable is a measurable map from ( S , F ) to ( H , B ) , where B is the Borel σ -algebra of H .
Formally, randomizing some steps in the PDDY algorithm amounts to defining (cid:0) ( x t , u t ) (cid:1) t ∈ N as a stochastic process, with x t being a X -valued random variable and u t a U -valued random variable, for every t ≥ 0 .
We use light notations and write our randomized algorithm RandProx using stochastic operators R t on U ; that is, for every t ≥ 0 and any r t ∈ U ,
R t ( r t ) is a U -valued random variable, which can be interpreted as r t plus ‘random noise’ (formally, r t is itself a U -valued random variable, but algorithmically, R t is applied to a particular outcome in
U , hence the notation as an operator on U ).
To fix the ideas, let us give two examples.
Example 1.
The first example is compression [1–3, 9, 28, 43, 59]: U = R d for some d ≥ 1 and R t is the well known rand - k compressor or sparsifier, with 1 ≤ k < d : R t multiplies k coordinates, chosen uniformly at random, of the vector r t by d/k and sets the other ones to zero.
An application to compressed communication is discussed in Section B.3.
Example 2.
The second example, discussed in Section B.1, is the Bernoulli, or coin flip, operator
[Equation]
for some p > 0 .
In that case, with probability 1 − p , the outcome of R t ( r t ) is 0 and r t does not need to be calculated; in particular, in the RandProx algorithm, prox τh ∗ is not called, and this is why one can expect the iteration complexity of RandProx to decrease.
Thus, in this example, R t ( r t ) does not really consist of applying the operator R t to r t ; in general, the notation R t ( r t ) simply denotes a stochastic estimate of r t .
Hereafter, we denote by F t the σ -algebra generated by the collection of ( X × U ) -valued random variables ( x 0 , u 0 ) , . . .
, ( x t , u t ) , for every t ≥ 0 .
In this work, we consider unbiased random estimates: for every t ≥ 0 ,
[Equation]
where E [ · ] denotes the expectation, here conditionally on F t , and r t is the random variable r t :
= prox τh ∗ ( u t + τK ˆ x t ) − u t , as defined by RandProx .
Note that our framework is general in that for t ̸ = t ′ , R t and R t ′ need not be independent nor have the same law.
In simple words, at every iteration, the randomness is new but can have a different form and depend on the past, so that the operators R t can be defined dynamically on the fly in RandProx .
We characterize the operators R t by their relative variance ω ≥ 0 such that, for every t ≥ 0 ,
[Equation]
The value of ω is supposed known and is used in the RandProx algorithm.
Note that ω
=
if and only if R t = Id , in which case there is no randomness and RandProx reverts to the original deterministic PDDY algorithm.
To characterize how the error on the dual variable propagates to the primal variable after applying K ∗ , we also introduce the relative variance ω ran ≥ 0 in the range of
K ∗ and the offset ζ ∈ [0 , 1] such that, for every t ≥ 0 ,
[Equation]
It is easy to see that (5) holds with ω ran =
∥ K ∥ 2 ω and ζ = 0 , so this is the default choice without particular knowledge on K ∗ .
But in some situations, e.g. sampling like in Section B.2, a much smaller value of ω ran and a positive value of ζ can be derived.
3. Convergence analysis of RandProx
Our most general result, whose proof is in the Appendix, is the following:
Theorem 1.
Suppose that µ f > 0 or µ g > 0 , and that µ h ∗ > 0 .
In RandProx , suppose that 0 < γ < 2 L f , τ > 0 , and γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 , where ω ran and ζ are defined in (5) .
For every t ≥ 0 , define the Lyapunov function
[Equation]
where x ⋆ and u ⋆ are the unique solutions to (1) and (2) , respectively.
Then RandProx converges linearly: for every t ≥ 0 ,
[Equation]
where
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
In the Appendix, we provide other linear convergence results, which do not require h to be smooth , as well as many examples of applications.
Appendix Appendix A. More linear convergence results
Remark 1 (choice of τ , in the conditions of Theorem 1) Given γ , the rate c in (8) is smallest if τ is largest.
So, there seems to be no reason to take τγ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) <
[Table]
and τγ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1)
= 1 should be the best choice in most cases.
Thus, one can set τ = 1 γ ((1 − ζ )
∥ K ∥ 2 + ω ran ) and keep γ as the only parameter to tune in RandProx .
In the rest of this section, we discuss some particular cases of (1), for which we derive stronger convergence guarantees than in Theorem 1 for RandProx .
Other particular cases are studied in the
Appendix; for instance, an instance of RandProx , called RandProx-ADMM , is a randomized version of the popular ADMM [12].
The different particular cases are summarized in Table 1.
A.1. Particular case g =
In this section, we assume that g
= 0 .
Then the PDDY algorithm becomes an algorithm proposed for least-squares problems [54] and rediscovered independently as the PDFP2O algorithm [19] and as the Proximal Alternating Predictor-Corrector (PAPC) algorithm [34]; let us call it the PAPC algorithm.
It has been shown to have a primal–dual forward–backward structure [22].
Thus, when g = 0 , RandProx is a randomized version of the PAPC algorithm.
We can note that f ∗ is strongly convex, which is not the case of ( f + g ) ∗ in general.
Let us define λ min ( KK ∗ ) as the smallest eigenvalue of KK ∗ .
λ min ( KK ∗ ) >
if and only if ker( K ∗ ) = { 0 } , where ker denotes the kernel.
If λ min ( KK ∗ ) > 0 , f ∗ ( − K ∗ · ) is strongly convex.
Thus, when g = 0 , λ min ( KK ∗ ) > 0 and µ h ∗ > 0 are two sufficient conditions for the dual problem (2) to be strongly convex.
We indeed get linear convergence of RandProx in that case:
Theorem 2.
Suppose that g = 0 , µ f > 0 , and that λ min ( KK ∗ ) > 0 or µ h ∗ > 0 .
In RandProx , suppose that 0 < γ < 2 L f , τ > 0 and γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 .
Then RandProx converges linearly: for every t ≥ 0 , E (cid:2)
Ψ t (cid:3) ≤ c t Ψ 0 , where the Lyapunov function Ψ t is defined in (6) , and
[Equation]
input: initial points x 0 ∈ X , u 0 ∈ X ; stepsize γ > 0 ; ω ≥for t = 0 , 1 , . . .
do ˆ x t := x t − γ ∇ f ( x t ) − γu t d t :=
R t (cid:0) ˆ x t − prox γ (1+ ω ) h (ˆ x t + γ (1+ ω ) u t ) (cid:1) u t +1 :=
u t + 1 γ (1+ ω )
d t x t +1 :
= ˆ x t − 1 1+ ω d t end for input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ > 0 ; ω ≥v 0 :=
K ∗ ufor t = 0 , 1 , . . .
do ˆ x t := x t − γ ∇ f ( x t ) − γv t u t
+1 :=
u t + τ 1+ ω R t
( K ˆ x t − b ) v t +1 :
=
K ∗ u t + x t +1 :
=
ˆ x t − γ (1 + ω )( v t +1 − v t ) end for
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
When R t = Id and ω = ω ran = 0 , RandProx reverts to the PAPC algorithm.
Even in this particular case, Theorem 2 proves linear convergence of the PAPC algorithm and is new.
In Chen et al.
[19, Theorem 3.7], linear convergence of an underrelaxed version of the algorithm was proved; underrelaxation slows down convergence.
In Luke and Shefi [55], Theorem 3.1 is wrong, since it is based on the false assumption that if λ min ( K i K ∗ i ) > 0 for linear operators K i , i = 1 , . .
, p , then λ min ( KK ∗ ) > 0 , with K : x (cid:55)→ ( K 1 x, . . .
, K p x ) .
Their theorem remains valid when p = 1 , but their rate is complicated and worse than ours.
We now consider the even more particular case of g = 0 and K
= Id .
Then the problems (1) and (2) consist in minimizing f ( x )+ h ( x ) and f ∗ ( − u )+ h ∗ ( u ) , respectively.
The dual problem is strongly convex and has a unique solution
u ⋆ = −∇ f
( x ⋆ ) , for any primal solution x ⋆ .
By setting τ := 1 /γ in the PAPC algorithm, with obtain the classical proximal gradient, a.k.a.
forward-backward (FB), algorithm, which iterates x
t +1 :
=
prox γh (cid:0) x t − γ ∇ f ( x t ) (cid:1) .
Thus, when randomness is introduced, we set ω ran := ω , ζ := 0 and, according to Remark 1, τ := 1 γ (1+ ω ) in RandProx .
By noting that, for every a > 0 , the abstract operators R t and a R t (cid:0) 1 a · (cid:1) have the same properties, we can put the constant γ (1 + ω ) outside R t to simplify the algorithm, and rewrite RandProx as RandProx-FB , shown above.
As a corollary of Theorem 2, we have:
Theorem 3.
Suppose that µ f > 0 .
In RandProx-FB , suppose that 0 < γ < 2 L f .
For every t ≥ 0 , define the Lyapunov function
[Equation] [Equation]
where x ⋆ is the unique minimizer of f + h and u ⋆ = −∇ f
( x ⋆ ) is the unique minimizer of f ∗ ( −· )+ h ∗ .
Then RandProx-FB converges linearly: for every t ≥ 0 ,
[Equation]
where
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
It is important to note that it is not necessary to have µ h ∗ > 0 in Theorem 3.
If we ignore the properties of h ∗ , the third factor in (11) can be replaced by its upper bound 1 −
[Equation]
A.2. Linearly constrained smooth minimization
Let b ∈ ran( K ) .
In this section, we consider the linearly constrained (LC) minimization problem
[Equation]
which is a particular case of (1) with g = 0 and h : u ∈ U (cid:55)→ (0 if u = b , + ∞ otherwise ) .
We have h ∗ : u ∈ U (cid:55)→ ⟨ u, b ⟩ and prox τh ∗ : u ∈ U (cid:55)→ u − τb .
The dual problem to (12) is
[Equation]
We denote by u ⋆ 0 the unique solution to (13) in ran( K ) .
Then the set of solutions of (13) is the affine subspace u ⋆
+ ker
( K ∗ ) .
Thus, the dual problem is not strongly convex, unless ker( K ∗ ) = { 0 } .
Yet, we will see that strong convexity of f is sufficient to have linear convergence of RandProx , without any condition on K .
We rewrite RandProx in this setting as RandProx-LC , shown above.
We observe that u t does not appear in the argument of R t any more, so that the iteration can be rewritten with the variable v t =
K ∗ u t , and u t can be removed if we are not interested in estimating a dual solution.
In any case, we denote by P ran( K ) the orthogonal projector onto ran( K ) and by λ +min ( KK ∗ ) >
the smallest nonzero eigenvalue of KK ∗ .
Then:
Theorem 4.
In the setup (12) – (13) , suppose that µ f > 0 .
In RandProx-LC , suppose that 0 < γ < 2 L f , τ > 0 and γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 .
Define the Lyapunov function, for every t ≥ 0 ,
[Equation]
where u t 0 :
= P ran( K ) ( u t ) is also the unique element in ran( K ) such that v t
=
K ∗ u t 0 , x ⋆ is the unique solution of (12) and u ⋆ 0 is the unique solution in ran( K ) of (13) .
Then RandProx-LC converges linearly: for every t ≥ 0 ,
[Equation]
where
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t 0 ) t ∈ N converges to u ⋆ 0 , almost surely.
Theorem 4 is new even for the PAPC algorithm when ω = 0 : its linear convergence under the stronger condition γτ ∥ K ∥ 2 < 1 has been shown in Salim et al.
[70, Theorem 6.2], but our rate in (15) is better.
We further discuss RandProx-LC , which can be used for decentralized optimization, in the
Appendix.
Another example of application is when X
= R d , for some d ≥ 1 , and K is a matrix; one can solve (12) by activating one row of K chosen uniformly at random at every iteration.
input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ > 0 ; p ∈ (0 , 1] v 0 :
=
K ∗ ufor t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0) x
t − γ ∇ f ( x t ) − γv t (cid:1)
Flip a coin θ t = (1 with probability p ,else ) if θ t = 1 then u t
+
=
prox τh ∗ ( u t + τK ˆ x t ) v t +1 :
=
K ∗ u t + x t +1 :
=
ˆ x t − γp ( v t +1 − v t ) else u t +1 :
= u t , v t +1 :
= v t , x t +1 :
= ˆ x t end if end for
Algorithm input: initial points x 0 ∈ X , ( u 0 i ) ni
=1 ∈ X n ; stepsize γ > 0 ; k ∈ { 1 , . .
. , n } v 0 :
= (cid:80) ni
=1 u
i for t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0) x
t − γ ∇ f ( x t ) − γv t (cid:1) pick Ω t ⊂ { 1 , . .
. , n } of size k unif.
at random for i ∈
Ω t do u t
+1 i :=
prox 1 γn h ∗
i ( u ti + 1 γn ˆ x t ) end for for i ∈ { 1 , . .
. , n }\ Ω t do u t
+1 i :=
u ti end for v t
+1 :
= (cid:80) ni
=1 u t
+
i x t +1 :
= ˆ x t − γnk ( v t +1 − v t ) end for
Appendix B. Examples B.1. Skipping the proximity operator
In this section, we consider the case of Bernoulli operators R t defined in (3), which compute and return their argument only with probability p > 0 .
RandProx becomes RandProx-Skip , shown above.
Then ω = 1 p − 1 , ω ran =
∥ K ∥ 2 ω , and ζ = 0 .
If g = 0 , RandProx-Skip reverts to the SplitSkip algorithm proposed recently [60].
Our Theorems 1 and 4 recover the same rate as given for SplitSkip in Mishchenko et al.
[60, Theorem D.1], if smoothness of h is ignored.
If in addition K = Id and τ = 1 γ (1+
[Equation]
) =
pγ , RandProx-Skip reverts to ProxSkip, a particular case of SplitSkip [60].
Our Theorem 3 applies to this case and allows us to exploit the possible smoothness of h in RandProx-Skip = ProxSkip, which is not the case of the results of [60].
As a practical application of our new results, let us consider personalized federated learning (FL) [41]: given a client-server architecture with a master and n ≥ 1 users, each with local cost function f i , i = 1 , . . .
, n , the goal is to
[Equation]
where ¯ x := 1 n (cid:80) ni
=1 x i .
Each f i is supposed L f -smooth and µ f -strongly convex.
We set X :=
( R d ) n , f : x = ( x i ) ni
=1 (cid:55)→ (cid:80) ni
=
f i ( x i ) , h : x (cid:55)→ λ 2 (cid:80) ni
=1 ∥ x i − ¯ x ∥ 2 .
f is L f -smooth and µ f -strongly convex, h is λ -smooth, so that µ h ∗ = 1 λ .
Thus, with γ = 1 L f , we have in (11):
[Equation]
Hence, with p = √ µ f min( L f ,λ )
L f , the communication complexity in terms of the expected number of communication rounds to reach ϵ -accuracy is O (cid:16)(cid:113) min
( L f ,λ ) µ f log 1 ϵ (cid:17) , which is optimal [41].
This shows that in personalized FL with λ < L f , the complexity can be decreased in comparison with non-personalized FL, which corresponds to λ = + ∞ .
This is achieved by properly setting p in
ProxSkip, according to our new theory, which exploits the smoothness of h .
B.2. Sampling among several functions
We first remark that we can extend Problem (1) with the term h ( Kx ) replaced by the sum (cid:80) ni
=1 h i ( K i x ) of n ≥ 2 proper closed convex functions h
i composed with linear operators K i :
X → U i , for some real Hilbert spaces U i , by using the classical product-space trick: by defining U :
= U 1 × · · · U n , h : u = ( u i ) ni
=1 ∈ U (cid:55)→ (cid:80) ni
=1 h i ( u i ) , K : x ∈ X (cid:55)→ ( K i x ) ni =1 ∈ U , we have h ( Kx )
=
(cid:80) ni
=1 h i ( K i x ) .
In particular, by setting K i := Id and U i :
= X , we consider in this section the problem:
[Equation]
We have h ∗ : ( u i ) ni
=1 ∈ X n (cid:55)→ (cid:80) ni
=
h ∗ i ( u i ) and we suppose that every function h ∗ i is µ h ∗ -strongly convex, for some µ h ∗ ≥ 0 ; then h ∗ is µ h ∗ -strongly convex.
Thus, the dual problem to (17) is
Find
[Equation]
u ⋆i
[Equation]
1 ∈ arg min
[Equation]
Since K ∗ K = n Id , ∥ K ∥ 2 = n .
Now, we choose R t as the rand - k sampling operator, for some k ∈ { 1 , . . . , n } : R t multiplies k elements out of the n of its argument sequence, chosen uniformly at random, by n/k and sets the other ones to zero.
It is known [26, Proposition 1] that we can set
[Equation]
Note that this value of ω ran is n − 1 times smaller than the naive bound ∥ K ∥ 2 ω = n ( n − k ) k .
We have (1 − ζ )
∥ K ∥ 2 + ω ran = n .
RandProx in this setting, with τ := 1 γn , becomes RandProx-Minibatch , shown above, and Theorem 1 yields:
Theorem 5.
Suppose that µ f > 0 or µ g > 0 , and that µ h ∗ > 0 .
In RandProx-Minibatch , suppose that 0 < γ < 2 L f .
Define the Lyapunov function, for every t ≥ 0 ,
[Equation]
where x ⋆ and ( u ⋆i ) ni =1 are the unique solutions to (17) and (18) , respectively.
Then RandProxMinibatch converges linearly: for every t ≥ 0 , E (cid:2)
Ψ t (cid:3) ≤ c t Ψ 0 , where
[Equation]
Algorithm
[58] input: initial points x 0 ∈ X , ( u 0 i ) ni
=1 ∈ X n ; stepsize γ >v 0 :
= (cid:80) ni
=1 u
i for t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0) x
t − γ ∇ f ( x t ) − γv t (cid:1) pick i t ∈ { 1 , . .
. , n } uniformly at random x t
+1 :
=
prox
γnh i ( γnu ti t + ˆ x t ) u t
+1 i t :=
u ti t + 1 γn (ˆ x t − x t +1 ) for every i ∈ { 1 , . . . , n }\{ i t } , u t
+1 i :=
u ti v t +1 :
= (cid:80) ni
=1 u t
+1 i //
= v t + u t +1 i t − u ti t end for
Algorithm
[32] input: initial points x 0 ∈ X , ( u 0 i ) ni
=1 ∈ X n ; stepsize γ >v 0 :
= (cid:80) ni
=1 u
i for t = 0 , 1 , . . .
do ˆ x t := x t − γv t pick i t ∈ { 1 , . . . , n } uniformly at random x t
+1 :
=
prox
γnh i ( γnu ti t + ˆ x t ) u t
+1 i t :=
u ti t + 1 γn (ˆ x t − x t +1 ) for every i ∈ { 1 , . . . , n }\{ i t } , u t
+1 i :=
u ti v t +1 :
= (cid:80) ni
=1 u t
+1 i //
= v t + u t +1 i t − u ti t end for
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u ti ) t ∈ N converges to u ⋆i , ∀ i , almost surely.
RandProx-Minibatch with k = 1 becomes the Stochastic Decoupling Method (SDM) proposed in
Mishchenko and Richtárik [58], where strong convexity of g is not exploited, but similar guarantees are derived as in Theorem 5 if µ g = 0 .
Linear convergence of SDM is also proved in Mishchenko and Richtárik [58] in conditions related to ours in Theorems 2 and 4.
Thus, RandProx-Minibatch extends SDM to larger minibatch size k and exploits possible strong convexity of g .
When f = 0 and g = 0 , SDM further simplifies to Point-SAGA [32].
In that case, our results do not apply directly, since there is no strong convexity in f and g any more, but when minimizing the average of functions
h i , with each function supposed to be L -smooth and µ -strongly convex, for some L ≥ µ > 0 , we can transfer the strong convexity to g by subtracting µ 2 ∥ · ∥ 2 to each h i and setting g = µ 2 ∥ · ∥ 2 .
This does not change the problem and the algorithm but our Theorem 5 now applies, and with the right choice of γ , we recover the result in Defazio [32], that the asymptotic complexity of Point-SAGA to reach ϵ -accuracy is O (cid:16)(cid:16) n + (cid:113) nLµ (cid:17) log 1 ϵ (cid:17) , which is conjectured to be optimal.
Thus, RandProx-Minibatch extends Point-SAGA to larger minibatch size and to the more general problem (17) with nonzero f or g .
When n = 1 , there is no randomness and SDM reverts to the DY algorithm discussed in
Appendix H.
B.3. Distributed and federated learning with compression
We consider in this section distributed optimization within the client-server model, with a master node communicating back and forth with n ≥ 1 parallel workers.
This is particularly relevant for federated learning (FL) [44, 51, 53, 57], where a potentially huge number of devices, with their owners’ data stored on each of them, are involved in the collaborative process of training a global machine learning model.
The goal is to exploit the wealth of useful information lying in the heterogeneous data stored across the devices.
Communication between the devices and the distant server, which can be costly and slow, is the main bottleneck in this framework.
So, it is of primary importance to devise novel algorithmic strategies, which are efficient in terms of computation and input: initial estimates ( x 0 i ) ni
=1 ∈ X n , ( u 0 i ) ni
=1 ∈
X n such that (cid:80) ni
=1 u 0 i = 0 ; stepsize γ > 0 ; ω ≥for t = 0 , 1 , . . .
do for i = 1 , . . .
, n at nodes in parallel do ˆ x ti := x ti − γ ∇ f i ( x ti ) − γu ti a ti :=
R t (ˆ x ti ) // send compressed vector a ti to master end for a t := 1 n (cid:80) ni
=1 a ti // aggregation at master // broadcast a t to all nodes for i = 1 , . .
, n at nodes in parallel do d ti := a ti − a t u t
+1 i :=
u ti + 1 γ
(1+ ω )
d ti x t +1 i :
= ˆ x ti − 1 1+ ω d ti end for end for communication complexities.
A natural and widely used idea is to make use of (lossy) compression , to reduce the size of the communicated message [1, 2, 5, 35, 48, 71, 76, 77, 80].
Another popular idea is to make use of local steps [39, 45, 46, 49, 56, 57, 60, 74, 78]; that is, communication with the server does not occur at every iteration but only every few iterations, for instance communication is triggered randomly with a small probability at every iteration.
Between communication rounds, the workers perform multiple local steps independently, based on their local objectives.
Our proposed algorithm RandProx-FL unifies the two strategies, in the sense that depending on the choice of the randomization process R t , we obtain a method with local steps or with compression, or both.
Thus, we consider the problem
Find x ⋆ ∈ arg min x ∈ R d (cid:32) n (cid:88) i
=
f i ( x ) (cid:33)
[Equation]
where d ≥ 1 is the model dimension and n ≥ 1 is the number of parallel workers, each having its own objective function f i .
Every function f
i : R d → R is µ -strongly convex and L -smooth, for some
L ≥ µ > 0 .
We define κ :=
L/µ .
Now, we can observe that (21) can be recast as (1) with K
= Id , U =
X , g = 0 ; that is, as the minimization of f + h , as studied in Section A.1, with
[Equation]
We can note that f is µ -strongly convex and L -smooth, and µ h ∗ = 0 .
Making these substitutions in RandProx-FB yields RandProx-FL , a distributed algorithm well suited for FL, shown above.
In
RandProx-FL , randomization takes the form of linear random unbiased operators R t applied to the vectors sent to the server.
Note that at every iteration, the same operator R t is applied at every node; that is, its randomness is shared.
We can easily check that RandProx-FL is an instance of
RandProx-FB , because of the linearity of the R t and because the property (cid:80) ni
=1 u ti = 0 is maintained at every iteration.
Formally, R t applied as a whole in RandProx-FB consists of n copies of R t applied individually at every node in RandProx-FL , that is why we keep the same notation; in particular, the value of ω is the same in both interpretations.
Interestingly, in RandProx-FL , information about the functions f i or their gradients is never communicated and is exploited completely locally.
This is ideal in terms of privacy.
As an application of Theorem 3, we obtain:
Theorem 10.
In RandProx-FL , suppose that 0 < γ < 2 L f .
Define the Lyapunov function, for every t ≥ 0 ,
[Equation]
where x ⋆ is the unique solution of (21) and u ⋆i := −∇ f i ( x ⋆ ) .
Then RandProx-FL converges linearly: for every t ≥ 0 , E (cid:2)
Ψ t (cid:3) ≤ c t Ψ 0 , where
[Equation]
Also, the ( x ti ) t ∈ N and (ˆ x ti ) t ∈ N all converge to x ⋆ and every ( u ti ) t ∈ N converges to u ⋆i , almost surely.
If R t is the Bernoulli compressor we have seen before in (3) and in Section B.1, RandProx-FL reverts to the Scaffnew algorithm proposed in Mishchenko et al.
[60], which communicates at every iteration with probability p ∈ (0 , 1] and performs in average 1 /p local steps between successive communication rounds.
We have ω = 1 p − 1 .
The analysis of Scaffnew in Theorem 10 is the same as in Mishchenko et al.
With γ = 1 L , the iteration complexity of Scaffnew is O (cid:0) ( κ + 1 p 2 ) log 1 ϵ (cid:1) , and since the algorithm communicates with probability p , its average communication complexity is
O (cid:0) ( pκ + 1 p ) log 1 ϵ (cid:1) .
In particular, with p = 1 √ κ , the average communication complexity of Scaffnew is O (cid:0) √ κ log 1 ϵ (cid:1) .
We now propose a new algorithm with compressed communication: in RandProx-FL we choose, for every t ≥ 0 , R t as the well-known rand - k compressor, for some k ∈ { 1 , . . .
, d } : R t multiplies k coordinates, chosen uniformly at random, of its vector argument by d/k and sets the other ones to zero.
We have ω = dk − 1 .
The iteration complexity with γ = 1 L is O (cid:0) ( κ + d 2 k 2 ) log 1 ϵ (cid:1) and the communication complexity, in terms of average number of floats sent by every worker to the master, is O (cid:0) ( kκ + d 2 k ) log 1 ϵ (cid:1) , since k floats are sent by every worker at every iteration.
Thus, by choosing k = ⌈ d/ √ κ ⌉ , as long as d ≥ √ κ , the communication complexity in terms of floats is O (cid:0) d √ κ log 1 ϵ (cid:1) ; this is the same as the one of Scaffnew with γ = 1 L and p = 1 √ κ , but RandProx-FL with rand - k compressors removes the necessity to communicate full d -dimensional vectors periodically.
Appendix C. Contraction of gradient descent
Lemma 1.
For every γ > 0 , the gradient descent operator Id − γ ∇ f is c γ -Lipschitz continuous, with c γ :=
max(1 − γµ f , γL f − 1) .
That is, for every ( x, x ′ ) ∈
X 2 ,
[Equation]
Proof Let ( x, x ′ ) ∈
X 2 .
By cocoercivity of ∇ f − µ f Id , we have [13, Lemma 3.11] ⟨∇ f ( x ) − ∇ f ( x ′ ) , x − x ′ ⟩ ≥ L f µ f L f + µ f ∥ x − x ′ ∥
+ 1 L
[Equation]
f ∥∇ f ( x ) −
[Equation]
f
[Equation]
x
[Equation]
∥ 2 .
Hence,
[Equation]
Thus, if γ ≤ 2 L f + µ f , since ∥∇ f ( x ) − ∇ f ( x ′ )
∥ ≥ µ f ∥ x − x ′ ∥ , ∥
[Equation]
On the other hand, if γ ≥ 2 L f + µ f , since ∥∇ f ( x ) − ∇ f ( x ′ ) ∥
≤ L f ∥ x − x ′ ∥ , ∥ (Id − γ ∇ f ) x − (Id − γ ∇ f ) x ′ ∥ 2 ≤ (cid:16) 1 − 2 γL
[Equation]
f L f
[Equation]
µ f +
[Equation]
γ 2 −
[Equation]
=
[Equation] [Equation]
x − x
[Equation]
Since max(1 − γµ f , γL f − 1) =
(1 − γµ f if γ ≤ 2 L f + µ f , γL f − 1 otherwise ) ≥ 0 , we arrive at the given expression of c γ .
□
We can note that if γ < 2 L f and µ f > 0 , c γ < 1 .
Appendix D. Proof of Theorem
Let t ∈ N .
Let p t ∈ ∂g (ˆ x t ) be such that ˆ x t = x t − γ ∇ f ( x t ) − γp t − γK ∗ u t ; p t exists and is unique, by properties of the proximity operator.
We also define p ⋆ :=
−∇ f ( x ⋆ ) − K ∗ u ⋆ ; we have p ⋆ ∈ ∂g
( x ⋆ ) .
Let q t := p t − µ g ˆ x t and q ⋆ :=
p ⋆ − µ g x ⋆ .
We have (1 + γµ g )
ˆ x t = x t − γ ∇ f ( x t ) − γq t − γK ∗ u t .
Let w t := x t − γ ∇ f ( x t ) and w ⋆ := x ⋆ − γ ∇ f ( x ⋆ ) .
We define
[Equation]
Then,
[Equation]
[Table]
Moreover,
[Equation]
We have ⟨ q t − q ⋆ , ˆ x t − x ⋆ ⟩ ≥ 0 .
Hence,
[Equation]
so that
[Equation]
On the other hand,
[Equation] [Equation]
Let s t +1 ∈
∂h ∗ (ˆ u t +1 ) be such that ˆ u t +
[Equation]
=
[Equation]
x t − τs t +1 ; s t +1 exists and is unique.
We also define s ⋆ :=
Kx ⋆ ; we have s ⋆ ∈
∂h ∗ ( u ⋆ ) .
Therefore,
[Equation]
Hence,
[Equation]
− γ 1 + γµ g (cid:13)(cid:13) q t − q ⋆ + K ∗ ( u t − u ⋆ ) (cid:13)(cid:13)+ γ (1 − ζ ) (cid:13)(cid:13) K ∗ (ˆ u t +1 − u t ) (cid:13)(cid:13) 2 − 2 ⟨ ˆ x t − x ⋆ , K ∗ (ˆ u t +1 − u ⋆ ) ⟩ + γω ran (cid:13)(cid:13)
ˆ u t +1 − u t (cid:13)(cid:13)
+ 1 τ (cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13) 2 − 1 τ (cid:13)(cid:13)
ˆ u t +1 − u t (cid:13)(cid:13)+ 2 ⟨ ˆ u t +1 − u ⋆ , K (ˆ x t − x ⋆ ) ⟩ − 2 ⟨ ˆ u t +1 − u ⋆ , s t
+1 − s ⋆ ⟩ + ω τ (cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13)≤ γ (1 + γµ g ) (cid:13)(cid:13)
w t − w ⋆ (cid:13)(cid:13) 2 − γ 1 + γµ g (cid:13)(cid:13)
q t − q ⋆ + K ∗ ( u t − u ⋆ ) (cid:13)(cid:13)+ 1 + ω τ (cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13)
+ (cid:18) γ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) − 1 τ (cid:19) (cid:13)(cid:13)
ˆ u t +1 − u t (cid:13)(cid:13)− 2 ⟨ ˆ u t +1 − u ⋆ , s t +1 − s ⋆ ⟩ ≤ γ (1 + γµ g ) (cid:13) (cid:13) w t −
w ⋆
(cid:13) (cid:13) 2 − γ 1 + γµ g
(cid:13) (cid:13) q t − q ⋆ + K ∗ ( u t − u ⋆ ) (cid:13) (cid:13)+ 1 + ω
τ (cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13)
2 − 2 ⟨ ˆ u t +1 − u ⋆ , s t +1 − s ⋆ ⟩ .
By µ h ∗ -strong monotonicity of ∂h ∗ , ⟨ ˆ u t +1 − u ⋆ , s t +1 − s ⋆ ⟩ ≥ µ h ∗ (cid:13)(cid:13)
ˆ u t +1 −
[Equation]
⋆ (cid:13)(cid:13) 2 , and using (26),
[Equation]
Hence,
[Equation]
After Lemma 1,
[Equation]
Plugging this inequality in (27) yields
[Equation]
Ignoring the last term in (28)
[Equation]
we obtain:
[Equation]
Using the tower rule
[Equation]
we can unroll the recursion in (29) to obtain the unconditional expectation of Ψ t +1 .
Since E (cid:2) Ψ t (cid:3) → 0 , we have E (cid:104)(cid:13)(cid:13) x t − x ⋆ (cid:13)(cid:13) 2 (cid:105) → 0 and E (cid:104)(cid:13)(cid:13) u t − u ⋆ (cid:13)(cid:13) 2 (cid:105) → 0 .
Moreover, using classical results on supermartingale convergence [8, Proposition A.4.5], it follows from (29) that Ψ t → 0 almost surely.
Almost sure convergence of x t and u t follows.
Finally, by Lipschitz continuity of ∇ f , K ∗ , prox g , we can upper bound ∥
ˆ x t − x ⋆ ∥ 2 by a linear combination of ∥ x t − x ⋆ ∥and ∥
u t − u ⋆ ∥ 2 .
It follows that E (cid:104)(cid:13)(cid:13) ˆ x t − x ⋆ (cid:13)(cid:13) 2 (cid:105) → 0 linearly with the same rate c and that ˆ x t → x ⋆ almost surely, as well.
□
Appendix E. Proof of Theorem
Let us go back to (28).
Since g = 0 , we have q t =
q ⋆ = 0 and µ g = 0 , so that
[Equation]
input: initial points x 0 ∈ X , v 0 ∈ ran( W ) ; stepsizes γ > 0 , τ > 0 ; ω ≥for t = 0 , 1 , . . .
do ˆ x t := x t − γ ∇ f ( x t ) − γv t d t +1 :=
τ S t ( W ˆ x t − a ) v t +1 :
= v t + 1 1+ ω d t + x t +1 :
= ˆ x t − γd t + end for
We have (cid:13)(cid:13) K ∗ ( u t − u ⋆ ) (cid:13)(cid:13) 2 ≥ λ min ( KK ∗ ) (cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13) 2 .
This yields
E (cid:2)
Ψ t +1 | F t (cid:3) ≤
[Equation]
γ max(
[Equation]
γµ f , γL f
[Equation]
(cid:13) (cid:13) x t − x ⋆ (cid:13)(cid:13)
[Equation]
+
[Equation]
τ
[Equation]
max
[Equation]
[Equation] [Equation]
(cid:19)
[Equation]
The end of the proof is the same as the one of Theorem 1.
□
Let us add here a remark on the PAPC algorithm, which is the particular case of RandProx when ω = 0 , in the conditions of Theorem 2:
Remark 2 (PAPC vs. proximal gradient descent on the dual problem)
If µ f > 0 , f ∗ is µ − 1 -smooth and L − 1 f -strongly convex.
Then f ∗ ◦ − K ∗ is µ − 1 f ∥ K ∥ 2 -smooth and L − 1 f λ min ( KK ∗ ) -strongly convex.
So, if ∇ f ∗ is computable, one can apply the proximal gradient algorithm on the dual problem (2), which iterates u t +1 = prox τh ∗ (cid:0) u t + τK ∇ f ∗ ( − K ∗ u t ) (cid:1) , with τ ∈ (cid:0) 0 , 2 µ f ∥ K ∥ 2 (cid:1) .
If λ min ( KK ∗ ) > 0 , this algorithm converges linearly: ∥ u t +1 − u ⋆ ∥ 2 ≤ c 2 ∥
u t − u ⋆ ∥ 2 with c = max (cid:0) 1 − τL − 1 f λ min ( KK ∗ ) , τµ − 1 f ∥ K ∥ 2 − 1 (cid:1) .
c is smallest with τ = 2 / (cid:0) µ − 1 f ∥ K ∥ 2 + L − 1 f λ min ( KK ∗ ) (cid:1) , in which case
[Equation]
This is much worse than the rate of the PAPC algorithm, since it involves the product of the condition numbers L f /µ f and ∥ K ∥ 2 /λ min ( KK ∗ ) , instead of their maximum.
This is due to calling gradients of f ∗ ◦ − K ∗ , whereas f and K are split, or decoupled, in the PAPC algorithm.
Appendix F. Proof of Theorem 4 and further discussion
We observe that in RandProx-LC and Theorem 4, it is as if the sequence ( u t 0 ) t ∈ N had been computed by the following iteration, initialized with x 0 ∈ X and u 00 :
= P ran( K ) ( u 0 ) :
[Equation]
Then we remark that this is simply the iteration of RandProx , with R t replaced by (cid:101) R t := P ran( K ) R t .
Since its argument r t = τ ( K ˆ x t − b ) is always in ran( K ) , (cid:101) R t is unbiased, and we have, for every t ≥ 0 ,
[Equation]
where (cid:101) F t the σ -algebra generated by the collection of random variables ( x 0 , u 00 ) , . .
, ( x t , u t 0 ) .
Also, ω ran is unchanged.
Therefore, the analysis of RandProx in Theorem 2 applies, with u t replaced by u t 0 and u ⋆ by u ⋆ 0 .
Now, for every u ∈ ran( K ) ,
[Equation]
and using this lower bound in the proof of Theorem 2, with µ h ∗ = 0 , we obtain Theorem 4.
□
Furthermore, the constraint Kx = b is equivalent to the constraint K ∗ Kx =
K ∗ b ; so, let us consider problems where we are given K ∗ K and not K in the first place:
Let W be a linear operator on X , which is self-adjoint, i.e. W ∗ =
W , and positive, i.e. ⟨ W x, x ⟩ ≥ 0 for every x ∈ X .
Let a ∈ ran( W ) .
We consider the linearly constrained minimization problem
[Equation]
Now, we let U :
= X and K
= K ∗ := √ W , where √ W is the unique positive self-adjoint linear operator on X such that √ W √ W = W .
Also, b is defined as the unique element in ran( W ) = ran( K ) such that √ W b = a .
Then (31) is equivalent to (12) and the dual problem is (13).
We consider the Randomized Primal Linearly Constrained minimization algorithm
( RandPriLiCo ), shown above.
We suppose that the stochastic operators S t in RandPriLiCo satisfy, for every t ≥ 0 ,
[Equation]
and
[Equation]
for some ω ≥ 0 , where r t := τW ˆ x t − τa .
In addition, we suppose that the S t commute with √ W : for every t ≥ 0 and x ∈ X ,
[Equation]
This is satisfied with the Bernoulli operators or some linear sketching operators, for instance.
Then
RandPriLiCo is equivalent to RandProx-LC , with S t playing the role of R t and ω ran =
∥ W ∥ ω , ζ = 0 .
Applying Theorem 4 with these equivalences, we obtain:
Theorem 6.
In the setting of (31) , suppose that µ f > 0 .
In RandPriLiCo , suppose that 0 < γ < 2 L f , τ > 0 and γτ ∥ W ∥ (1 + ω ) ≤ 1 .
Define the Lyapunov function, for every t ≥ 0 ,
[Equation]
[16] input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ >ˆ x 0 :
= prox γg (cid:0) x 0 − γK ∗ u 0 (cid:1) for t = 0 , 1 , . . .
do u t +1 :
= prox τh ∗ (cid:0) u t + τK ˆ x t (cid:1) //
x t +1 :
= ˆ x t − γK ∗ ( u t +1 − u t ) ˆ x t +1 :
= prox γg (cid:0) ˆ x t − γK ∗ (2 u t +1 − u t ) (cid:1) end for input: initial points x 0 ∈ X , u 0 ∈ U ; stepsizes γ > 0 , τ > 0 ;
ω
≥ˆ x 0 := prox γg (cid:0) x 0 − γK ∗ u 0 (cid:1) for t = 0 , 1 , . . .
do d t := R t (cid:0) prox τh ∗ ( u t + τK ˆ x t ) − u t (cid:1) u t
+1 :=
u t + 1 1+ ω d t // x t +1 :
= ˆ x t − γK ∗ d t ˆ x t +1 :
= prox γg (cid:0) ˆ x t − γK ∗ ( u t +1 + d t ) (cid:1) end for where u t 0 is the unique element in ran( W ) such that v t = √ W u t 0 , x ⋆ is the unique solution of (31) and u ⋆ 0 is the unique element in ran( W ) such that −∇ f ( x ⋆ ) =
√ W u ⋆ 0 .
Then RandPriLiCo converges linearly: for every t ≥ 0 ,
[Equation]
(34) where
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ almost surely.
RandPriLiCo can be applied to decentralized optimization, like in Kovalev et al.
[52], Salim et al.
[69] but with randomized communication; we leave the detailed study of this setting for future work.
Appendix G. Particular case f = 0 : randomized Chambolle–Pock algorithm
In this section, we suppose that f = 0 .
The primal problem (1) becomes:
[Equation]
and the dual problem (2) becomes:
[Equation]
The PDDY algorithm becomes the Chambolle-Pock (CP), a.k.a.
PDHG, algorithm [16], shown above.
RandProx can be rewritten as RandProx-CP , shown above, too.
In both algorithms, the variable x t is not needed any more and can be removed.
Since f = 0 , L f > 0 can be set arbitrarily close to zero, so that Theorem 1 can be rewritten as:
Theorem 7.
Suppose that µ g > 0 and µ h ∗ > 0 .
In RandProx-CP , suppose that γ > 0 , τ > 0 , γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 .
Define the Lyapunov function, for every t ≥ 0 ,
[Equation]
input: initial points x 0 ∈ X , u 0 ∈ U ; stepsize γ >for t = 0 , 1 , . . .
do
[Equation]
end for input: initial points x 0 ∈ X , u 0 ∈ U ; stepsize γ > 0 ; ω ≥for t = 0 , 1 , . . .
do
[Equation]
u
[Equation]
1 γ (1+ ω ) 2 d t end for where x ⋆ and u ⋆ are the unique solutions to (36) and (37) , respectively.
Then RandProx-CP converges linearly: for every t ≥ 0 ,
[Equation]
(39) where
[Equation] [Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
It would be interesting to study whether the mechanism in the stochastic PDHG algorithm proposed in Chambolle et al.
[18] can be viewed as a particular case of RandProx-CP ; we leave the analysis of this connection for future work.
In any case, the strong convexity constants µ g and µ h ∗ need to be known in the linearly converging version of the stochastic PDHG algorithm, which is not the case here; this is an important advantage of RandProx-CP .
Now, let us look at the particular case K =
Id in (36) and (37).
The primal problem becomes:
[Equation]
and the dual problem becomes:
[Equation]
When K = Id , the CP algorithm with τ = 1 γ reverts to the Douglas–Rachford algorithm, which is equivalent to the Alternating Direction Method of Multipliers (ADMM) [12, 27], shown above.
Therefore, in that case, with ω ran = ω , ζ = 0 and τ = 1 γ (1+ ω ) , RandProx-CP can be rewritten as
RandProx-ADMM , shown above.
Theorem 7 becomes:
Theorem 8.
Suppose that µ g > 0 and µ h ∗ > 0 .
In RandProx-ADMM , suppose that γ > 0 .
For every t ≥ 0 , define the Lyapunov function
[Equation] [Equation]
[31] input: initial points x 0 ∈ X , u 0 ∈ X ; stepsize γ >for t = 0 , 1 , . . .
do
[Equation]
end for input: initial points x 0 ∈ X , u 0 ∈ X ; stepsize γ > 0 ; ω ≥for t = 0 , 1 , . . .
do ˆ x t := prox γg (cid:0)
[Equation]
(cid:1) d
[Equation] [Equation]
u
[Equation]
+
[Equation]
d t end for where x ⋆ and u ⋆ are the unique solutions to (42) and (43) , respectively.
Then RandProx-ADMM converges linearly: for every t ≥ 0 ,
[Equation]
(45) where
[Equation]
(cid:19)
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
Appendix H. Particular case K = Id : randomized Davis–Yin algorithm
After the particular case g = 0 discussed in Section A.1 and the particular case f = 0 discussed in Section G, we discuss in this section the third particular case K
= Id in (1) and (2).
The primal problem becomes:
[Equation]
and the dual problem becomes:
[Equation]
When K = Id , the PDDY algorithm with τ = 1 γ reverts to the Davis–Yin (DY) algorithm [31], shown above.
Therefore, in that case, with ω ran = ω , ζ = 0 and τ = 1 γ (1+ ω ) , RandProx can be rewritten as RandProx-DY , shown above, too.
When g = 0 , RandProx-DY reverts to RandProx-FB and when f = 0 , RandProx-DY reverts to RandProx-ADMM ; in other words, RandProx-DY generalizes
RandProx-FB and RandProx-ADMM into a single algorithm.
Theorem 1 yields:
Theorem 9.
Suppose that µ f > 0 or µ g > 0 , and that µ h ∗ > 0 .
In RandProx-DY , suppose that 0 < γ < 2 L f .
For every t ≥ 0 , define the Lyapunov function,
[Equation]
where x ⋆ and u ⋆ are the unique solutions to (48) and (49) , respectively.
Then RandProx-DY converges linearly: for every t ≥ 0 ,
[Equation]
where
[Equation]
Also, ( x t ) t ∈ N and (ˆ x t ) t ∈ N both converge to x ⋆ and ( u t ) t ∈ N converges to u ⋆ , almost surely.
We can note that in Theorem 9, µ h ∗ > 0 is required.
It is only in the case g = 0 , when RandProxDY reverts to RandProx-FB , that one can apply Theorem 3, which does not require strong convexity of h ∗ .
Appendix I. Convergence in the merely convex case
In all theorems, strong convexity of f or g is assumed; that is, µ f > 0 or µ g > 0 .
In this section, we remove this hypothesis, so that the primal problem is not necessarily strongly convex any more.
But ∇ f ( x ⋆ ) is the same for every solution x ⋆ of (1), and we denote by ∇ f ( x ⋆ ) this element.
We define the Bregman divergence of f at points ( x, x ′ ) ∈
X 2 as
[Equation]
For every t ≥ 0 , D f ( x t , x ⋆ ) is the same for every solution x ⋆ of (1), and we denote by D f ( x t , x ⋆ ) this element.
D f
( x t , x ⋆ ) can be viewed as a generalization of the objective gap f ( x t ) − f ( x ⋆ ) to the case when ∇ f ( x ⋆ ) ̸ = 0 .
D f
( x t , x ⋆ ) is a loose kind of distance between x t and the solution set, but under some additional assumptions on f , for instance strict convexity, D f ( x t , x ⋆ ) → 0 implies that the distance from x t to the solution set tends to zero.
Also, D f ( x t , x ⋆ ) ≥
L f ∥∇ f ( x t ) − ∇ f ( x ⋆ ) ∥ 2 , so that D f ( x t , x ⋆ ) → 0 implies that (cid:0) ∇ f ( x t ) (cid:1) t ∈ N converges to ∇ f ( x ⋆ ) .
Theorem 11.
In RandProx , suppose that 0 < γ < 2 L f , τ > 0 , and γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 .
Then D f ( x t , x ⋆ ) → 0 , almost surely and in quadratic mean.
Moreover, for every t ≥ 0 , we define ¯ x t := 1 t (cid:80) ti =1 x i .
Then, for every t ≥ 0 ,
[Equation]
If, in addition, µ h ∗ > 0 , there is a unique dual solution u ⋆ to (2) and ( u t ) t ∈ N converges to u ⋆ , in quadratic mean.
The counterpart of Theorem 2 in the convex case is:
Theorem 12.
Suppose that g = 0 , and that λ min ( KK ∗ ) > 0 or µ h ∗ > 0 .
In RandProx , suppose that 0 < γ < 2 L f , τ > 0 , and γτ (cid:0) (1 − ζ )
∥ K ∥ 2 + ω ran (cid:1) ≤ 1 .
Then there is a unique dual solution u ⋆ to (2) and ( u t ) t ∈ N converges to u ⋆ , in quadratic mean.
We can derive counterparts of the other theorems in the same way.
These theorems apply to all algorithms presented in the paper.
For instance, Theorems 11 and 12 apply to Scaffnew [60], a particular case of RandProx-FL seen in Section B.3, and provide for it the first convergence results in the non-strongly convex case.
I.1. Proof of Theorems 11 and
Proof of Theorem 11 We have, for every ( x, x ′ ) ∈
X 2 ,
[Equation]
where the second inequality follows from cocoercivity of the gradient.
Moreover, for every ( x, x ′ ) ∈
X 2 , D f ( x, x ′ ) ≤ ⟨∇ f ( x ) − ∇ f ( x ′ ) , x − x ′ ⟩ .
Therefore, in the proof of Theorem 1, for every primal–dual solution ( x ⋆ , u ⋆ ) and t ≥ 0 , since (cid:13)(cid:13) w t − w ⋆ (cid:13)(cid:13) 2 =
[Equation]
(27) yields
[Equation]
Ignoring the last term, this yields
[Equation]
) with c = 1 − 2 τµ h ∗ (1+ ω )(1+2 τµ h ∗ ) in (54).
Using classical results on supermartingale convergence [8,
Proposition A.4.5], it follows from (55) that Ψ t converges almost surely to a random variable Ψ ∞ and that
[Equation]
almost surely .
Hence, D f ( x t , x ∗ ) → 0 almost surely.
Moreover, for every T ≥ 0 ,
[Equation]
56) and
[Equation]
Therefore, E (cid:2) D f ( x t , x ∗ ) (cid:3) → 0 ; that is, D f ( x t , x ∗ ) → 0 in quadratic mean.
The Bregman divergence is convex in its first argument, so that for every T ≥ 0 ,
D
[Equation]
t =
[Equation]
D f (
[Equation]
Combining this last inequality with (56) yields
[Equation]
Now, if µ h ∗ > 0 , then c < 1 in (54), and since Ψ t converges almost surely to Ψ ∞ , it must be that E (cid:104)(cid:13)(cid:13) u t − u ⋆ (cid:13)(cid:13) 2 (cid:105) → 0 .
□
Proof of Theorem 12 Considering the proof of Theorem 2, the same arguments as in the proof of
Theorem 11 apply, with c in (54) now equal to
[Equation]
Hence, E (cid:104)(cid:13)(cid:13)
u t − u ⋆ (cid:13)(cid:13) 2 (cid:105) → 0 .
□