PARS-Push: Personalized, Asynchronous and Robust Decentralized Optimization
Authors
Abstract
We study the decentralized multi-step ModelAgnostic Meta-Learning (MAML) framework where a group of n agents seeks to find a common point that enables “few-shot” learning (personalization) via local stochastic gradient steps on their local functions.
We formulate the personalized optimization problem under the MAML framework and propose PARSPush, a decentralized asynchronous algorithm robust to message failures, communication delays, and directed message sharing.
We characterize the convergence rate of PARS-Push for smooth and strongly convex and smooth and non-convex functions under arbitrary multi-step personalization.
Moreover, we provide numerical experiments showing its performance under heterogeneous data setups.
1. Introduction
Distributed and decentralized optimization problems considers a set of n agents where the objective is to jointly minimize the sum of a set of functions where each function is accessible to one agent only.
Conventionally, we consider the following stochastic optimization problem:
[Equation]
where f i ( z ) :
= E
[Equation]
∼ p
[Equation]
[ ℓ ( z , ξ )] , (
[Equation]
the function ℓ : R d × S → R generally denotes the loss function for a model that we seek to learn in the parameter space R d , and S denotes the sample space of random variable ξ which represents the data.
In (2), f i ( z ) is the population loss of model z over samples selected uniformly with respect to a distribution p i , i.e., agent i data distribution.
In general, the data distribution is unknown and agents have access to empirical realizations of ξ only.
In a homogeneous data setting, all the agents have the same data distributions.
Thus, the quality of a minimizer of the empirical loss of f i ( · ) on a finite data set of realizations of ξ will improve as the number of data samples increases.
Therefore, minimizing an empirical estimator of f ( · ) is guaranteed to achieve lower costs with a high probability as it will include more realizations from the data space.
On the other hand, if p i ’s are different, there is no guarantee that a minimizer of the empirical estimator of f ( · ) will have a lower cost when evaluated on each f i ( · ) .
A minimizer of f ( · ) will only perform well on f i ( · ) on average.
Finn et al.
(2017) proposed Model-Agnostic MetaLearning (MAML) as an optimization-based technique to address data heterogeneity for any optimization problem that can be solved by (stochastic) gradient descent.
Fallah et al.
(2020a)
; Ji et al.
; Rajeswaran et al.
Finn et al.
(2019) studied the convergence of MAML for different function classes.
Moreover, this problem has been extensively studied for various learning setups (Finn et al., 2019; Fallah et al., 2021b; Collins et al., 2020; Fallah et al., 2021a; Charles & Kone ˇ cn ` y, 2021; Kayaalp et al., 2022).
Most of prior works analyze MAML with one gradient descent step as the personalization budget, while Fallah et al.
(2021a)
; Ji et al.
(2022) study this problem with multi-step fine-tuning budget.
Moreover, several recent works have studied personalization in collaborative learning as a form of multi-task learning (Fallah et al., 2020b;
Dinh et al., 2020; Gasanov et al., 2021; Collins et al., 2021).
Most of the recent works on the personalized collaborative learning problem, often assume the existence of reliable synchronized communications to a single server (Fallah et al., 2020b;
Dinh et al., 2020;
Gasanov et al., 2021;
Collins et al., 2021), or synchronous local communications over an undirected network (Kayaalp et al., 2022).
These strong communication assumptions restrict the application of personalization for modern machine learning problems.
Xie et al.
; Chen et al.
; Nguyen et al.
(2021) propose updates with staleness to deal with asynchronous communications with the server in distributed scenarios.
Here, we leverage recent works on unreliable communications for distributed/decentralized inference (Mojica-Nava et al., 2021), consensus (Hadjicostis et al., 2015; 2018;
Spiridonoff et al., 2020b; Assran et al., 2019), and optimization problems (Spiridonoff et al., 2020b; Assran et al., 2019; Xie et al., 2019; Chen et al., 2021; Nguyen et al.,
The main objective of this work is to study the convergence of the personalized multi-step MAML problem under robust asynchronous communications over directed networks.
We summarize our contributions as follows:
• We study the multi-step MAML cost for smooth and strongly convex and non-convex functions.
We show that the multi-step model preserves the function class, and we explicitly characterize its condition number as a function of the personalization budget.
• We propose an algorithm named PARS-Push, to minimize the personalized optimization objective function on a directed network with asynchronous communications under message delays and losses.
• We establish convergence guarantees for PARS-Push and provide numerical examples to show the advantages of our method for collaborative learning with heterogeneous data samples over the agents.
The remainder of this paper is arranged as follows.
In Section 2, we introduce the personalization setup and discuss the communication framework between the agents.
In Section 3, we describe the PARS-Push algorithm for personalized decentralized optimization and present the assumptions and convergence results.
We discuss the sketch of proofs in Section 4, and provide numerical experiments in
Section 5.
We end by concluding the remarks in Section 6.
2. Problem Setup
This section formalizes the personalized decentralized optimization problem under asynchronous communications over a directed network.
Moreover, we discuss the problem setup and challenges for the personalization and communication network model.
Personalization
via
Multi-Step MAML
: We denote the ˜ f
i ( z , D i ) as the empirical cost of agent i ,
[Equation]
) where D i is a data batch ( |D i | = b ) with samples drawn from a probability distribution p i , hence E p i ˜ f
i ( z , D i ) =
f i ( z ) .
The goal in personalized distributed learning under the
MAML approach is to collaboratively find z that performs well on average after applying u ≥ 1 (integer) local stochastic gradient descent steps for each agent.
Formally, instead of solving Problem (1), we seek to solve the following problem:
[Equation]
where Ψ i ( z , D i ) :
=
z − α ∇ ˜
f
i ( z , D i ) with personalized learning rate α ≥ 0 , and independent data batches D test i,r selected uniformly at random with respect to distribution p i , for all i ∈ [ n ] , and r ∈ { 0 }∪ [ u ] .
For simplicity of exposition, we assume that all batches have the same size of b , i.e., |D test i,r | = b .
Note that u represents the available budget for few-shot learning.
For example, Problem (4) becomes
Problem (1) when u = 0 , i.,e.
, no budget for personalization.
On the contrary, u →∞ implies there is no need for cooperation, as each agent can find a minimizer of its local function.
Communication Network Model: We consider a static, directed, and strongly connected network G = { [ n ] , E} with no self-loops, and E ⊆
[ n ] × [ n ] , where ( i, j ) ∈ E if there is an edge from node i to j .
For each agent i ∈ [ n ] , we define N − i =
{ j s.t.
( j, i ) ∈ E} as the set of in-neighbors to i , as well as N + i = { j s.t.
( i, j ) ∈ E} as the set of out-neighbors from i , where the communications take place over the edges of G .
We also denote d − i =
|N − i | , d + i = |N
+ i | , and m = (cid:80) ni
=1 d − i .
In this work, we assume that the agents perform updates and communicate asynchronously.
We moreover consider the possibility of message losses and delays.
Hence, inspired by Spiridonoff et al.
(2020b), we assume the following communication model.
Assumption 1 (Spiridonoff et al.
(2020b)) .
The communication network has the following properties:
(a) graph G is static, directed, and strongly connected (b) the delays on each communication link ( i,
j ) ∈
E are bounded by some Γ d ≥ 1 , (c) each communication link ( i,
j ) ∈
E fails at most Γ f ≥ 0 consecutive times, (d) every agent i ∈ [ n ] wakes up and performs updates at least once every Γ w ≥ 1 iterations.
Assumption 1 is a rather weak model that allows robust asynchronous communications under the possibility of idle agents, as well as a finite number of consecutive failures and bounded delays in the communication.
Moreover, it implies an effective maximum delay Γ e := Γ w +Γ d − 1 , where for each receives a message from its in-neighbors at least once every Γ s :
=
Γ w (Γ f +1)+Γ e iterations.
Based on this assumption, we will discuss an augmented communication model in Section 3.
3. PARS-Push Algorithm
We first introduce empirical cost
[Equation]
where ϑ i = {D i,r } ur =0 denotes a set of independent data batches uniformly drawn from distribution p i , and
F ( u ) i ( z ) =
E p i (cid:2) ˜ F ( u ) i ( z , ϑ i ) (cid:3) .
Several prior works (Finn et al., 2017; Fallah et al., 2020b; Ji et al., 2022; Kayaalp et al., 2021) consider surrogate cost functions with biased gradient estimators which lead to a non-vanishing bias in the convergence.
Following Fallah et al.
(2021a;b), we minimize the cost in (4) which has an unbiased stochastic estimators of its deterministic gradients for each i ∈ [ n ] .
Algorithm
1: Initialize: y i = 1 , κ i = − 1 , ϕ x i = 0 , ϕ yi = 0 , ∀ i ∈ [ n ] , and κ ij = − 1 , ρ x ij = 0 , ρ yij = 0 , ∀ ( j, i )
∈ E .
2: for t = 0 , 1 , 2 , . . .
, in parallel for all i ∈ [ n ] do 3: if node i wakes up then 4: η i ( t ) :
= t (cid:80) r
= κ i +1 θ ( r ) 5: w (0) i :=
z i 6: for r = 0 , 1 , 2 , . . .
, u − 1 do 7:
Sample a batch D ti,r with size b from p i 8: w ( r +1)
i := w ( r ) i − α ∇ ˜
f
i (cid:16) w ( r ) i , D ti,r (cid:17) 9: end for 10:
Sample a batch D ti,u with size b from p i 11: x i := x i − η i ( t ) (cid:20) u − 1 (cid:81) r = (cid:16) I − α ∇ 2 ˜ f
i (cid:16) w ( r ) i , D ti,r (cid:17)(cid:17)(cid:21) ×∇ ˜ f
i ( w ( u ) i , D ti,u ) 12: κ i :=
t 13: x i := x i d
+ i +1 ,
y i :
=
y i
d + i + 14: ϕ x i :=
ϕ x i + x i , ϕ yi :
= ϕ yi
+ y i 15:
Node i sends ( ϕ x i , ϕ yi , κ i ) to N
+ i 16:
R i := messages received from N − i 17: for (cid:0) ϕ x j , ϕ yj , κ j (cid:1) in R i do 18: if κ j > κ ij then 19: ρ ∗ x ij := ϕ x j , ρ ∗ yij := ϕ yj , κ ij := κ j 20: end
if 21: end for 22: x i := x i + (cid:80) j ∈N − i ( ρ ∗ x ij − ρ x ij ) 23: y i :=
y i + (cid:80) j ∈N − i ( ρ ∗
yij − ρ yij ) 24: ρ x ij := ρ ∗ x ij , ρ yij := ρ ∗ yij , z i := x i y i 25: end if 26: end for
We consider vector x i ( t ) ∈ R d , and slack scalar y i ( t ) ∈
R
+ as the parameters of node i ∈ [ n ] , where z i ( t ) ∈
R d represents the ratio of x i ( t ) /y
i ( t ) .
Moreover, each agent i ∈ [ n ] allocates ϕ x i ( t )
∈
R
d and ρ x ij ( t ) ∈ R d , for all j ∈ N − , to keep the sum of x i ( t ) and x j ( t ) , respectively over the time.
Similarly, node i has the same parameters ϕ yi ( t )
∈ R + and ρ yij ( t ) ∈ R + for variables
y
i ( t ) and y j ( t ) .
For the sake of simplicity, we drop t from the description of the algorithm.
At any round t that node i ∈ [ n ] is idle, its parameters will not be updated.
Once node i wakes up, it performs three set of operations.
First, agent i selects u +1 independent data batches ϑ ti =
{D ti,r } ur =0 with respect to distribution p i , and after performing u steps of stochastic gradient descent starting from z i
(Lines 5-10), agent i computes an unbiased stochastic gradient of (4), and then updates x i in Line 11.
Second, node i updates its parameters x
i , y i , ϕ x i , and ϕ yi according to Line 13-14, and sends the running sum parameters to its out-neighbors N + i (Line 15).
Finally, agent i processes the received messages from its in-neighbors N − i which leads to selecting the most recent updates (Lines 16-21).
Consequently, agent i updates x i , y i , and z i in Lines 22-24 by combining newly received messages.
In a nutshell, PARS-Push contains two key operations, (i) robust asynchronous aggregation over a virtual, augmented graph, and (ii) stochastic gradient descent with respect to unbiased stochastic gradients of (4).
Dynamics of the Update Rule: Next, we introduce a linear formulation for Algorithm 1.
The variables x i ( t ) and y i ( t ) denote node i ’s parameters at round t .
Next, we will discuss the update model for vectors x i ( t ) only, the same discussion would also hold for y
i ( t ) .
According to Spiridonoff et al. (2020b, Section 2), we can model message losses and delays between every two nodes ( j, i ) ∈ E , using Γ e +1 additional virtual nodes, where one of them represents the information that has not successfully been transferred to node i , and the other Γ e nodes indicate the information that will be delivered with a delay. We formally define variables ˆ x lji ( t ) to denote the message sent from node j to i and arriving with an effective delay of l , for all l ∈ [Γ e ] . We also consider variable ˜ x ji ( t ) as the information that has failed to be sent.
We moreover define indicator variables τ
i ( t ) and τ lji ( t ) .
τ i ( t ) = 1 if node i wakes up at time t , and τ i ( t )
=otherwise.
{+Similarly, τ lji ( t ) = 
+}
{+if τ i ( t ) = 1 and the sent message from node j to i arrives after an effective delay of l ∈ [Γ e ] rounds, and τ l ji ( t ) = 
+}
otherwise.
We now can write the update rule of Algorithm 1 as follows Spiridonoff et al.
(2020b):
[Equation]
t
[Equation]
t
[Equation]
F ( u
[Equation]
i ( z i ( t ) , ϑ ti ) , x i ( t +1) :
= (cid:18) 1 − τ i ( t )
+ τ i ( t )
d
+ i +(cid:19) x i ( t + 12 )+ (cid:88)
j ∈N − i x 1 ji ( t ) , ˆ x lji ( t +1) :
= τ lji ( t ) (cid:34) ˜
x ji ( t )+ x j ( t ) d + j +(cid:35) + 1 { l< Γ e } ˆ x l +1 ji ( t +1) , ˜ x l ji ( t +1) :
= (cid:32) 1 − Γ d (cid:88)
[Equation]
[Equation]
where ϑ ti =
{D ti,r } ur =0 is a set of u +1 independent data batches uniformly drawn from p i at round t .
A similar update rule holds for y i ( t ) , except for the gradient descent step of (6).
Let X ( t ) ∈ R ( n + m ′ )
× d be the concatenation of x i ( t ) ⊤ , ˜ x ij ( t ) ⊤ , and ˆ x lij ( t ) ⊤ for all i ∈ [ n ] , ( i, j ) ∈ E , and l ∈ [Γ e ] , where m ′ :
=
(Γ e +1) m .
Similarly, y ( t ) ∈ R n + m ′ denotes the vector containing y
i ( t ) , ˜ y
ij ( t ) , and ˆ y lij ( t ) .
Additionally, for all ( i, j ) ∈ E and l ∈ [Γ e ] , virtual parameters ˜ x ij (0) , ˆ x lij (0) , ˜
y ij (0) , and ˆ y lij (0) are all initialized to zero.
As a consequence, one can show that the following linear system describes Algorithm 1:
[Equation]
where { M ( t ) } t ≥ 0 is a sequence of column stochastic mixing matrices of size ( n + m ′ )
× ( n + m ′ ) , and matrices ∆ ( t ) ∈ R ( n + m ′ )
×
d with rows as follows: [
[Equation]
Note that each matrix M ( t ) is associated with an augmented virtual graph over n + m ′ nodes.
According to Assumption 1, one can check that these augmented virtual graphs build a sequence of time-varying Γ s -strongly connected graphs describing Algorithm 1.
For more details on the specific formulation of the robust asynchronous communications, see Spiridonoff et al.
(2020a). Next section discusses the convergence of (7). 4. Convergence Result This section states our main result, which shows the convergence of PARS-Push under standard assumptions for the class of smooth and strongly convex and smooth and nonconvex functions.
We consider a series of standard assumptions on the function ℓ ( ., ξ ) , for almost all ξ ∈ S .
Our assumptions are commonly used in several prior works (Finn et al., 2019; Fallah et al., 2020b; 2021a;b; Ji et al., 2022).
Assumption 2.
The function ℓ ( ., ξ ) , for almost all ξ ∈S , is twice continuously differentiable and bounded from below.
Furthermore, the following properties hold for all z , ˆ z ∈ R d :
(i) there exist constant G such that ∥∇ ℓ ( z , ξ ) ∥ ≤ G , (ii) ℓ ( ., ξ ) is L -smooth,
i
e., ∥∇ ℓ ( z , ξ ) − ∇ ℓ (ˆ z , ξ ) ∥
≤ L ∥
z − ˆ z ∥ , (iii) the Hessian of ℓ ( ., ξ ) is H -Lipschitz, i.e. (cid:13)(cid:13) ∇ 2 ℓ ( z , ξ ) − ∇ 2 ℓ (ˆ z , ξ ) (cid:13)(cid:13)
≤ H ∥
z − ˆ z ∥ , [Equation] iv) [Equation]
[Equation]
ξ ) is µ
[Equation]
strongly convex
[Equation]
i
[Equation]
e.
[Equation]
Now, we are ready to discuss the convergence results.
⋄ Smooth & Strongly Convex Functions:
Here, we first present the convergence of Algorithm 1 for solving Problem (4) under Assumptions 1 and 2.
Before stating the main result, let us introduce the following lemmas.
Lemma 1.
Suppose that Assumptions 2(i)-(iv) hold.
Then, for any
[Equation]
strongly convex and
[Equation]
for all i ∈ [ n ] , and z ∈ R d .
Proofs can be found in Appendices C and D.
Lemma 1 implies strong convexity and smoothness of (4) under appropriate α .
One can check that ε 1 ε 2 µ ≤ ˆ µ ( u ) and (1+ ε 1 − ε 1 ε 2 )
L ≥ ˆ L ( u ) .
Note that a larger value of α implies higher personalization level.
Nevertheless, one can infer from Lemma 1 that the condition number ( ˆ L ( u ) / ˆ µ ( u ) ) of the surrogate cost F ( u ) i ( z ) increases with larger α .
Lemma 2.
Let Assumptions 2(i)-(iv) hold, and α as in
Lemma 1.
Then for all i ∈ [ n ] and z ∈ R d , we have:
[Equation]
where ˆ σ ( u ) 2 := 4(1 − αµ ) 2 u G 2 and ϑ i = {D i,r } ur =0 represents a set of data batches uniformly drawn from p i .
Lemma 2 indicates a bounded variance between the deterministic and stochastic gradients for each agent i ∈ [ n ] .
It is worth mentioning that Assumptions 2(i)-(iv) can be relaxed if we borrow Assumption 4 in Fallah et al.
(2020b).
Proposition 1 (Smooth and Strongly Convex) .
Let Assumptions 1 and 2 hold, stepsize α be as in Lemma 1, θ ( t )
= 1 ˆ µ ( u ) t , for t ≥ 1 , and θ (0)
= 0 .
Then, for all i ∈ [ n ] , the following property holds for the iterates of Algorithm 1:
[Equation]
where ˆ µ ( u ) and ˆ σ ( u ) as defined in Lemma 1 and Lemma 2.
Proposition 1 suggests a sublinear convergence rate
O (1 /T ) for Algorithm 1 to solve (4), under Assumptions 1-(iv), where all of the decentralization terms ( γ, δ, λ ) are associated with O (1 /T 3 / 2 ) . [Figure] [Figure] u
[Figure]
s t
E rr o r [Figure] time, optimizing the surrogate loss in (4) yields a lower error compared to (1) in heterogeneous data settings. We move the result on smooth non-convex functions to Appendix B.
5. Numerical Experiment
In this section, we study a decentralized linear regression problem with regularization on a synthetic heterogeneous data.
We consider a random vector β ∗ ∈ R d , and a set of vectors { β ∗ i } i ∈ [ n ] , where each β ∗ i ∈
R
d is obtained by perturbing β ∗ with Gaussian noise.
Hence, we define the q -th data sample on agent i as follows:
[Equation]
where ξ iq =
( b iq , a iq ) represents a data sample, and ζ iq ∼ N ( 0 , σ 2 I ) .
We can formally define f i ( z ) as follows:
[Equation]
We consider a setting with n = 10 agents, where each agent maintains data samples in a d = 30 dimensional space.
We also consider batches with b = 20 samples, and selecttest data points for each agent i ∈ [ n ] .
For the communication, we select a fixed, directed, and strongly connected Erd˝os-R´enyi network with a low connection probability ( p = 0 . 15 ).
We simulate the asynchronous setup with Γ w = 2 , Γ d = 2 , and Γ f = 1 , for 6000 iterations.
We compare the average test error of (11) under personalization with that of (1) after personalization with the same budget u in Fig.
In words, we consider the surrogate loss in (4), and after obtaining z ∗ ( u ) for each u ∈ [5] , we update the personalized parameters under u consecutive stochastic gradient descent steps.
We also solve (1) under the same communication setting, and for each u ∈ [5] , we apply u steps of stochastic gradient descent after the train phase to have a fair comparison. [Figure]
Figure 2 shows the training error and validates that PARSPush is robust to asynchronous communications with message losses and delays, compared to stochastic gradientpush (Assran et al., 2019), which is state-of-the-art for delayed asynchronous communications
6. Conclusions
This work studied personalized decentralized optimization under robust asynchronous communications over directed networks.
We considered the multi-step MAML problem under heterogeneous data setting and proposed PARS-Push algorithm to solve this problem with arbitrary u personalization budget.
We showed the convergence of our method for smooth and strongly convex, and non-convex functions.
We proposed a numerical setup to illustrate the convergence and personalization of our method.
A. Discussion on Personalization Dynamic
Some formulations of Meta-Learning consider the slightly different scenario where one has access to a set of functions { f i } (Nichol et al., 2018).
However, instead of finding the minimizer of their average as in (1), Meta-Learning seeks to find a point that achieves a smaller function value on each f i after a small number of local steps of a selected optimization algorithm is applied when compared with the same number of steps with random initialization.
From the machine learning perspective, one can consider each of the local functions f i ( · ) as an individual task to be learned based on some finite dataset.
Minimizing the empirical approximation of f i ( · ) will correspond to finding an approximate model that performs well on that task.
However, in some scenarios, the number of data points for a specific task might be limited.
Thus, leveraging the data from other tasks might improve performance even with limited data.
In practice, Meta-Learning translates into the task of finding a new objective function F ( · ) that depends on { f i ( · ) } such that when minimizing an empirical estimator of F ( · ) one can achieve good performance locally on f i ( · ) with only a few steps of the desired optimization method, or few-shot learning.
Usually, the number of available local steps is termed the personalization budget.
In (4) when u = 1 , we have:
[Equation]
Compared to (1), considering (11) implies an initial point z ∗ (1) such that each agent i can perform one step of stochastic gradient descent with respect to its local cost function to obtain a personalized model w ∗ ( [Equation]
i
[Equation]
Figure 3 illustrates the personalization mechanism for a case of n = 2 agents with u = 2 steps of local stochastic gradient descent, where after two steps of personalization, the solution to (4) has a lower average cost compared (1).
The choice of cost function in (4) enables us to efficiently compute the stochastic gradient as an unbiased estimator of ∇ F ( u ) i ( z ) , which we will discuss in Section 3.
[Figure]
[-Similarly, τ l ji ( t ) = 
-]
B. Convergence Result: Non-Convex
We show first-order stationary convergence of Algorithm 1 under Assumptions 1-(iii) and Γ w =1 .
Note that Γ w =1 also incorporates message losses and asynchronous communications.
However, each node must compute a local gradient and perform consensus every round.
Before stating the non-convex results, let us state the following lemmas.
Lemma 3.
Let Assumptions 2(i)-(iii) hold.
Then, for α ≥ 0 , F ( u ) i ( z ) in (4) is ˆ L ( u ) -smooth as follows: for all i ∈ [ n ] and z ∈ R d , we have:
[Equation]
where ˆ σ ( u ) 2 = 4(1 + αL ) 2 u G 2 .
Lemma 3 indicates that (4) is smooth with ˆ L ( u ) ≥ L .
Lemma 3 does not require an upper bound on α .
However, ˆ L ( u ) grows exponentially with α .
Moreover, this lemma shows an upper bound on the variance of stochastic gradients.
Now, we are ready to present the convergence result for smooth and non-convex functions.
Theorem 1 (Smooth and Non-Convex) .
Let Assumptions 1 and 2(i)-(iii) hold with Γ w :=1 , stepsizes α ≥ 0 , θ ( t ) =
√ n ˆ L ( u ) √ T , and x i (0):= 0 , for all i ∈ [ n ] .
Then, the following property holds for the iterates of Algorithm 1: for any T ≥ n
[Equation]
where ˆ L ( u ) and ˆ σ ( u ) as in Lemma 3.
Theorem 1 shows a sublinear convergence to a first-order stationary point with rate O (1 /T 1 / 2 ) for Algorithm 1 under
Assumption 1 with Γ w =1 .
It is worth mentioning that this assumption may be relaxed with a uniformly probabilistic wake up once in every Γ w ≥ 1 rounds, but we leave this for future studies.
The terms concerning the network, i.e., λ, δ , appear in O (1 /T ) .
C. Proof of Lemma 1, Lemma 2, and Proposition
Before proving Lemma 1, let us state the following lemma from Finn et al.
Lemma 4 (Mean Value Inequality) .
Let Λ : x ∈ R d → R m be a differentiable function.
Let ∇ Λ be the function Jacobian, such that ∇ Λ i,j = ∂
Λ i ∂ x j .
Let M = max x ∈ R d ∥∇ Λ ( x ) ∥ .
Then, we have
[Equation]
Note that F ( u ) i ( z ) is the average of ˜ F ( u )
i ( z , ϑ i ) that also implies ∇ ˜ F ( u ) i
( z , ϑ i ) is an unbiased stochastic estimator of ∇ F ( u ) i ( z ) .
Therefore, it is enough to show that ∇ ˜ F ( u ) i ( ., ϑ i ) is ˆ L ( u ) -smooth
and ˆ µ ( u ) -strongly convex, because
[Equation]
) for every x , y ∈ R d , and any set ϑ i of u +1 data batches drawn from distribution p i .
This also implies that:
[Equation]
[Equation]
By a similar argument, it can be seen that ˜ f
i ( ., D i,r ) is µ -strongly convex and L -smooth, due to Assumption 2, for all i ∈ [ n ] , and D i,r data batches drawn uniformly from p i .
Before proving the smoothness and strong-convexity for ˜ F ( u ) i ( ., ϑ i ) , let us discuss the Mean Value Inequality for Ψ i ( w , D i ) .
Note that by definition, we know that all properties in Assumptionhold for ˜
f i ( ., D i ) , for all i ∈ [ n ] and data batches D i uniformly sampled from p i .
Therefore we have:
[Equation]
[Equation]
for all z ∈ R d , and data batches D i drawn from distribution p i .
This indicates that the following property holds for ∇ Ψ i [Equation] w , D
[Equation]
= I − α ∇ 2 ˜ f
i ( w , D i ) for Ψ i ( w , D i ) :
[Equation]
15) where due to Lemma 4, the following property holds:
[Equation]
[Equation] for all i ∈ [ n ] and uniformly sample data batches D i with respect to p i .
According to what we discussed so far, we are now ready to prove the smoothness and strong convexity of ˜ F ( u ) i .
First note that, for all w ∈ R d ,
[Equation]
17) where w ( r +1) i = w ( r ) i
[Equation]
∇
[Equation] i ( [Equation] i ,
[Equation]
i, [Equation] , r = 0 , 1 , . . , u − 1 , (18) and w
[Equation]
[Equation]
where ϑ i = {D i,r } . We also know that:
[Equation]
Now, consider two points w , v ∈ R d .
According to the definition in (17) as well as the property in (16), we can see that for any two points w , v ∈ R d :
[Equation]
We first show that ˜ F ( u ) i ( ., ϑ i ) is ˆ L ( u ) -smooth, by providing an upper bound on the following expression:
[Equation]
To complete the proof for smoothness
[Equation]
we now propose a recursive upper bound on ∥Q i,u − 1 ∥ :
[Equation]
[Equation] [Equation] (cid:17)(cid:17)(cid:35)(cid:16) [Equation]
(cid:17)(cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (34)
[Equation]
Consequently, we can infer the following recursion:
[Equation]
and we also have:
[Equation]
According to (39) and (40)
[Equation]
we can see that:
[Equation]
As a result of (25) and (41)
[Equation]
we can conclude the smoothness of ˜ F ( u ) i ( ., ϑ i ) as follows:
[Equation]
Before showing the strong convexity result [Equation] let us show a similar property to 21:
[Equation]
= (cid:13)(cid:13)(cid:13) w ( u − 1)
i − α ∇ ˜ f i ( w ( u − 1) i , D i,r ) − v ( u − 1) i + α ∇ ˜
f i ( v ( u − 1) i , D i,r ) (cid:13)(cid:13)(cid:13) (47) tri. ineq. ≥ (cid:13)(cid:13)(cid:13) w ( u − 1) i − v ( u − 1) i
(cid:13)(cid:13)(cid:13) − α (cid:13)(cid:13)(cid:13) ∇ ˜ f i ( w ( u − 1) i , D i,r ) − α ∇ ˜ f i ( v ( u − 1) i , D i,r ) (cid:13)(cid:13)(cid:13) (48)
Assump.
2 ≥ (cid:13)(cid:13)(cid:13) w ( u − 1)
i − v ( u − 1) i
(cid:13)(cid:13)(cid:13) − αL (cid:13)(cid:13)(cid:13) w ( u − 1)
i − v ( u − 1) i
(cid:13)(cid:13)(cid:13) (49) ≥ (1 − αL ) (cid:13) (cid:13)(cid:13) w ( u − 1)
i − v ( u − 1)
i (cid:13) (cid:13)(cid:13) (50) ≥ (1 − αL ) 2 (cid:13)(cid:13)(cid:13) w ( u − 2)
i − v ( u − 2) i
(cid:13)(cid:13)(cid:13) (51) ...
[Equation]
[Equation]
Now, we are ready to show that ˜ F ( u )
i
[Equation]
, ϑ i ) is ˆ µ ( u ) -strongly convex, by providing a lower bound on the following expression:
[Equation]
[Table]
[Equation]
where by applying (41)
[Equation]
we can see that:
[Equation]
Equations (46) and (61) conclude the proof of Lemma 1.
The choice of personalized stepsize α , ensures that ˆ µ ( u ) is a positive constant.
Indeed, given arbitrary integer u> 0 , ε 1 , ε 2 ∈ (0 , 1) and personalized stepsize α as in Lemma 1, we have ˆ µ ( u ) ≥ ε 1 ε 2 µ , and ˆ L ( u ) ≤ (1+ ε 1 −
ε 1 ε 2 ) L .
Now, note that for all x , y ∈ R d , i ∈ [ n ] , and ϑ i , we have: [Equation] [Equation]
[Equation]
Therefore, we have:
[Equation]
[Equation]
(67) where we can conclude the lemma by taking expectation from the square of (65).
Finally, Lemma 1 implies the strong convexity and smoothness parameters for the surrogate loss (4).
Lemma 2 states a bounded variance for the stochastic gradients of the surrogate cost.
Therefore, the proof of Proposition 1 is an immediate consequence of Spiridonoff et al.
(2020b, Theorem 15) under these two lemmas.
D. Proof of Lemma 3 and Theorem
The proof of Lemma 3 is similar to the proof of Lemma 1 and Lemma 2.
First note that, according to Assumptions 2(i)-(iii), ˜ f
i ( ., D i,r ) is L -smooth, for all i ∈ [ n ] , and D i,r data batches drawn uniformly from p i .
Therefore we have:
[Equation]
(68) for all z ∈ R d , and data batches D i drawn from distribution p i .
This suggests that:
[Equation]
[Equation]
where due to Lemma 4, the following property holds:
[Equation]
70) for all i ∈ [ n ] and uniformly sample data batches D i with respect to p i .
Then, for each two points w , v ∈ R d , we have:
[Equation]
( u
[Equation]
u
[Equation]
(cid:13)(cid:13)(cid:13) (71)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
[Equation]
We now show that ˜ F ( u ) i ( ., ϑ i ) is ˆ L ( u ) -smooth, as follows:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Equation]
Moreover, we have:
[Equation]
[Table]
[Equation]
Thus
[Equation]
we can infer that:
[Equation]
therefore,
[Equation]
As a result of (75) and (90)
[Equation]
we can conclude the smoothness of ˜ F ( u ) i ( ., ϑ i ) as follows:
[Equation]
Moreover, note that for all x , y ∈ R d , i ∈ [ n ]
[Equation]
and ϑ i , we have:
[Equation]
[Equation]
Therefore, we have:
[Equation]
Equations (95), and (100) conclude the proof of Lemma 3.
Now that we have shown the smoothness of F ( u ) i ( · ) and bounded variance for the stochastic gradients, we are ready to state the convergence result for the smooth non-convex functions.
Before stating the proof of Theorem 1, note that under Γ w =1 , matrices ∆ ( t ) ∈ R ( n + m ′ )
× d in the linear system (7), turn into:
[Equation]
where θ ( t ) = θ is a constant stepsize.
Moreover, the following lemma helps us to provide a bound between z i ( t ) and X ( t ) ⊤ 1 n , for all i ∈ [ n ] , and t ≥ 0 .
Lemma 5.
Suppose Assumption 1 holds.
Consider the sequence of z
i
t ) , i ∈ [ n ] , generated by (7) .
Then the following property holds: for all t >
[Equation]
where δ := 1 1 − nγ 6 , λ :=(1 − nγ 6 ) 1 / (2 n Γ s ) , γ :=(1 /n ) n Γ s , and ∥ X ∥ 1 denotes the sum of 1 -norm of matrix X ’s rows, for any matrix X .
According to (7), and the fact that M ( t ) is a column stochastic matrix, we have: [Equation]
[Equation]
Due to Lemma 3, we can infer that F ( u ) is ˆ L ( u ) -smooth.
Therefore, the following property holds:
[Equation]
107) where using 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a − b ∥ 2 , and taking the expectation, we have:
[Equation]
where Ψ x ( t ) =
∥
Z − 11 ⊤ n X ( t ) ∥ 2 F .
Note that according to Lemma 5, we know that Ψ x ( t )
=
O ( θ 2 ) .
Thus, summing (108) from t = 0 to T − 1 and dividing by T leads to the proof.