ColRel: Collaborative Relaying for Federated Learning over Intermittently Connected Networks
Abstract
Intermittent connectivity of clients to the parameter server (PS) is a major bottleneck in federated edge learning.
It induces a large generalization gap, especially when the local data distribution amongst clients exhibits heterogeneity.
To overcome communication blockages between clients and the central PS, we have introduced the concept of collaborative relaying (ColRel), wherein the participating clients relay their neighbors’ local updates to the PS in order to boost the participation of clients with poor connectivity to the PS.
For every communication round, each client initially computes a local consensus of a subset of its neighboring clients’ updates and subsequently transmits to the PS, a weighted average of its own update and those of its neighbors’.
In this work, we view ColRel as a variance reduction technique that helps in improving the convergence rate for different optimization setups.
Consequently, our ColRel approach can be readily integrated as a black box with existing federated learning systems.
We provide analytical upper bounds on the resulting convergence rate, which we reduce by optimizing the weights subject to an unbiasedness condition for the global update.
Numerical evaluations on the CIFAR-10 dataset demonstrate that our ColRel-based approach achieves a higher test accuracy over Federated Averaging based benchmarks for learning over intermittently-connected networks.
Introduction
Federated learning (FL) algorithms aim to learn a shared model over data samples that are localized over distributed clients.
FL approaches aim to reduce communication overhead and improve clients’ privacy by letting each client train a local model on its local dataset and forwarding them periodically to a centralized parameter server (PS).
In practical FL scenarios, some clients are stragglers and cannot send their updates regularly.
There can be two types of stragglers: (i) computation stragglers , which cannot finish their computation within the deadline or (ii) communication stragglers , which cannot transmit their updates to the PS successfully due to communication limitations [7].
The latter may happen when clients suffer from intermittent connectivity to the PS due to temporary blockage of their communication channel [2, 9–11, 31, 47, 55].
In general, persistent stragglers deteriorate the convergence of FL algorithms as the computed local updates become stale and useless, and can even result in bias in the final model.
However, communication stragglers that are limited due to loss of direct communication opportunities to the PS are inherently different from computation stragglers, since they can be mitigated by relaying their updates via neighboring clients.
Communication quality at the edge is a key guiding design principle for FEderated Edge Learning (FEEL) frameworks [13].
Existing works in the FEEL paradigm have primarily focused on direct communication from the clients to the PS; aimed at improving the performance by resource allocation across clients [3, 4, 8, 13, 14, 28, 33, 36, 38, 39, 46, 49, 57].
However, these works ignore the possibility of cooperation among clients in the case of intermittent communication blockages.
Robustness to channel blockages is critical to the reliable operations of mmWave communication systems and robotic systems, where mobile robots explore remote parts of areas of interest and thus can be disconnected from the PS [11, 31, 47, 55].
PS pppppp 6 pppp
This work studies the deterioration in performance of existing FL algorithms in the presence of intermittent connections to the PS (Fig. 1).
We show that communication stragglers, which suffer from such random unreliable network connectivity, introduce a topology induced variance (TIV) in the local updates of clients.
To mitigate this, we propose a new FEEL paradigm, which we name
Collaborative Relaying ( ColRel ), where client cooperation is utilized to improve connectivity to the
PS.
In ColRel, clients share their local updates with one another so that each client can send to the PS a weighted average of its own update and those of its neighbors.
The PS receives updates from clients with intermittently absent uplink connections, which would otherwise be discarded.
ColRel optimizes the averaging weights in order to ensure that the updates received at the PS (i) achieve weak unbiasedness that avoids objective inconsistency, and (ii) minimizes the TIV and subsequently, the convergence time of the learning algorithm under intermittent connectivity.
1.1 Significance and Related Works
Fully Decentralized and Hierarchical FL :
The conventional FL framework [30] is orchestrated by a centralized entity called PS, which helps participating clients reach a consensus on the model parameters by aggregating their locally trained models.
Decentralized learning or peer-to-peer FL has been introduced as an alternative, in which the PS is removed to mitigate a potential communication bottleneck and a single point of failure.
The aggregation strategy at each client is determined according to the network topology, that is the connection pattern between the clients, and often a fixed topology is considered [17, 18, 21, 26, 37, 44, 45, 51, 54, 56].
An alternative approach to both centralized and decentralized schemes is the hierarchical FL (HFL) framework [1, 6, 27, 29], wherein clients are divided into clusters and a PS is assigned to each cluster to perform local aggregation, while the aggregated models at the clusters are later aggregated at the main PS in a subsequent step to obtain the global model.
The idea of hierarchical collaborative learning can be redesigned to combine hierarchical and decentralized learning concepts, which is referred to as semi-decentralized FL .
One of the major challenges in FL that is not considered in the aforementioned works on semi-decentralized FL is the partial client connectivity [12, 48].
The connectivity of the clients is a particularly significant challenge in FEEL.
Due to their different physical environments and distances to the PS, clients may have different connectivity to each other and the PS.
Customized client selection mechanisms seek a balance between the participation of the clients and the latency for the model aggregation in order to speed up the learning process [4, 15, 28, 35, 36, 40, 46, 49, 50].
In this work, we adopt a different approach to the connectivity problem, and instead of designing a client selection mechanism, or optimizing resource allocation to balance client participation, we introduce a relaying mechanism that benefits from the local connectivity between neighboring clients, and ensures that in case of poor connectivity, their local information is conveyed to the PS with the help of their neighbors.
Coded Computation for Straggler Mitigation : Coded computing is a well studied solution for stragglers in distributed learning [23, 24, 32, 34, 43], where data is allocated strategically to clients to create redundant computations that can be exploited by the PS using additional information about the stragglers’ identities.
Nonetheless, the coded computing paradigm as it is cannot be incorporated into FL systems, which champion the privacy of clients’ data as one of their major goals.
Our ColRel approach does not require redundancies in clients’ data since clients share their updates but not their local data , and thus preserves privacy to the same extent as conventional FL algorithms.
Another notable difference between ColRel and coded computing includes relaying constraints.
Coded computation schemes spread the data among clients without constraints on the allowed combinations.
In contrast, in our problem setup, clients cannot choose their neighbors as they depend on the channel characteristics, and thus coded computing techniques cannot be readily adopted.
1.2 Our Contributions
We start with the problem of distributed mean estimation (DME) in the presence of intermittent connectivity.
Our proposed algorithm, ColRel ensures that the mean squared error (MSE) of this estimation is minimized by appropriately relaying the clients’ information to the PS.
We then study
ColRel to mitigate intermittent connectivity for several federated optimization setups.
In particular, for an iteration horizon of T , we show that: (i) For general convex and non-smooth objectives, a subgradient-based ColRel method converges at the rate of O (cid:0) 1 T (cid:1) , with constants proportional to the
TIV ( σ 2tv ).
(ii) For smooth functions that satisfy the Polyak-Lojasiewicz condition, gradient descent based ColRel with a constant step-size converges linearly to a neighborhood of the optimal solution, whose size is proportional to σ 2tv .
(iii) For smooth and non-convex objectives, gradient descent based
ColRel converges at a rate of O (cid:0) 1 T (cid:1) to the neighborhood of a stationary point (with size proportional to σ 2tv ).
(iv) Numerical simulations on FL benchmarks validate our theoretical analyses.
DME over Intermittently Connected Networks
The DME problem is a fundamental primitive for FL algorithms, where at every iteration, the PS estimates the mean of the local updates of a subset of clients [16, 19, 20, 41, 42].
To this end, analyzing DME is a useful and insightful abstraction.
We study DME over intermittently connected networks so as to generalize the conclusion to federated optimization algorithms in subsequent sections.
We consider a DME model comprising of n clients, each with a vector x i ∈ R d , for i ∈ [ n ] that satisfy ∥ x i ∥ ≤ R for some known R > 0 , and the goal of the PS is to estimate their mean, i.e., x ≜ 1 n (cid:80) ni
=1 x i .
The performance of any estimate (cid:98) x at the PS is measured by its MSE, E ≜ E ∥ (cid:98) x − x ∥ 22 .
2.1 Communication Model
Communication between clients and the PS : We model the intermittent connectivity of any client i ∈ [ n ] to the PS by a Bernoulli random variable, τ i ∼ Ber( p i ) , where τ i = 1 implies a communication opportunity between client i and the PS, while τ i = 0 means that the connection is blocked/dropped [12].
We assume that the connectivity is independent across the clients, and the downlink from the PS to the clients is permanently connected.
2 We denote p ≡ ( p 1 , . . .
, p n ) .
Communication between clients : The intermittent connectivity of a transmission from client i to client j is captured by another Bernoulli random variable, τ ij ∼ Ber( p ij ) , where p ii = 1 for every i ∈ [ n ] .
Additionally, if client j can never transmit information successfully to client i , we set p ij = 0 .
For simplicity of exposition, we assume that τ ij and τ ml are statistically independent for every i, j, l, m ∈ [ n ] such that ( i,
j ) ̸
= ( l, m ) and ( j, i )
̸
= ( l, m ) .
Furthermore, τ ij , and τ l are statistically independent for every i, j, l ∈ [ n ] .
We use the notation E { i,j } = E
[ τ ij τ ji ] for every i, j ∈ [ n ] to denote the correlation due to channel reciprocity.
Finally, we assume that
E { i,j } ≥ p ij p ji , that is, P ( τ ij = 1 | τ ji = 1) ≥ P ( τ ij = 1) , for every i, j ∈ [ n ] .
We use the notation
P ≡ ( p ij ) i,j ∈ [ n ] ∈ [0 , 1] n × n .
Remark 1.
We assume that the connectivity probabilities p
i , i ∈ [ n ] , are known.
In practice, they can be modeled [2] or estimated in a pre-training phase.
On the other hand, we do not assume that the instantaneous connectivity information { τ i , τ ij } for i, j ∈ [ n ] , is available to any of the clients.
Our prior work [52] considered communication links between pairs of clients (if present) to be perfect, i.e., for any i ̸
= j , p ij = 0 or 1 , with an extended version [53] that allows for both client-PS and client-to-client communications to be intermittent.
This workshop paper generalizes the collaborative approach in these works for the fundamental problem of distributed mean estimation, consequently demonstrating its potential efficacy as a black-box tool that can be integrated into existing FL systems.
2.2 ColRel for DME
Since communicating clients can intermittently send their vectors to one another, each client can send the PS a weighted average of its own vector and those of its neighbors.
Consequently, the
PS can receive the vectors of clients with failing uplink connections.
Let { α ij } i,j ∈ [ n ] denote the collaboration weights.
Each client i , transmits to the PS the weighted average (cid:101) x
i = (cid:80) j ∈ [ n ] α ij τ ji x j .
This transmission is received at the PS with a probability p i .
The PS computes the following estimate (cid:98) x of the mean via ColRel:
[Equation]
We denote by A ≡ ( α ij ) i,j ∈ [ n ] the matrix of collaboration weights, where α ij ≥ 0 for all i, j ∈ [ n ] .
TIV: Error Analysis and Optimizing Collaboration
Due to the stochasticity of intermittent connections in the network, the estimate (cid:98) x is a random variable, and we consider the MSE of (cid:98) x with respect to the true mean x .
When R = 1 , i.e., ∥ x i ∥ 2 ≤ 1 for all i ∈ [ n ] , we refer to the MSE as the TIV , σ 2tv .
Our primary goal is to obtain an unbiased estimate of x at the PS.
Under this unbiasedness condition, we derive a worst-case upper bound for σ 2tv in
Theorem 3.2.
In general, the TIV is a function of connection probabilities and collaboration weights, i.e., σ 2tv ≡ σ 2tv ( p , P , A ) .
Consequently, we choose the weights A ≡ { α ij } i,j ∈ [ n ] so as to minimize σ 2tv .
Moreover, we consider α ij ≥ 0 for all i, j ∈ [ n ] .
In §4 and §5, it is shown how the TIV ( σ 2tv ) affects the convergence performance of federated optimization algorithms.
Unbiasedness : Recall that α ji is the collaboration weight client j assigns to the vector it receives from client i .
Client i , and each client j that receives the transmission from client i successfully, i.e. τ ij = 1 , try to send to the PS α ji x i on behalf of client i .
Consequently, the accumulated contribution of client i at the PS is given by, (cid:80) j ∈ [ n ] τ ij α ji x i .
The following lemma presents a sufficient condition on the weights { α ij } i,j ∈ [ n ] that ensures unbiasedness.
Lemma 3.1 ( Sufficient condition for unbiasedness ) .
Let { α ij } be such that for every i ∈ [ n ] , (cid:88) j ∈
[Equation]
n
[Equation]
p
[Equation]
p
[Equation]
Then, E (cid:104)(cid:80) j ∈ [ n ] τ j τ ij α ji x i (cid:12)(cid:12)(cid:12) x i (cid:105) =
x i , i.e., the contribution of client i at the PS is unbiased. As we show next in Theorem 3.2, the effect of the network topology on the TIV is captured by the expression S( p , P , A ) , which is defined as,
[Equation]
Theorem 3.2.
For a given p , P and A such that (2) holds, the MSE with ColRel ( E ColRel ) satisfies,
E ColRel ( p , P , A ) ≤ R 2 n 2 · S( p , P , A ) =
R 2 σ 2tv ( p , P , A ) , where σ 2tv ( p , P , A )
≜ S( p , P , A ) n.
Remark 2.
In the absence of any collaboration, we set p ij = 0 for all
i ̸
= j , which yields suboptimal σ 2tv compared to the weights obtained with optimized collaboration as shown next in §3.1.
3.1 Optimizing Collaboration: Minimizing the TIV
As a consequence of Theorem 3.2, we optimize the weight matrix
A so as to minimize σ 2tv ( p , P , A ) subject to the unbiasedness condition (2).
In other words, we solve the optimization problem,
[Equation]
Interestingly, this is the same optimization problem as in [53, Eq.
Due to the term (cid:80) i,l ∈ [ n ] p i p l (E { i,l } − p il p li )
α il α li in (3), problem (4) is not necessarily convex.
To this end, as in [53], we minimize a convex relaxation S( p , P , A ) of S( p , P , A ) instead, where,
S( p , P , A
[Equation]
≜
[Equation]
To find the optimal collaboration weights A opt , we first minimize the convex upper bound S using
Gauss-Seidel method.
Subsequently, we fine tune this solution by using it as a warm-start initialization for converging to a stationary point of (4).
For the sake of completeness, a detailed description of the resulting iterative water-filling type weight optimization algorithm is provided in App.
C.
Federated Optimization over Intermittently Connected Networks
Here, we use the insights from the analysis of DME in §2 and §3 to derive convergence guarantees for federated optimization problems.
Our goal is to solve, x ∗ ≡ arg min x ∈ R d [Equation] x
[Equation] = [Equation] where x ∈ R
d are the parameters of the model, n is the number of clients, f i ( x ) is the local loss of model x on the data stored on client i .
We consider various standard assumptions of the objective functions
f i and study the convergence guarantees when the gradients computed at the clients are shared with the PS over unreliable networks.
We conclude that as a consequence of intermittent connections, the convergence rates are affected by an additional error which is proportional to the
TIV σ 2tv .
The algorithms and their analyses are deferred to the appendices.
4.1 Subgradient Method for Non-Smooth and General Convex Objectives
When f i , i ∈ [ n ] are convex but not necessarily smooth, the iterates of a subgradient method with
ColRel in the presence of intermittent connectivity is given by,
[Equation]
Here, g j ( · ) ∈ ∂f j ( · ) denotes a subgradient of f j , and { τ ti , τ tij } are Bernoulli random variables that model the intermittent connectivity at iteration t .
The convergence of iterations in (6) is stated below.
Theorem 4.1.
Suppose that for every i ∈ [ n ] , the local loss function f
i is G i - Lipschitz continuous with G max ≜ max i ∈ [ n ] G i .
Additionally, suppose f ∗ ≜ inf x ∈ R d f ( x ) >
−∞ with f ( x ∗ ) =
f ∗ , and there exists R > 0 such that ∥ x 0 − x ∗ ∥ 22 ≤ R 2 .
Then, given p , P , and weights A satisfying (2) , for a learning-rate sequence { γ t } , we have,[-
 [Equation]-]
[Equation]
Theorem 4.1 states that for a constant learning rate choice of γ t = γ for all t = 0 , 1 , . .
. , the algorithm (6) converges to a neighborhood of the solution at a rate of O ( 1 T ) ; with the size of the neighborhood being proportional to the TIV, σ 2tv . [Figure] [Figure]
4.2 Gradient Descent for Smooth and Polyak-Lojasiewicz (PL) Objectives
A function f is said to be µ -PL if f ( x ) − f ( x ∗ ) ≤ 12 µ ∥∇ f ( x ) ∥ 22 holds true for all x ∈ R d , where x ∗ ≜ arg min x ∈ R d f ( x ) .
Gradient descent with ColRel follows the same iteration as (6) with the gradient ∇ f j ( x t ) instead of a subgradient g j ( x t ) .
Theorem 4.2 below states that the gradient descent iterates converge linearly to a neighborhood (with size proportional to σ 2tv ) of the solution.
Theorem 4.2.
Suppose f is µ -PL and L -smooth, and for every i ∈ [ n ] , f i is G i -Lipschitz continuous with G max = max i ∈ [ n ] G i .
For a given p , P , weights A satisfying (2) , and a constant γ ∈ (cid:0) 0 , 1L (cid:3) , let x t +1 ← x t − γn (cid:80)
i ∈
[ n ] τ ti (cid:80) j ∈ [ n ] α ij τ tij ∇ f j ( x t ) .
Then, we have,
[Equation]
4.3 Gradient Descent for Smooth and Non-Convex Objectives
When f i ’s are smooth and non-convex, gradient descent with ColRel once again converges to a neighborhood of a stationary point at a rate of O ( 1 T ) .
This is formalized below.
Theorem 4.3.
Suppose f ∗ ≜ inf x ∈ R d f ( x ) >
−∞ , f i , i ∈ [ n ] is L
i
-smooth
and G i -Lipschitz continuous with G max = max i ∈
[ n ] G i , and there exists D 0 >
such that E f ( x 0 ) − f ∗ ≤ D 0 .
Then, given p , P , and weights A satisfying (2) , for γ ∈ (cid:0) 0 , 1L (cid:3) , we have,
[Equation]
Remark 3.
To specifically focus on the stochasticity due to random connections, Thms. 4.1, 4.2 and 4.3 state the guarantees with deterministic (sub) gradients at clients.
The results can be extended for stochastic (sub) gradients as we show in App.
G. We can further improve the convergence rates in these settings by using ColRel together with schemes such as global momentum, periodic averaging, variance reduction, etc.
The key takeaway here is that intermittent connectivity over unreliable networks introduces a TIV that can be treated similarly to sources of stochasticity such as stochastic gradient oracle.
Furthermore, the TIV can be minimized with optimized ColRel between clients.
Numerical Simulations
We now provide numerical simulations by training a ResNet-20 model for image classification on the
CIFAR-10 dataset [22].
Detailed results and discussions are provided in App. H.
Optimizing collaboration with heterogeneous client connectivity :
In Fig.
2a, we consider a decentralized ring-topology amongst the clients where p ij = [-
-]1 if j = ( i − 1 , i or, i + 1)mod n , and p ij = 0 otherwise.
The local datasets of clients are independently and identically distributed (i.i.d.), but they have different connectivity to the PS with p =
We observe that with { α ij } optimized according to §3.1, ColRel achieves the same performance as with perfect connectivity.
We also compare the results with Federated Averaging (FedAvg), where
Blind refers to when the PS is unaware of the identity of transmitting clients such as in over-the-air aggregation schemes [3, 4, 33].
Non-Blind refers to when the PS knows exactly how many clients successfully transmitted at each iteration.
Intermittent client-client connectivity :
In Fig.
2b, a single client has p 1 = p gc = 0 .
9 and p i = 0 .otherwise.
The clients can collaborate intermittently by relaying their update to the good client over an Erd˝os-Rényi topology with p ij = p c for i ̸
= j .
This resembles a setup where clients are clustered together and only one of them has good connectivity to the PS (for e.g., physical proximity or power constraint).
Even with p c = 0 .
5 , the performance is comparable to FedAvg with perfect connectivity.
Non-IID local data distribution and global momentum :
In Fig.
2c, the CIFAR-10 dataset is sorted and partitioned before distributing it across the clients.
Each client has samples from at most 3 classes, in order to emulate a more realistic FL scenario with non-i.i.d. data.
There is still just a single client with good PS connectivity as in Fig. 2b.
Global momentum is employed at the PS to update the global model.
FedAvg (non-blind) fails to converge in this setting, because in the absence of collaboration, clients with important training samples that are critical for training a good model with high accuracy, have a low probability of successful transmission and thus are rarely able to convey their updates to the PS.
Consequently, the resulting test accuracy of the global model is ∼ 10% , as good as a random classifier for 10 classes.
On the other hand, ColRel ensures that the information from these critical datapoints are conveyed to the PS even when the data owner does not have connectivity to the PS.
Conclusions
FL has been proposed as a distributed learning strategy to train a common model with localized data.
However, resource-constrained edge devices often suffer from intermittent connectivity to the PS, and are communication stragglers.
On the other hand, thanks to their wireless connectivity, they can communicate locally with their neighbors even when the channel to the PS is blocked.
Hence, to mitigate communication stragglers in FL, we proposed a collaborative relaying strategy (ColRel), which exploits the connections between clients to relay potentially missing model updates to the
PS.
Through the abstraction of DME, we showed that intermittent connectivity introduces a TIV, which we minimized by optimizing the collaboration weights.
We then used these optimized weights to improve the convergence rate of FL algorithms.
ColRel can be implemented even when the PS is blind to the identities of clients that successfully communicate with it at each round.
Numerical results showed the improvement in the training accuracy and the convergence rate that our approach provides in various settings.
Acknowledgments and Disclosure of Funding
R. Saha, M. Yemini and A. J. Goldsmith are partially supported by the AFOSR award # and a Huawei Intelligent Spectrum grant. E. Ozfatura and D. Gündüz received funding from the European Research Council (ERC) through Starting Grant BEACON (no. 677854) and the UK EPSRC (grant no.
EP/T023600/1) under the CHIST-ERA program. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] (b) Did you include complete proofs of all theoretical results? [Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)? [Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Contents 1 Introduction 1.
Significance and Related Works . .
Our Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 DME over Intermittently Connected Networks 2. Communication Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ColRel for DME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 TIV: Error Analysis and Optimizing Collaboration 3.
Optimizing Collaboration: Minimizing the TIV . . . . . . . . . . . . . . . . . . .
4 Federated Optimization over Intermittently Connected Networks 4.
Subgradient Method for Non-Smooth and General Convex Objectives . . . . . . .
Gradient Descent for Smooth and Polyak-Lojasiewicz (PL) Objectives . . . . . . .
Gradient Descent for Smooth and Non-Convex Objectives . . . . . . . . . . . . .
5 Numerical Simulations 6 Conclusions
A Proof of Lemma 3.1: Sufficient Condition for Unbiasedness
B Proof of Theorem 3.2: MSE for DME with Client Collaboration under Intermittent
Connectivity
C Optimizing the Collaboration Weights: Minimizing σ 2tv ( p , P , A )
{+C.
+}
Deriving the Convex Relaxation for Minimizing S( p , P , A ) . . . . . . . . . . . . C.2 Algorithm to optimize the collaboration weights . C
Iterative Gauss-Seidel update using Lagrange multipliers
D Proof of Theorem 4.1: Non-Smooth and General Convex Objectives E Proof of Theorem 4.2: Smooth and Polyak-Lojasiewicz (PL) Objectives F Proof of Theorem 4.3: Smooth and Non-Convex Objectives G Extensions to Stochastic (Sub) Gradient Oracle
H Numerical Simulations: Extended H.1 Distributed Mean Estimation (DME H.2 Federated Learning Simulations A Proof of Lemma 3.1: Sufficient Condition for Unbiasedness
Since τ j and τ ij are statistically independent of x i and each other, for all i, j ∈ [ n ] , we have that, [Equation]
Substituting (2) concludes the proof.
Note that the standard model of FL with random client sampling and no connectivity among clients is captured by substituting p ij = 0 , p i = p , α ii = 1 , and α ij = 0 for all i, j ∈ [ n ] such that j ̸ = i .
B Proof of Theorem 3.2: MSE for DME with Client Collaboration under Intermittent Connectivity
The proof of Thm. 3.2 is similar to that of [53, Lemma 6] and utilizes the fact that ∥ x i ∥ 2 ≤ R .
Nonetheless, we reproduce it here for the sake of completeness.
The MSE for DME with ColRel using a weight matrix A is given by, [Equation]
[Equation] [Equation] n ] [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88)
[Equation]
[ n ] (cid:88)
[Equation]
[ n ]
[Equation]
x
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
The last equality follows since the random variables τ
i ∈
[ n ] and τ ij , i, j ∈ [ n ] are assumed to be statistically independent of the vectors x i .
Simplifying the first term above gives us,
E 
[Equation]
[ n ] E [ τ j τ ij α ji ] +(i) = (cid:88)
j ∈ [ n ] p j p ij α 2 ji + (cid:88) j 1 ,j 2 ∈ [ n ]: j 1 ̸ = jp j 1 p j 2 p ij 1 p ij 2 α j 1 i α j 2 i − 2 (cid:88) j ∈ [ n ] p j p ij α ji +(ii) =
(cid:88) j ∈ [ n ] p j p ij α 2 ji +   (cid:88) j ∈ [ n ] p j p ij α ji   − (cid:88) j ∈ [ n ] p 2 j p 2 ij α 2 ji −
[Equation]
Here, (i) follows from the statistical independence of τ i , i ∈ [ n ] and τ ij , i, j ∈ [ n ] , while (ii) and (iii) follow from the sufficient requirement for unbiasedness (2) [Equation]
To evaluate the second term [Equation] note that for every l ̸
= i ,
[Equation]
Here [Equation] (i) follows from the statistical independence of τ ’s as before.
The equality (ii) follows from the sufficient conditions for unbiasedness which lead to the following chain of equalities:
[Equation]
From (11) and (12), the MSE with ColRel in (10) simplifies to,
E ColRel ( p , P , A ) = 1 n 2 (cid:88)
[Equation]
[Equation]
where the last inequality follows from E { i,i } − p ii p ii = 1 − 1 · [Equation] = 0 .
Note that we initially assumed E { i,l } ≥ p il p li , and constrained our weights α il , α li to be non-negative.
Furthermore, since ∥ x i ∥ 2 ≤ R , ∀ i ∈ [ n ] , an application of Cauchy-Schwarz inequality to upper bound the inner products yields, x ⊤ i x l ≤ ∥ x i ∥ {+
+} ∥ x l ∥ 2 ≤ R 2 .
This yields,
[Equation]
where, S( p , P , A ) is defined as in (3).
This completes the proof.
C Optimizing the Collaboration Weights: Minimizing σ 2tv ( p , P , A )
In this section, we show how to choose the optimal weights for collaboration.
The results of this section follow from [53].
We omit the proofs and refer the reader to [53, Lemma 2] and [53, Lemma 7].
As discussed already in §3 and §4, in order to reduce the TIV, we want to solve the following optimization problem (4) (reproduced below):
[Equation]
Here, S( p , P , A ) is given by,
[Equation]
C.1 Deriving the Convex Relaxation for Minimizing S( p , P , A )
Due to the last term (cid:80) i,l ∈ [ n ] p i p l (E { i,l } − p il p li )
α il α li , the function S( p , P , A ) is not necessarily convex with respect to A .
To this end, we consider the convex upper bound S( p , P , A ) where,
S( p , P , A ) ≜ (cid:88)
[Equation]
The following lemma formally proves the validity of this convex relaxation.
Lemma C.1.
[53, Lemma 2]
For every A such that α ij ≥ 0 , ∀ i, j ∈ [ n ] , S( p , P , A ) ≤ S( p , P , A ) .
Furthermore, S( p , P , A ) is convex with respect A for p ∈ [0 , 1] n .
The proof of Lemma C.1 utilizes the result from the following auxiliary lemma.
Lemma C.2.
[53, Lemma 7] Let y , c ∈ R ˜ d where ˜ d ∈ N + .
Denote h c ( y ) =
(cid:16)(cid:80)
˜ di
=1 c i y i (cid:17) 2 , then h c ( y ) is convex.
Lemmas C.1 implies that the following optimization problem is a convex relaxation of (4):
[Equation]
C.2 Algorithm to optimize the collaboration weights
Let A i denote the i th column of A , that is, A i ≜ ( A 1 i , . . .
, A ni ) ⊤ .
Since the domain of the problems (4) and (19) is separable with respect to A i , we can use the Gauss-Seidel method to iteratively solve (19) and converge to an optimal solution [5, Proposition 2.7.1] for the upper bound (19).
We can then converge to a stationary point of (4) in the vicinity of this solution by employing the Gauss-Seidel method once again.
Remark 1.
We note that when p ij ∈ { 0 , 1 } , ∀ i, j ∈ [ n ] , i.e., the client-client connections are always present or always absent, the problem (4) is convex.
Additionally, in this case the problems (4) and (19) coincide since E { i,l } − p il p li = [-
-]0 for every i, j ∈ [ n ] .
Optimizing the convex relaxation (19): Let A ( ℓ ) i denote the approximated value for A i in the ℓ th iteration of the Gauss-Seidel method of (19).
We choose the initial solution to be
[Equation]
We can improve our solution iteratively by the Gauss-Seidel method until convergence to an optimal point of (19).
That is, at every iteration ℓ we compute A ( ℓ ) as follows,
[Equation]
otherwise
[Equation]
where,
[Equation]
Using Lagrange multipliers [Equation] we show in App.
C.3 that the optimal value for A ( ℓ ) i is:
[Equation]
where ( a ) + ≜ max { a, 0 } and λ i is set such that (cid:80) j ∈ [ n ] p j p ij (cid:98) α ( ℓ ) ji ( λ i ) =
We can find λ
i using the bisection method over the interval: 
[Equation]
Fine tuning (4) : Let A ( ℓ ) i denote the approximated value for A i in the ℓ th iteration of the GaussSeidel method of (4), and assume a given initialization A (0) i .
We can improve our solution iteratively by the Gauss-Seidel method until convergence to a stationary point of (4).
At every iteration ℓ we compute A ( ℓ ) i as follows,
[Equation]
where 1 {·} denotes the indicator function, and
[Equation]
Using Lagrange multipliers, we show in Appendix C.3 that the optimal value for (cid:98) A ( ℓ ) i is as follows:
[Equation]
where λ i is set such that (cid:80) j ∈ [ n ] p j p ij (cid:98) α ( ℓ ) ji ( λ i ) = 1 .
We can find λ
i using the bisection method over the interval:  
[Equation]
[Equation] We summarize the centralized optimization procedure for A in Algorithm 1.
When the connectivity between two communicating clients is reliable, i.e., two clients are either connected with high probability or disconnected, Algorithm 1 can be implemented in a distributed fashion by the clients.
In this case each client does not need to fully know p , P and A , but only the weights and transmission probabilities of all its direct neighbors and its second degree neighbors (i.e., neighbors of its neighbors).
Such a distributed algorithm can be used to optimize the weights when the PS is blind to the identities of the transmitting clients at all times and cannot use a training period for learning the transmission probabilities.
Finally, we note the weights resulting from Algorithm 1 can be used as long as the probabilities p and P are fixed, and are not needed to be calculated in every communication round.
Computation complexity
The overall computational complexity of Algorithm 1 is O
( I · ( n 2 + K )) , where K is the number of iterations used in the bisection method for optimizing λ i .
C.3 Iterative Gauss-Seidel update using Lagrange multipliers
First, observe that we can set α ji = 0 for every j such that p ij p j = 0 .
Additionally, if max k ∈ [ n ] { p k p ik } = 1 , then we can set α ji = 1 { p ij p j =1 } · ( (cid:80) k ∈ [ n ] 1 { p k p ik =1 } ) − 1 .
Therefore, hereafter we assume that i is such that max k ∈ [ n ] { p k p ik } < 1 and that j is such that p ij p j ∈ (0 , 1) .
We proceed to solve each of the problems (21) and (24), respectively.
Solving the Convex Optimization Problem (21): The Lagrangian of (21) is,
Input: A set of clients [ n ] , the functions S( p , P , A ) and S( p , P , A ) , vector of connectivity probabilities p , matrix of inter-client connectivity probabilities P , maximal number of iteration I .
Output:
A matrix
A ( L ) that approximately minimizes (4) 1 Initialize (convex relaxation) :
A (0) ji = 1 (cid:16)(cid:80) j ∈ [ n ] 1 { pj> 0 ,pij> 0 } (cid:17) p j p ij · 1 { p j > 0 ,p ij > 0 } , and ℓ = 0 . 2 while ℓ ≤ I − 1 do ℓ ← ℓ + i ← ℓ mod n + n · 1 { ℓ mod n =0 }
Compute (cid:98) A ( ℓ ) i according to (22)
Set A ( ℓ ) k according to (20) for every k ∈ [ n ] 7 end 8 Warm-initialize (fine tuning) :
A (0) =
A ( L ) and ℓ = 0 .
9 while ℓ ≤ I − 1 do ℓ ← ℓ + i ← ℓ mod n + n · 1 { ℓ mod n =0 }
Compute (cid:98) A ( ℓ ) i according to (25)
Set A ( ℓ ) according to (23) for every k ∈ [ n ] 14 end
L ( A ( ℓ ) i , λ i ) =
(cid:88)
j ∈ [ n ] p j p ij (1 − p
j p ij )
α 2 ji +(cid:88) l ∈ [ n ]: l ̸
= i (cid:88) j ∈ [ n ] p j (1 − p j ) p ij p lj α ji α ( ℓ − 1) jl + (cid:88)
j ∈ [ n ] p i p
j (E { i,j } − p ij p ji )
α 2 ji − λ i   (cid:88)
j ∈ [ n ] p j p ij α ji −  − µ ji ( α ji ) .
Additionally,
[Equation]
A
[Equation]
[Equation]
i
[Equation]
It follows from the Karush–Kuhn–Tucker conditions that
[Equation]
(cid:33) + , and λ i ≥ 0 is set such that (cid:80) j ∈ [ n ] p j p ij α ji
( λ i )
= 1 .
Solving the Convex Optimization Problem (24): The Lagrangian of (24) is
L ( A ( ℓ ) i , λ i ) = (cid:88) [Equation] [ n
[Equation]
(cid:88) l ∈ [ n ]: l ̸
= i (cid:88)
j
[Equation]
[ n ] [Equation] +
[Equation]
p
[Equation] − [Equation] j
[Equation]
p
[Equation]
ij α ji −  − µ ji ( α ji ) .
Additionally, ∂ L
[Equation]
A
[Equation]
i ) ∂α
[Equation]
=
[Equation]
p j p
[Equation]
[Equation]
p j p
[Equation]
[Equation] p j [Equation] 1 − p j ) p ij (cid:88) l ∈ [ n ]: l ̸
= i p lj α ( ℓ − 1) jl + 2 p i p j [Equation]
E { i [Equation] j } − p ij p ji ) α ( ℓ − 1) ij − λ i p ij p j + µ ji , ∂ L ( A ( ℓ ) i , λ i )
∂λ i = 1 − (cid:88)
j ∈ [ n ] p j p ij α ji , ∂ L ( A ( ℓ ) i , λ i ) ∂µ ji = − α ji . It follows from the Karush–Kuhn–Tucker conditions that α ji ( λ i ) = [Equation] p
[Equation]
) (cid:80)
[Equation]
[ n ]
[Equation]
̸
[Equation] − 2 p i
[Equation]
E
[Equation]
i
[Equation]
j } /p [Equation] p [Equation]
[Equation]
ℓ
[Equation]
1) ij + λ i 2 (
[Equation] p j p [Equation] ) (cid:33) + , and λ i ≥ 0 is set such that (cid:80) j ∈ [ n ] p j p ij α ji
( λ i )
= 1 .
D Proof of Theorem 4.1: Non-Smooth and General Convex Objectives
In this section, we discuss and analyze in detail, a subgradient method with collaborative relaying for federated optimization over intermittently connected networks.
The convergence rate of this analysis is derived following the analysis of DME and we use the optimized collaboration weights, i.e., A
=
A opt in order to minimize the TIV, σ 2tv ( p , P , A ) .
The algorithm pseudocode is presented in Alg. 2.
Input: A set of clients [ n ] with local losses f i , a client connectivity graph G , optimized weight matrix
A
=
A opt , maximal number of iteration T , initial point x sequence {
To derive the convergence rate of Alg. 2 for non-smooth and general convex objective functions, we make the following assumptions (as stated in statement of Theorem 4.3).
Assumption D.1.
For i ∈ [ n ] , the local loss function f i is G
i -Lipschitz continuous.
That is, for any x , y ∈ R d , | f i ( x ) − f i ( y ) | ≤ G i ∥ x − y ∥ 2 .
G i -Lipschitz continuity implies for any x ∈ R d , the subgradient g i ( x ) ∈
∂f i ( x ) satisfies ∥
g i ( x ) ∥ 2 ≤
G i .
Here, ∂f i ( x ) denotes the subdifferential of f i at x .
Assumption D.2.
The objective function is lower bounded and the infimum is attainable, i.e.,
[Equation]
[Equation]
Assumption D.3.
The initial distance to optimality can be upper bounded by some known R > 0 , i.e.,
[Equation]
In general, these assumptions can be relaxed.
Here, they are merely made here to simplify the proofs.
Denote, G max ≜ max i ∈ [ n ] G i and g ( x t ) ≜ 1 n (cid:80) i ∈ [ n ] g i ( x t ) .
We have that,
[Equation]
In this chain of inequalities
[Equation]
the expectation is over the stochasticity of the intermittent links at iteration t .
Here, (i) follows since we ensure that the stochastic subgradient computed at the parameter server despite the intermittent connectivity is unbiased
[Equation] (ii) holds true for the same reason [Equation] and (iii) follows from Lipschitz continuity of f i for all i ∈ [ n ] and the result on MSE for DME with collaborative relaying in the presence of intermittent connectivity.
Now taking expectations over all sources of stochasticity, we have,
[Equation]
Summing all these inequalities for t = 0 , . .
, T − 1 , we get,
[Equation]
[Equation] Since E (cid:2) ∥ x T − x ∗ ∥ 22 (cid:3) ≥ 0 , this gives us,
[Equation]
So, the algorithm converges to the optimal solution as long as the learning rate sequence { γ t } t ≥ 0 is square-summable and not summable, i.e. γ t ≥ 0 ,
[Equation]
(cid:88) t = γ 2 t < ∞ , and
[Equation]
(cid:88) t = γ
[Equation]
= ∞
[Equation]
For a constant step-size sequence γ t = γ for all t , we have,
[Equation]
So the algorithm converges to a neighborhood of the optimal solution at a convergence rate of O (cid:0) 1 T (cid:1) and the size of the neighborhood depends on S( p , P , A ) .
The results of this section can also be extended to setting with a stochastic subgradient oracle as shown in App.
G.
E Proof of Theorem 4.2: Smooth and Polyak-Lojasiewicz (PL) Objectives
In this section, we derive the linear convergence rate of gradient descent with collaborative relaying ( ColRel-GD ) for smooth and PL objective functions.
The algorithm is the same as Alg.
2 with each client evaluating the gradient ∇ f i ( x t ) at iteration t .
We formally state the assumptions on the objective functions below.
Assumption E.1.
( Smoothness and Lower boundedness ).
For every i ∈ [ n ] , the function f i is L ismooth, i.e., ∥∇ f i ( x ) − ∇ f i ( y )
∥
≤ L i ∥ x − y ∥ 2 , for all x , y ∈ R d and f ∗ ≜
inf x ∈ R d f i ( x ) > −∞ .
As a consequence, f = 1 n (cid:80) i ∈ [ n ] f
i is L -smooth with L ≜ 1 n (cid:80) i ∈
[ n ] L i .
Assumption E.2.
( Lipschitz continuity ).
For i ∈ [ n ] , the local loss function f i is G i -Lipschitz continuous, i.e., for any x , y ∈ R d , | f i ( x ) − f i ( y ) | ≤ G i ∥ x − y ∥ 2 .
Assumption E.3.
( Polyak-Lojasiewicz (PL) condition ).
For some µ > 0 , the global objective f µ -PL, i.e., for all x ∈ R d , where x ∗ = arg min x ∈ R d f ( x ) , we have, f ( x ) − f ( x ∗ ) ≤ 12 µ ∥∇ f ( x ) ∥ 22 .
We now state a descent lemma (without proof) from [25] that we use for our analysis.
Lemma E.4 (Lemma 2, [25]) .
Suppose that function f is L -smooth and let x t +1 ← x t − γ g t .
Then for any g t ∈ R d and γ > 0 , we have,
[Equation]
Conditioned on x t , the descent lemma as seen before, states that for a constant learning-rate γ > 0 ,
[Equation]
[Equation] Here [Equation] the inequality (i) uses the fact that we choose our learning rate γ ∈ (cid:0) 0 , 1L (cid:3) so that the term (cid:16) 12 γ − L2 (cid:17) is non-negative, and the MSE result of DME (Theorem 3.2), as the last term can be upper bounded as,
[Equation]
Here, (ii) follows from the tower rule of conditional expectation, and the inner expectation is over the stochasticity of the intermittent connectivities, i.e., { τ i } i ∈ [ n ] and { τ ij } i,j ∈ [ n ] .
The inequality (iii) follows from the result on DME (Theorem 3.2), and G max ≜ max i ∈ [ n ] G i .
Using the PL-inequality and taking expectation over all sources of stochasticity, we have,
[Equation]
Recursively applying this for t = 0 , . .
, T − 1 , we get,
[Equation]
This shows that for smooth objective functions that satisfy the PL condition, ColRel converges to a neighborhood of the optimal solution and the size of the neighborhood is once again determined by
S( p , P , A ) .
Note that the size of the convergence neighborhood goes to 0 as the number of clients n → ∞ .
Furthermore, it is possible to get rid of the neighborhood and achieve exact convergence to the optimal solution using a diminishing learning-rate schedule.
However, the convergence rate fails to be linear in that case, and deteriorates to a rate of O (cid:0) 1 T (cid:1) .
Once again, results of Theorem 4.2 can be extended to the case with stochastic gradient oracle as discussed in App.
G.
F Proof of Theorem 4.3: Smooth and Non-Convex Objectives
In this section, we derive the rate of convergence of gradient descent with collaborative relaying ( ColRel-GD ) to a stationary point for general non-convex objective functions.
We still assume that the function is smooth and continuous as in Assumptions E.1 and E.2.
The algorithm is the same as
Alg. 2 with gradients ∇ f i ( x t ) evaluated at each client i ∈ [ n ] instead of a subgradient.
Using the general descent Lemma E.4,
[Equation]
Choosing the learning rate γ ∈ (cid:0) 0 , 1L (cid:3) [Equation] we get,
[Equation]
Summing these inequalities for t = 0 , . .
, T − [Equation] we have,
[Equation]
Since the expected suboptimality gap D T ≥ 0 and D 0 is the initial suboptimality gap.
This yields,
[Equation]
Note that the LHS of (42) can be interpreted as E (cid:104) ∥∇ f ( (cid:98) x ) ∥ 22 (cid:105) where (cid:98) x is chosen uniformly at random from { x 0 , . . .
x T − 1 } .
Furthermore, (42) also implies that the best approximate stationary point amongst the iterates satisfies,
[Equation]
The convergence rate if O (cid:0) 1 T (cid:1) , and the iterates convergence to a neighborhood whose size is determined by S( p , P , A ) .
This is typical since in stochastic gradient descent, the iterates convergence to a neighborhood of a stationary point and the size of that neighborhood is determined by the variance of the stochastic gradients.
In our setting, the stochasticity inherently arises from the intermittent links in the network, i.e., the TIV σ 2tv ( p , P , A ) .
Note that the size of the convergence neighborhood goes to 0 as the number of clients n → ∞ .
This completes the proof.
Once again, the results of this section can be extended to the setting with stochastic gradient oracle, as discussed in App.
G.
G Extensions to Stochastic (Sub) Gradient Oracle
In §3, we saw that intermittently connected links introduce an additional source of variance to the information received from each client at the PS.
In principle, the TIV σ 2tv is similar to the variance due to other sources of stochasticity such as that arising from a stochastic (sub) gradient oracle.
Since these sources of stochasticity are independent, the resulting variance at the PS is the sum of all the variances.
Stochastic subgradient method with ColRel .
In the presence of a stochastic subgradient oracle, at any iteration t , each client i ∈ [ n ] now returns (cid:98) g i ( x t ) .
We consider the following standard assumptions on the subgradient oracle.
Unbiasedness :
For any i ∈ [ n ] an x t , the stochastic subgradients satisfy, E [ (cid:98)
g i ( x t ) (cid:12)(cid:12) x t ] ∈
∂f i ( x t ) .
Bounded second moment : There exists a constant (cid:101) G > 0 such that for any t , the stochastic subgradients returned by the oracle satisfy, E [ ∥ (cid:98) g i ( x t ) ∥ 22 ] ≤ (cid:101) G 2 i .
A straightforward modification of the analysis in §D yields the same convergence rate as in Theorem 4.1 with (cid:101) G max instead of G max , where (cid:101) G max ≜ max i ∈ [ n ] (cid:101) G i
Stochastic gradient descent with ColRel .
Suppose at every iteration t , the stochastic gradient oracle at each client returns a stochastic gradient (cid:98)
g i ( x t ) that satisfies the following:
Unbiasedness :
For all i ∈ [ n ] and any iteration t , we have, E [ (cid:98)
g
i ( x t )]
= ∇ f i ( x t ) .
Bounded second moment : For all i ∈ [ n ] and any iteration t , we have, E (cid:2) ∥ (cid:98) g
i ( x t ) ∥ 22 (cid:3) ≤ β ∥∇ f i ( x t ) ∥
22 + σ 2 , where β and σ 2 are non-negative constants and depend inversely with the batch size used for computing the stochastic gradient.
With minimal modification in the analysis that we derive in App.
E, one can show that the following holds for smooth and PL objectives,
[Equation]
Whereas, for smooth non-convex objectives, we have,
[Equation]
H Numerical Simulations: Extended
In this section, we first present numerical simulation results on DME in §H.1, and then we discuss the federated optimization simulations of §5 in more detail.
H.1 Distributed Mean Estimation (DME)
For the simulations in this section, we have considered a cluster of clients that communicate with the PS to estimate the mean of their vectors.
Amongst the clients, a few “good" clients have a high probability of successfully transmitting their data to the PS, while the remaining “bad" clients can only convey their data to the PS by relaying through one of the good clients.
The number of good clients is varied along the X -axis while the MSE for estimating the mean is plotted on the Y -axis. In Fig.
3, we plot and visually compare naïve and ColRel strategies.
The naïve strategy uses no collaborations, i.e., we set p ij = 0 for all
i ̸
= j , with A = diag( p − 1 1 , . .
. , p − 1 n ) . Although without collaboration, this choice of A for the naïve strategy ensures an unbiased estimate at the PS (i.e. (2) is satisfied), albeit with a higher MSE. We consider n = 10 clients and the dimension d = 100 , with the probability of a good client successfully transmitting to the PS is p good = 0 . 9 (varied across the different plots), whereas the bad clients have a probability p bad = 0 .
The clients can collaborate
9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good = 0.5 Naive
Collaboration        9 No. of good connectivity clients 7.0 7.
M e a n s q u a r e d e rr o r p good = 0.7 Naive
Collaboration        9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good = 0.9 Naive
Collaboration
9 No. of good connectivity clients 27.
M e a n s q u a r e d e rr o r p good = 0.5 Naive
Collaboration        9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good = 0.7 Naive
Collaboration        9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good = 0.9 Naive
Collaboration of
M e a n s q u a r e d e rr o r p good =0.5 Naive
Collaboration        9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good =0.7 Naive
Collaboration        9 No. of good connectivity clients
M e a n s q u a r e d e rr o r p good =0.9 Naive
Collaboration with each other over a fully connected topology with intermittent client-client connections that are successful with probability p c = 0 .
That is, the collaboration is done over an Erd ˝ os-Rényi graph,
G( n = 10 , p = 0 .
The data at each client is generated from a Gaussian distribution N (0 , 1) , and then each coordinate is raised to the power of 3 .
Raising the coordinates of a Gaussian vector to the third power generates a heavy-tailed distribution.
Consequently, if a client that has vector with a few large coordinate values is unable to convey its data to the PS due to a failed transmission (as is the case in naïve DME), this can incur a significant MSE.
On the contrary, ColRel ensures that the poor connectivity clients can still relay their updates to neighboring nodes, resulting in comparatively smaller MSE.
The plot are averaged over 50 realizations.
The data and the intermittent connectivity is generated independently at each realization.
While generating the intermittent connectivity between clients, we ensure that if client- i does not collaborate with client- j , then even client- j does not collaborate with client- i .
For instance, this is motivated by the setting where a physical obstacle blocks communication between two clients over a wireless network.
In Fig.
4, we increase the dimension to d = 1000 and obtain the same plots.
In Fig.
5, we consider a different topology for client-client collaboration, wherein clients are constrained to communicate with other clients over a ring topology, i.e., p ii = 1 for all i ∈ [ n ] , p ij = p c = 0 .
8 iff j = i ± 1 , i ± 2 , i ±mod n , and p ij = 0 otherwise.
In the next subsections §H.2, we discuss our federated learning simulation setup in more details.
H.2 Federated Learning Simulations
Simulation setup : We train a ResNet- 20 model for image classification on the CIFAR-10 [22] dataset.
We distributed the training set of 50 , 000 images across n = 10 clients in both independent and identically distributed (IID) and non-IID fashions.
Non-IIDness of the data distribution amongst clients is prevalent in FL setups and to emulate it, we consider the sort-and-partition approach wherein the training data is initially sorted based on the labels, and then they are divided into blocks and distributed among the clients randomly based on a parameter s , that measures the skewness of [Figure]
the data distribution. More precisely, s defines the maximum number of different labels present in the local dataset of each user, and therefore, smaller s implies more skew in the data distribution.
We use s = 3 , i.e. each client has images from at most 3 classes.
The plotted results of all simulations are averaged over 5 independent realizations.
In between every communication round to the parameter server (PS), the clients execute 8 local training steps of local-SGD.
We utilize the SGD optimizer at the clients with a global momentum ( β = 0 . 9) at the PS, a learning rate of 0 . 05 for SGD, a coefficient of 10 − 4 for ℓ 2 -regularization to prevent overfitting, and a batch-size of 64 .
All simulations were carried out on NVIDIA GeForce GTX 1080 Ti with a CUDA
Version 11 . 4 .
We also note that we have ensured all the simulations to have the same step-size, i.e., the learning rate for different simulations has not been tuned individually.
We consider the following benchmarks with which we compare our proposed ColRel scheme:
FedAvg – perfect connectivity .
We consider FedAvg when all clients are able to successfully transmit their local updates to the PS at every communication round.
This serves as a natural upper bound to the performance of any algorithm proposed in the presence of intermittent connectivity.
FedAvg – Blind .
As a natural performance lower bound in the presence of intermittent client connectivity, we consider a naïve FedAvg strategy wherein the PS is unaware of the identity of clients.
In this strategy, for the clients that are unable to send their updates to the PS due to a communication failure, the PS simply assumes that their update is zero.
Essentially, the PS adds all the local updates it receives at any communication round, and divides it by the total number of clients irrespective of the knowledge of the number of actual successful transmissions.
Such blind averaging strategies are often the norm for FEEL employing OAC. 3.
FedAvg – Non-Blind .
As another benchmark, we also consider a non-blind strategy, where the PS is aware of the identity of the clients, and knows exactly, how many and which clients have successfully been able to send their local update to the PS.
This is common in point-to-point learning settings.
In this case, the PS simply ignores the clients that have been unable to send their updates, and averages the successful updates by dividing the global aggregate at the PS by the number of successful transmissions
In our simulations in Figs.
2a, 2b, and 2c (reproduced here as Figs. 6, 7, and 8 for better resolution), we compare the above-mentioned benchmarks with our proposed ColRel strategy in the presence of intermittent connectivity of clients to the PS, as well as amongst themselves.
In order to demonstrate the improved performance of our proposed strategy with respect to the above-mentioned benchmarks, we consider the following setups: [Figure] [Figure]
Effect of topology : The decentralized topology according to which the clients collaborate in sharing their updates plays an important role in the performance of ColRel.
In Fig.
2a, we consider a ring topology in which each client is constrained to communicate with only 2 of their neighboring clients.
In this setting, when a connection is present between any two clients (so that it presents an opportunity for collaboration), the communication link is not intermittent.
More specifically, the matrix P is given by, p ij = 1 iff j = − 1 , 0 , or, + mod n .
On the other hand, in Fig.
2b, we consider client-client collaboration over an
Erd˝os-Rényi topology In other words, each client can collaborate with any other client but over intermittently connected links, i.e., 0 < p ij < 1 for i ̸
= j .
Although over intermittent links, this provides an increased opportunity for collaborations, and consequently, the final test accuracy (after 1000 communication rounds) is higher in Fig.
2b compared to Fig.
2a.
Furthermore, later in this section, we also present a brief discussion for FL over networks with mmWave links, where the connectivity probabilities are determined by the clients’ and PS’s geographical locations.
In particular, we investigate a setting where intermittent connectivity between clients might be preferred over perfect connectivity because it leads to more increased collaboration on average. [Figure]
Non-IID data distribution : We evaluate the performance of C OL R EL in the presence of non-IID local data distribution.
C OL R EL outperforms other strategies when some clients with important local data have persistently poor connectivity.
In Figs.
2a an 2b, we consider an iid data distribution amongst clients.
In this case, FedAvg –
Non-Blind performs quite comparably to ColRel.
This is because in the absence of any noniidness in the local data distribution amongst clients, a naïve strategy like simply ignoring the clients with failed transmissions, and averaging only the received updates still performs reasonably well, because the aggregated gradient at the PS is still an unbiased estimate of the true global gradient at the PS, albeit with a slightly higher variance due to a fewer number of clients that successfully transmit.
However, such a naïve strategy fails in Fig.
2c where we consider a non-iid local data distribution across clients.
This is because an important subset of the training data is being possessed by clients that have poor connectivity to the PS and cannot relay their updates to the PS.
ColRel ensures that even the clients with bad connectivity to the PS can relay their updates, ensures a convergence rate comparable to FedAvg - perfect connectivity.
Heterogeneity in client connectivity : Different clients may have different connectivity to the PS, and we show that as long as there is one client with reasonably good connection to the PS, its neighbors can relay their updates to it – yielding improved performance for
C OL R EL .
We consider this in all the simulation plots we have simulated.
Network topology with mmWave links : We now consider a network topology with mmWave links in which the probability of successful transmissions between clients and between clients and the PS are determined according to [2] as a function of distance ( δ ) between them as, p = min (cid:0) 1 , e − δ/ 30+5 .
2 (cid:1) .
Here, we consider a comparison between non-intermittent (perfectly available) client-client collaboration, vs. intermittent client-client collaboration.
This is similar to the difference in topologies of Fig.
2a (where the client-client collaboration is deterministic), versus Fig. 2b (which considers intermittency that we have allowed for in the analytical formulation as well).
In Fig.
9a, we consider that two clients can collaborate perfectly if p ij ≥ 0 .
99 and determine the network topology accordingly.
In other words, p ij and p ji are high enough to consider the links to be consistently present.
The PS is at the origin and the clients are distributed in a way such that only three of them can communicate with the PS.
Similarly, in Fig.
9b, the connectivities between clients are intermittent.
To avoid links that are too unreliable, if p ij < 0 .
5 , we consider those clients do not collaborate.
The geographical locations of the clients is the same in both the scenarios.
Note that Fig.
9b has a few additional links compared to Fig. 9a.
In Fig.
10, we note that allowing for intermittent collaboration amongst clients results in improved convergence rate of training.
Moreover, both perfect as well as intermittent connectivity outperform FedAvg without collaboration.
Network topology with mmWave links for n = 20 clients : We increase the number of clients in our mmWave system from 10 to 20 .
In this setup, we also reduced the non-iid skewness parameter of local data distribution to s = 2 .
We consider two topologies (hereafter referred to as Setup [Figure] and Setup 2) for geographic locations of the clients.
Setups 1 and 2 are visualized in Figs.and 13 respectively; clients in Fig.
11 enjoy better connectivity to the PS and to one another.
In these figures red lines depict client-client connections, and black lines denote client-PS connections.
Additionally, solid lines denote stable connections where the connectivity probability is 1 , and dashed and dotted lines denote intermittently connected links.
As before, the connectivity probabilities are determined as characterized by mmWave links [2], according to p = min (cid:0) 1 , e − δ/ 30+5 . 2 (cid:1) , where δ is the distance between two nodes.
In Setup 1, we have the following probabilities of intermittently connected client-client links: p ij = 0 .
8 where the unordered pairs { i, j } can take the values { 1 , 4 } , { 2 , 3 } , { 6 , 9 } , { 7 , 8 } , { 12 , 13 } , { 14 , 17 } , { 16 , 19 } , { 17 , 18 } .
In Setup 2, we have the following probabilities of intermittently connected client-client links: p ij = 0 .
93 where the unordered pairs { i, j } can take the values { 2 , 3 } , { 7 , 8 } , { 12 , 13 } , { 17 , 18 } .
Furthermore, the clientPS connectivity probabilities for Setups 1 and 2 are p =[0 .
01] , and p =[0 .
11] , respectively.
We plot the resulting test accuracy as a function of the number of rounds for Setups 1 and 2 in Figs.and 14, respectively.
The numerical results for each of these figures were averaged over 5 realizations.
Additionally, we compare our proposed ColRel algorithm with the FedAvg - perfect connectivity,
FedAvg - Blind and FedAvg - Non-Blind.
The performance improvement of our proposed ColRel algorithm is consistent as seen in Figs.
12 and 14; in Setup 1, ColRel has approximately 40% higher accuracy with respect to FedAvg - Blind and 68% with respect to FedAvg - Non-Blind.
Furthermore, a comparison between Figs.
12 and 14 shows that increasing the connectivity of client-client links and client-PS lead to a better final accuracy.
We note that while the accuracy of all the setups with intermittently connected links increased as the connectivity improves, our proposed ColRel algorithm still outperforms the FedAvg - Blind and FedAvg - Non-Blind algorithms by a large margin.
In fact for Setup 2, the ColRel algorithm leads to a training loss that is comparable to the FedAvg without any intermittent connectivity, i.e., FedAvg - Perfect Connectivity. [Figure] c [Figure]
c
[Figure]
c
[Figure]
c 6 cc 8 c 9 ccccccccccc
PS mmWave: 20 clients Setup
[Figure]
[Figure]
[Figure]
[Figure]