C ONVERGENCE A NALYSIS OF S PLIT L EARNING ON N ON -IID D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Split Learning (SL) is one promising variant of Federated Learning (FL), where the AI model is split and trained at the clients and the server collaboratively.
By offloading the computation-intensive portions to the server, SL enables efficient model training on resource-constrained clients.
Despite its booming applications,
SL still lacks rigorous convergence analysis on non-IID data, which is critical for hyperparameter selection.
In this paper, we first prove that SL exhibits an
O (1 / √ T ) convergence rate for non-convex objectives on non-IID data, where T is the number of total iterations.
The derived convergence results can facilitate understanding the effect of some crucial factors in SL (e.g., data heterogeneity and local update steps).
Comparing with the convergence result of FL, we show that the guarantee of SL is worse than FL in terms of training rounds on non-IID data.
The experimental results verify our theory.
Some generalized conclusions on the comparison between FL and SL in cross-device settings are also reported.
I NTRODUCTION
Federating Learning (FL) is a popular distributed learning paradigm where multiple clients collaborate to train a global model under the orchestration of one central server.
There are two settings in Federating Learning (FL) (McMahan et al., 2017) including (i) cross-silo where clients are organizations and the client number is typically less than 100 and (ii) cross-device where clients are Iot devices and the client number can be up to 10 10 (Kairouz et al., 2021).
To alleviate the computation bottleneck at resource-constrained IoT devices in the cross-device scenario, Split Learning (SL) (Gupta & Raskar, 2018; Vepakomma et al., 2018) splits the AI model to be trained at the clients and server separately.
The computation-intensive portions are typically offloaded to the server, which is critical for the model training at resource-constrained devices.
SL is regarded as one of the enabling technologies for edge intelligence in future networks (Zhou et al., 2019).
r x
[Figure]
local optima global optima 1 r + x 1  x 2  x  x
IID
Non-IID
SL
FL
The comparisons of FL and SL are of practical interest for the design and deployment of intelligent networks.
Existing studies focus on various aspects for their comparisons (Thapa et al., 2020; Gao et al., 2020; 2021), e.g., in terms of learning performance (Gupta & Raskar, 2018), computation efficiency (Vepakomma et al., 2018), communication overhead (Singh et al., 2019), and privacy issues (Thapa et al., 2021).
For example, with the emphasis on the learning performance comparison, Gao et al.
(2020; 2021) find that SL exhibits (i) faster convergence speed than FL under IID data in terms of communication rounds; (ii) better learning performance under imbalanced data; (iii) worse learning performance under (extreme) non-IID data, etc.
The difference arises from the distinct process of model updates of FL and SL.
In particular,
FL takes the average of the local model parameters at the end of each round; SL only trains the clients in sequence and does not average the client updates.
Figure 1 plots the client drift (Karimireddy et al., 2020;
Wang et al., 2020; Li et al., 2022) of FL and SL under both IID and non-IID data to visualize the update process.
Under the IID setting, SL approach the global optima x ∗ faster than FL given the sequential training mechanism.
In contrast, under the non-IID setting, SL may deviate from the global optima for the same reason.
Convergence analysis is critical for the performance comparison between SL and FL.
Specifically, a rigorous analysis is of paramount importance for the vital research questions raised by (Gao et al., 2020) (which are only empirically evaluated but remain unsolved in theory):
RQ1-“ What factors affect SL performance? ”
and RQ2-“ In which setting will the SL performance outperform FL? ”.
A wealth of work has analyzed the convergence of FL in the cases of IID (Stich, 2018;
Zhou &
Cong, 2017; Khaled et al., 2020), non-IID
(Li et al., 2020; 2019; Khaled et al., 2020; Karimireddy et al., 2020) and unbalanced data (Wang et al., 2020).
However, with the distinct update process, the convergence analysis of SL has yet to be solved on non-IID data.
To this end, this paper first derives a rigorous convergence results of SL and draw the comparison results of FL and SL theoretically.
Main contributions.
The main contributions can be summarized with respect to the two research questions above:
• We prove the convergence of SL on non-IID data with the standard assumptions used in FL literature 1 with a convergence rate of O (1 / √ T ) in Section 4.2.
By this, we find that the convergence of SL is affected by factors such as data heterogeneity and the number of local update steps.
Experimental results verify the analysis results empirically in Section 5.2.
To the best of our knowledge, this work is the first to give the convergence analysis of SL on non-IID data.
• We compare FL 2 and SL in theory (Section 4.3) and in practice (Section 5.3).
Theoretically, the guarantee of SL is worse than FL in terms of training rounds on non-IID data.
Empirically, we provide some generalized conclusions of FL and SL in cross-device settings, including (i) the best and threshold learning rate of SL is smaller than FL; (ii) the performance of SL is worse than FL when the number of local update steps is large on highly non-IID data; (iii) the performance of SL can be better than FL when choosing small the number of local
update steps is small on highly non-IID data.
P RELIMINARIES AND A LGORITHM OF S PLIT L EARNING
As two of the most popular distributed learning frameworks, both FL and SL aim to train the global model from distributed datasets.
The optimization problem of FL and SL with N clients can be given by
[Equation]
where p i = n i /n is the ratio of local samples at client i ( n and n i are the sizes of the overall dataset D and local dataset D i at client i , respectively), x is the model parameters, f ( x ) is the global objective, f i ( x ) denotes the local objective function on client i .
In particular, f i ( x ) :=
E ξ i ∼D i [ f i ( x ; ξ i )]
=n i (cid:80) ξ i ∈D i
f i ( x ; ξ i ) , where ξ i represents a data sample from the local dataset D i .
SL with the global learning rate.
The relay-based (sequential) training process across clients makes SL significantly different (from FL), which has been described in Algorithm 1.
Considering the massive number of clients in cross-device setting, only a subset S of clients are selected for model training at each round.
The update order of the selected clients can be meticulously designed or randomly determined (used in this paper).
The i -th client requests and initializes with the lasted model (step 4) and then performs multiple local updates (step 5-11) 3 .
After K local updates, the client will send the model parameters to the next client (i.e., the i + 1 -th client).
The local update process can be stated as:
[Equation]
where x ( r,k ) i denotes the complete local model (parameters) after the k -th local update on the local dataset D i in the r -th round, η l is the local learning rate and g i ( x ( r,k ) i ) :
= ∇ f i ( x ( r,k ) i ; ξ ( r,k ) i ) represents the stochastic gradients over on the mini-batch ξ ( r,k ) i sampled randomly from D i .
Note that the complete model x ( r,k ) i consists of the client-side model x ( r,k ) c,i and server-side model x
( r,k ) s,i , as shown in Algorithm 1.
The clients and server update their models synchronously and without loss of generality, we do not highlight the model locations in the following.
After the last client in S (i.e., the |S| -th client) completes its local updates, it will request the initial client-side model of the current round.
The global update is conducted in both the client and server:
[Equation]
where x r denotes the complete global model (parameters) in the r -th round ( x r equals x ( r
[Equation]
i only when i = 1 , which is different from FL), η g is the global learning rate.
Note in Eq.
(4) that we propose adding the global learning rate η g against the vanilla SL algorithm (Gupta & Raskar, 2018;
Vepakomma et al., 2018; Thapa et al., 2020; 2021).
The global learning rate design mechanism is originally developed in the FL setting (Karimireddy et al., 2020;
Reddi et al., 2020; Wang et al., 2020) to reduce the client drift, and can be readily adopted in SL for the same function.
Algorithmcan operate in both the centralized and peer-to-peer mode (see Appendix A) and is reduced to the vanilla SL if steps 13-14 removed.
For brevity, we have omitted some unconcerned details of SL in Algorithm 1, e.g., the security and privacy settings.
More details about SL can be found in
Appendix A or Gupta & Raskar (2018); Thapa et al.
Some notations :
– x c denotes the client-side model (parameters) – x s denotes the server-side model (parameters) – x :=
[ x c , x s ] denotes the complete model (parameters)
1: for round r = 0 , . . .
, R − do 2:
Sample a subset S of clients and determine their update order 3: for client index i = 1 , . .
. , |S| do 4:
Client i : Request the latest client- model and initialize x ( r, 0) c,
i
[Equation]
5: for local update
. , K do 6:
Client
Local model updates:
[Equation]
Server:
[Equation]
11: end for 12: end for 13:
Client side model x r c of the current round 14:
Global model updates:
[Equation] [Equation]
Client |S| and Server: Store the global model 16: end for
R ELATED WORK
Variants of SL.
SL is deemed as a promising paradigm for distributed model training at resourceconstrained devices, given its computational efficiency on the client side.
Most existing works focus on reducing the training delay arising from the relay-based training manner in the multi-user scenario.
SplitFed (Thapa et al., 2020) is one popular model parallel algorithm that combines the strengths of FL and SL, where each client has one corresponding instance of server-side model in the main server to form a pair.
Each pair constitutes a complete model and conducts the local update in parallel.
After each training round, the fed server collects and aggregates on the clien-side local updates.
The aggregated client-side model will be disseminated to all the clients before next round.
The main server does the same operations to the instances of the server-side model.
SplitFedv (Thapa et al., 2020), SplitFedv3 (Gawali et al., 2021) and SFLG (Gao et al., 2021), FedSeq (Zaccone et al., 2022) are the variants of SplitFed .
In particular, SFLG (Gao et al., 2021) is one generalized variants of SplitFed , combining SplitFed and SplitFedv2 .
Convergence analysis of SL.
In the case of IID data, the convergence analysis of SL is identical to standard Minibatch-SGD (Wang et al., 2022; Park et al., 2021), so some convergence properties of Minibatch SGD is applied to SL too.
The algorithm in Han et al.
(2021) reduces the latency and downlink communication on SplitFed by adding auxiliary networks at client-side for quick model updates.
Their convergence analysis combines the analysis of Belilovsky et al.
(2020) and
FedAvg .
Wang et al.
(2022) proposed FedLite to reduce the uplink communication overhead by compressing activations with product quantization and provided the convergence analysis of
FedLite .
However, their convergence recovers that of Minibatch SGD when there is no quantization.
SGD with biased gradients (Ajalloeian & Stich, 2020) is also related.
However, it only converges to a neighborhood of the solution.
Furthermore, we find that Woodworth et al.
(2020a;b) compared the convergence of distributed Minibatch SGD and local SGD under homogeneous and heterogeneous settings, respectively.
To differentiate our work, we show how these algorithms operate in Appendix B. As a result, the convergence of SL on non-IID data is still lacking.
C ONVERGENCE A NALYSIS OF SL 4. A SSUMPTIONS
We make the following standard assumptions on the local objective functions { f i ( x ) } Ni =1 .
Assumption 1 ( L -smooth) .
Each local objective function f
i is L -smooth, i ∈ { 1 , 2 , . . .
, N } , i.e., there exists a constant L > 0 such that ∥∇ f i ( x ) − ∇ f i ( y )
∥
≤ L ∥ x − y ∥ for all x and y .
Assumption 2 (Unbiased gradient and bounded variance) .
For each client i , i ∈ { 1 , 2 , . . . , N } , the stochastic gradient g i ( x ) := ∇ f i ( x ; ξ i ) is unbiased E [ g i ( x )]
= ∇ f i ( x ) and has bounded variance
E ξ i [ ∥ g i ( x ) − ∇ f i ( x ) ∥ 2 ] ≤ σ 2 .
Assumption 3 (Bounded dissimilarity) .
There exist constants B ≥ 1 and G ≥ 0 such that
N (cid:80) Ni =1 ∥∇ f i ( x ) ∥ 2 ≤ B 2 ∥∇ f
( x ) ∥ 2 + G 2 .
In the IID case, B = 1 and G = 0 , since all the local objective functions are identical to each other.
In the non-IID case, B and G measure the heterogeneity of data distribution.
4. C ONVERGENCE R ESULT AND D ISCUSSION
Without loss of generality, all the clients participate in the SL process ( |S| = N ) and the unweighted global objective function f ( x ) =
1 N (cid:80) Ni =
1 f i ( x ) is adopted.
Note that the results in the unweighted case can be readily extended to the weighted case of Eq.
Following the proof of Wang et al.
; Karimireddy et al.
(2020); Khaled et al.
(2020), we give the convergence results of SL (details are in Appendix D), as follows:
Theorem 1.
Let Assumptions 1, 2 and 3 hold.
Suppose that the local learning rate satisfies η l ≤
[Equation]
For Algorithm 1, it holds that
[Equation]
where ¯ x R = 1 R (cid:80) R − 1 r =0 x r is the averaged global model over the R rounds.
Corollary 1.
Choose η g η l = 1 L √ T and apply the result of Theorem 1.
For sufficiently large T ( T ≥ 4 N
[Equation]
K
[Equation]
max
[Equation]
+
[Equation]
g , 1 (cid:111) ), it holds that
[Equation]
variance , (6) where T is the total number of iterations (i.e., NKR in SL).
The upper bound of E [ ∥∇ f
(¯ x R ) ∥ 2 ] consists of three types of terms: (i) initialization error, (ii) client drift error, caused by the client drift (see Lemma 4 in Appendix D.2), (iii) global variance.
We can see that the result demonstrates the relationships between convergence and factors such as the number of clients, the data heterogeneity, the global/local learning rates and the local update steps.
According to the result, a large η l means higher rate that the initialization error decreases at but causes large client drift error and global variance.
Next, we discuss the convergence rate and the influence factors of SL in detail.
Convergence rate.
By Corollary 1, for sufficiently large T , the convergence rate is determined by the initialization error and global variance (see Eq.
(6)), resulting in a convergence rate of O (1 / √ T ) .
We can recover the convergence of SGD (Bottou et al., 2018; Wang et al., 2020) when N = 1 and
K = 1 (i.e., without the client drift error) — It is true but not shown in Theorem 1 directly, since it is complicated to write the constant details in Eq.
We defer the discussion to Appendix D.5.
Effect of data heterogeneity.
According to Theorem 1, when on highly non-IID data ( B and G are large), a small η l is required for the convergence of SL (see the condition of Theorem 1).
In addition, the client drift error also increases.
As a result, large data heterogeneity harms the convergence of
SL, which is consistent with the previous study (Gao et al., 2020; 2021).
Effect of K .
K is the number of local update steps.
An immediate question is whether we can improve the convergence by adding local update steps when R is fixed.
The answer is yes.
By Eq.
(5), as K increases, the initialization error decreases and the client drift error increases, which implies that the optimal K exists.
We can further get that larger data heterogeneity makes the optimal K smaller based on Eq.
This property is analogous to FL (McMahan et al., 2017).
Effect of η g .
The global learning rate η g can be used to reduce the client drift without hurting the progress of SL.
A large η g reduces the client drift error, hence improving the convergence rate.
For example, by Corollary 1, SL shows O ( LF + σ
[Equation]
) convergence rate if
[Equation]
( F := f ( x 0 ) − f ( x ∗ ) ); while it shows a convergence rate of O ( LF + σ 2 √ T
[Equation]
Some detailed analysis of these factors are deferred to Section 5.2, combined with the experiments.
4. C OMPARISON BETWEEN FL AND SL
In this section, we first give the convergence result of FL based on our setting, and then compare the results of FL and SL to answer the second question “ In which setting will the SL performance outperform FL? ”
theoretically.
We summarize the convergence results of FL ((Wang et al., 2020), reproduced by Theorem 2 in
Appendix D.3) and SL (Theorem 1) in Table 1.
The convergence guarantee of FL is one of the best known convergence results for the non-convex functions of FL, which makes our comparison [Table] persuasive.
Choosing η l = √ N L √ T for Theorem 2 provides the O (1 / √ NT ) convergence rate, the linear speedup in FL.
The constraint on the local learning rate of SL is tougher than FL.
The local learning rate of SL (see Theorem 1) has tougher constraints than FL (see the 2-nd row of Table 1), which indicates SL is more sensitive to the heterogeneity of data.
This is significant for the selection of the learning rate of SL in practice (see the comparison experiments).
Effective learning rate.
We next focus on comparing FL and SL in terms of rounds.
Note that this comparison (running for the same R ) is fair given the same total computation cost (including the computation cost on client-side and server-side).
Beginning with the observation that the convergence guarantees and constraints seem very alike if choosing η l ( SL ) =
η l ( FL ) /N —
η l ( FL ) and η l ( SL ) denote the local learning rate of FL and SL respectively, we define the effective learning rate ˜ η
FL :=
Kη g η l for FL and ˜ η SL :=
NKη g η l for SL as Karimireddy et al.
Wang et al.
(2020) did.
Note that the effective learning rate is unequal for FL and SL.
As a result, we obtain the convergence guarantee of the effective learning rate version exhibited in the 3-rd and 5-th rows in
Table 1.
The guarantee of SL is worse than FL in terms of training rounds on non-IID data.
To make a comparison, we need to choose appropriate η l for both.
Considering ˜ η FL and ˜ η SL has the same constraints, we can choose ˜ η
FL = ˜ η SL = 1 / √ R for both bounds (see the 3-rd and 5-th rows in Table 1).
Then we get: (i) O (cid:16) F [Equation] R + KG [Equation]
+
[Equation]
2 KR + [Equation] R (cid:17) of FL and (ii) O (cid:16) [Equation]
R + KG 2 +
[Equation] 2 KR [Equation] (cid:17) of SL after R rounds, i.e., the round complexity R ϵ shown in 4-th and 6-th rows in Table 1.
Then for sufficiently large R , R ϵ is determined by the first and the second term (see the 4-th and 6-th rows).
In particular, the only difference in the complexity appears in the second term (we have marked it with red), which indicates that the guarantee of SL is worse than FL in terms of rounds.
However, we note that the gap is not obvious when K or σ is small.
Further, considering the constants advantage of SL (It is true but not shown in Theorem 1, see Appendix D.2), the performance comparison between FL and SL is still uncertain .
To make a more detailed comparison, we have conducted adequate experiments and given some generalized conclusions in Section 5.3.
E XPERIMENTS
Our experiment environment is ideal (without the communication and computation restrictions), nevertheless, is enough to examine our convergence theory.
The convergence rate is evaluated in terms of the training rounds.
We demonstrate the detailed experimental setup in Section 5.1, evaluate the effect of the factors on the performance of SL in Section 5.2, and compare FL and SL in crossdevice settings in Section 5.3.
More experimental details are in Appendix E.
5. E XPERIMENTAL S ETUP
Datasets and models.
We adopt the following setups: (i) training LeNet-5 (LeCun et al., 1998) on the MNIST dataset (LeCun et al., 1998); (ii) training LeNet-5 on the Fashion-MNIST dataset (Xiao et al., 2017); (iii) training VGG-11 (Simonyan & Zisserman, 2014) on the CIFAR-10 dataset (Krizhevsky et al., 2009).
For SL, the LeNet-5 is split after the second 2D MaxPool layer, with 6% of the entire model size retained in the client; the VGG-11 is split after the third 2D MaxPool layer, with 10% of the entire model size at the client.
Ideally, the split layer position has no effect on the performance of SL (Wang et al., 2022).
Data distribution.
Both IID and non-IID datasets are considered in the experiments.
For the nonIID setting, we adopt two mechanisms to generate non-IID data: (i) using a Dirichlet distribution
Dir ( α ) to generate mildly non-IID data, where a smaller α indicates higher data heterogeneity (Hsu et al., 2019; Zhu et al., 2021); (ii) using a similar mechanism like (McMahan et al., 2017; Zhao et al., 2018) to generate the pathological non-IID data, e.g., one distribution in which most clients only contain samples from 2 (5) classes, denoted as C = 2 (5) .
Note that the data distribution generated by the first mechanism is unbalanced and the second is balanced.
We will use “ α = # ” and “ C = # ” to represent the different data distributions in the following, where # is the parameter.
5. E XPERIMENTAL R ESULTS OF SL
[Figure]
In this section
[Figure]
we study the effects of data heterogeneity local update steps using the MNIST dataset.
The training samples are assigned to 10 clients.
For data heterogeneity, we use four distributions:
IID, C = 8 , 5 , 2 .
For K , we use five settings, E = 1 , 2 , 4 , 8 , 10 over Dir 10 (1 . 0) distribution.
E is the local epochs (McMahan et al., 2017), which satisfies K = max { En i /b } , where b is the mini-batch size.
So E can measure the value of K with n i and b fixed.
The results on Fashion-MNIST and
CIFAR-10 are deferred to Appendix E.1.
Effect of data heterogeneity.
As shown in Figure 2a, the training loss curve of IID distribution is the lowest and most stable.
When the data heterogeneity increases ( C decreases from 8 to 2),
SL shows worse performance.
The phenomenon is in accordance with our analysis that large data heterogeneity harms the convergence of SL in Section 4.2.
Effect of K .
Figure 2b shows the training loss in terms of communication rounds R .
SL shows the best performance when E = 2 .
This verifies that optimal K exists and suitable K can improve the convergence.
Over-large K can even harm the convergence rate (see curves E = 8 and E = 10 ).
Effect of η g .
The effect of η g in FL has been empirically studied in Reddi et al.
They tune η l and η g by grid search.
We follow a similar method to study η g of SL by choosing different combinations of η g and η l .
As shown in Figure 2c, the dark green grids with high test accuracy concentrate on the left bottom triangular regions, which shows that η g η l should avoid being too large.
Also note that η g cannot be set infinitely too large and has a limited range, which is also observed in FL (Reddi et al., 2020).
Thus tuning η g in the limited range is suggested.
Besides, we find that η g can be introduced to other SL frameworks, like SplitFed .
However, further research on η g is needed to address the issues such as how to tune η g to gain improvement?
in FL and SL.
5. E MPIRICAL C OMPARISON BETWEEN FL AND SL
We have compared FL and SL theoretically in Section 4.2.
The question arises that how about the learning performance of SL in practice compared to FL?
In the previous work, the same learning rates are used to evaluate the performances of SL and FL (Gao et al., 2020; 2021).
However, theoretical analyses in Section 4.2 show that the appropriate learning rate for SL may deviate from that of FL.
Thus, we evaluate the performance of SL and FL with different learning rates for fair comparison.
The learning rates are selected from { 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1 } .
To evaluate the effect of K on the performance comparison of FL and SL, we adopt two settings: E = 1 and
E = 10 .
We run 1000 rounds of training on MNIST, Fashion-MNIST and 4000 rounds on CIFAR10 dataset when E = 1 ; run 100 rounds of training on MNIST, Fashion-MNIST and 400 rounds on CIFAR-10 dataset when E = 10 .
The average top-1 test accuracy over the last 10% rounds are shown in Table 2, e.g., the average accuracy over the last 400 rounds of the total 4000 rounds on
CIFAR-10 when E = 1 .
Cross-device setting.
We compare the performance of FL and SL in cross-device settings, where the client number is enormous and the local dataset size is small.
Specifically, (i) MNIST: the training data is split into 1000 clients; (ii) Fashion-MNIST: the training data is split into 1000 clients; (iii)
CIFAR-10: the training data is split into 500 clients.
The number of samples per client depends on the data distribution, e.g., 60 (MNIST), 60 (Fashion-MNIST) and 100 (CIFAR-10) samples per client in the second partition mechanism.
The mini-batch size is 10 for all setups under the consideration of low computation power of IoT devices.
The original test sets are used to evaluate the generalization performance of the global model (test accuracy) after each training round.
The best and threshold learning rate of SL is smaller than FL.
We refer to the learning rate making the best test accuracy and minimal learning rate making the training die as the best learning rate and the threshold learning rate.
According to our theory, the tougher constraint indicates the smaller threshold learning rate of SL and the math property of Eq.
(5) indicates a smaller best learning rate.
To verify this point, we use the “best” learning rate, which makes the “best” test accuracy among the learning rates we choose, to substitute the actual best learning rate.
The “threshold” learning rate is defined alike.
As shown in Table 2, the “best” and “threshold” learning rates of SL are smaller than FL, especially on highly non-IID data (e.g., α = 0 . 2 ).
Furthermore, we note that the “best” and “threshold” learning rate turn small as the heterogeneity of data becomes large, which is also in accordance with our theory.
Performance comparison on IID data.
SL can obtain a faster convergence rate with comparable or even higher test accuracy than FL (see the left plot in Figure 3 and Table 2).
This is identical to the conclusion in Gao et al.
Performance comparison on non-IID data.
Our theory proves that the guarantee of SL is worse than FL in terms of rounds when K is large.
As shown in Table 2, almost all the “best” test accuracy of FL on very highly non-IID data (i.e., α = 0 .
2 and C = 2 ) beats that of SL
when E = 10 .
When
E ( K ) is large, FL converges faster than SL too (see the middle plot in Figure 3).
However, we find that SL has a better performance than FL when E ( K ) is small (see the E = 1 column in Table 2 and the right plot in Figure 3).
Even in some cases ( α = 0 .
5 and C = 5 on CIFAR-10) when E = 10 ,
SL is better.
[Figure]
[Table]
C ONCLUSION
In this work, we first derived the convergence guarantee of SL for non-convex objectives on nonIID data.
The results reveal that the convergence of SL is affected by the factors such as data heterogeneity and local update steps.
Furthermore, we compare SL against FL theoretically and empirically, ending up with the conclusions that (i) the best and threshold learning rate of SL is smaller than FL; (ii) the performance of SL is worse than FL when the number of local update steps is large on highly non-IID data; (iii) the performance of SL can be better than FL when the number of local update steps is small on highly non-IID data.
Our work can bridge the gap between FL and
SL, provide deep understanding of these two approaches and guide the deployment of these two in real-world applications.
C ONTENTS 1 Introduction
2 Preliminaries and Algorithm of Split Learning 3 Related work 4 Convergence Analysis of SL 4.
Assumptions . .
Convergence Result and Discussion . .
Comparison between FL and SL . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Experiments 5.
Experimental Setup . .
Experimental Results of SL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Empirical Comparison between FL and SL . . . . . . . . . . . . . . . . . . . . . .
6 Conclusion
A More details about Split Learning
Additional related work
C Summary of theories
D Proof of results
D.1 Basic technical lemmas and Notations . . .
D.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.4 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.
Extreme cases . .
E More experimental details
E.1 More results of SL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E
setting
A M ORE DETAILS ABOUT S PLIT L EARNING
In this section, we provide more details about SL.
More discussions can be found in Gupta & Raskar (2018);
Thapa et al.
; Duan et al.
In SL, the complete (or full) ML model is split into two portions.
The portion of the complete model maintained by the clients is called client-side model .
The portion maintained by the server is called server-side model .
In SL, the client-side model is owned by all the clients, while the server-side model is merely owned by the server.
Any client can operate with the server to complete the model training task but can not do it independently (without the help of the server).
Then we prepare some basic concepts in the paper.
Local update (step) The process that one client and the server cooperate to conduct the model updates, i.e., the inner loop of Algorithm 1.
Note that in SL, though the model update requires communication between the client and server, the model is still trained on the local dataset .
So the process is called local update (Thapa et al., 2020).
Local model
The concatenation of the client-side and server-side models after each local update.
Training round The process that all the clients (or a subset of clients selected) complete their local updates, i.e., the outer loop of Algorithm 1.
It is also called a global epoch or global round .
Global model
The concatenation of the client-side and server-side models after each global round.
Table 3 summarizes the notations.
In particular, the superscripts and subscripts of random variables appearing in the paper have the same form — a ( r,k ) i , where r is the index of rounds, k is the index of local update steps, i is the index of clients.
It means that the random variable a after the k -th local update on the local dataset D i in the r -th round.
The random variable a can be x (model parameters), ξ (random samples) or g (stochastic gradients).
[Table]
of iterations
R, r
Number, index of (training) rounds
N, i
Number, index of clients
S , |S|
Subset of clients sampled for training every round and its size
K, k
Number, index of local update steps;
K represents the maximum local update steps when the number of local update steps varies across clients n, n i
Sizes of the overall dataset D and local dataset D i at client
i
E
Local epochs; number of training passes each client makes over its local dataset
[Table]
The global, local learning rate
[Table]
[Table]
They are used to denote different distributions by the two mechanisms generating non-IID data respectively (see Section 5.1) x r
The complete global model (parameters) in the r -th round x (
[Table] [Table]
i
The complete local model (parameters) after the k -th local update on the local dataset D i in the r -th round x c , x
[Table]
The client-side model, server-side model g
i (
[Table]
( r,k ) i )
Stochastic gradients over on the mini-batch ξ ( r,k ) i sampled randomly from D i , also denotes as ∇ f i ( x ( r,k ) i ; ξ ( r,k ) i )
Two modes of SL.
In fact, there are two approaches of training in SL (i) with client-side model synchronization and (ii) without client-side model synchronization (Singh et al., 2019; Duan et al., 2022).
In this paper, SL is referred to the first approach, i.e., SL with client-side model synchronization.
As shown in Figure 4, there are two modes of SL with client-side model synchronization, including (i) the peer-to-peer mode and (ii) centralized mode.
The only difference between the peer-to-peer and centralized mode is the synchronization way of model parameters 4 .
• Peer-to-peer mode.
Clients store the model parameters by themselves after the local updates.
So clients get the latest model from other clients.
• Centralized mode.
Clients send the model parameters to the server after the local updates.
So clients get the latest model from the server.
The centralized mode requires more communication than peer-to-peer mode.
Implementation of SL with η g in the peer-to-peer and centralized modes.
In each round, the last client (client N in Figure 4) needs to request for the initial client-side model and conduct the global model update after completing its local update.
The server will conduct the global model update synchronously.
For the peer-to-peer mode, this process will cause (i) additional communication cost (the red arrow line at the left side of Figure 4) and (ii) additional storage cost of devices and the server (storing the global model).
For the centralized mode, this process will cause (i) additional communication cost (the red arrow line at the right side of Figure 4) (ii) additional storage cost of the server (the client-side global model is also stored in the server).
The communication cost of SL with global update is given in Table 4 based on Singh et al.
[Figure]
[Table] [Table]
B A DDITIONAL RELATED WORK
Variants of SL.
SL is deemed as a promising paradigm for distributed model training at resourceconstrained devices, given its computational efficiency on the client side.
Most existing works focus on reducing the training delay arising from the relay-based training manner in the multi-user scenario.
SplitFed (Thapa et al., 2020) is one popular model parallel algorithm that combines the strengths of FL and SL, where each client has one corresponding instance of server-side model in the main server to form a pair.
Each pair constitutes a complete model and conducts the local update in parallel.
After each training round, the fed server collects and aggregates on the clien-side local updates.
The aggregated client-side model will be disseminated to all the clients before next round.
The main server does the same operations to the instances of the server-side model.
However, in
SplitFed , the main server is required for great computing power to support the multiple instances of server-side model.
To address this issue, Thapa et al.
(2020) proposed SplitFedv2 .
The only change in SplitFedv2 is that only one server-side model in main server .
So the server-side model has to process the activations (or smashed data) of the client-side model in sequence.
The model gets updated in every single forward-backward propagation, which means that the server-side model makes more updates than the client-side model.
The main point of SplitFedv3 (Gawali et al., 2021) is substituting alternative client training with alternative the mini-batch training.
In vanilla
SL, the local model makes one or more training passes over the local dataset (alternative client training).
This may cause “catastrophic forgetting” issue (Gawali et al., 2021; Duan et al., 2022).
So
SplitFedv3 propose to use alternate mini-batch training, where a client updates its client-side model on one mini-batch, after which the client next in order takes over (Gawali et al., 2021), to mitigate the issue.
SFLG (Gao et al., 2021) is one generalized variants of SplitFed .
The clients are allocated to multiple groups.
There is one server in each group.
The training inside the group is identical to SplitFedv2 .
Then the server-side “global” models per group are aggregated (e.g., weighted averaging) to obtain the server-side global model of all groups.
FedSeq (Zaccone et al., 2022) is the same as SFLG except that the training inside the group is identical to vanilla SL.
More variants can be found in Thapa et al.
; Duan et al.
Convergence analysis of SL.
The convergence analysis of SL is identical to standard MinibatchSGD on IID data (Wang et al., 2022; Park et al., 2021), so some convergence properties of Minibatch
SGD is applied to SL too.
The algorithm in Han et al.
(2021) reduces the latency and downlink communication on SplitFed by adding auxiliary networks at client-side to generate the local loss for model updating.
Their convergence analysis combines the analysis of Belilovsky et al.
(2020) and FedAvg .
Wang et al.
(2022) proposed FedLite to reduce the uplink communication overhead by compressing activations with product quantization and provided the convergence analysis of FedLite .
However, their convergence recovers that of Minibatch SGD when there is no quantization.
SGD with biased gradients (Ajalloeian & Stich, 2020) is also related.
However, it only converges to a neighborhood of the solution.
Furthermore, we find that Woodworth et al.
(2020a;b) compared the convergence of distributed Minibatch SGD and local SGD under homogeneous and heterogeneous settings.
To differentiate our work, we show how these algorithms operate in Table 5.
For Minibatch SGD, there is no local update and each client computes K stochastic gradients at the same point x r .
FL and SL make K times more updates than Minibatch SGD.
However, models in
FL are training in parallel, while models in SL are in sequence.
notations in Section 2.
ξ ( r,k ) i ∼ D
i denotes one random sample from client i for the k + 1 -th local update in the r -th round.
[Table]
C S UMMARY OF THEORIES
There are two methods to give the convergence of SL: (i) bounding the progress of all clients in one round; (ii) bounding the progress of one client in one round.
The second method can be given based on the techniques of FL (Khaled et al., 2020;
Karimireddy et al., 2020; Wang et al., 2020) directly.
However, it only converges to a neighborhood around the stationary point of the global function.
This case is similar to the biased SGD (Ajalloeian & Stich, 2020).
It is intuitive to get the conclusion since the local stochastic gradient ∇ f i ( x ; ξ i ) ( ξ i ∼ D i ) generated by local data of any client i is a biased gradient estimator of the global gradient ∇ f ( x ) of the global function.
For the first method — bounding the progress of all clients in one round, our main contribution, shows that
SL can converge to the stationary point of the global function.
The main theories are summarized in
Table
[Table] [Table]
ROOF OF RESULTS D. B ASIC TECHNICAL LEMMAS AND N OTATIONS
Lemma 1.
x 1 , . . .
, x N are N vectors, then
[Equation]
Lemma 2 (Jensen’s inequality) .
For any convex function f and any vectors x 1 , . . .
, x N we have
[Equation]
As a special case with f ( x ) =
∥ x ∥ 2 , we obtain
[Equation]
Lemma 3.
Suppose { A k } Tk =1 is a sequence of random matrices and E
[ A k | A k − 1 , A k − 2 , . . .
, A 1 ] = 0 , ∀ k .
Then,
[Equation]
Proof.
This is the Lemma 2 of Wang et al.
[Equation]

[Equation]
Assume i < j .
Then, using the law of total expectation,
[Equation]
Lemma 4 (Bounded Drift) .
For any local learning rate satisfying η l ≤ 1 2 NKL , the client drift caused by local updates is bounded, as given by:
[Equation]
Proof.
This proof is based on the proof of Theorem 1: Convergence of Surrogate Objective of Wang et al.
Considering
[Equation]
we have
[Equation]
2 iη 2 l E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88) t
=
[ g i ( x ( r,t ) i ) − ∇ f i ( x ( r,t ) i )] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ i − 1 (cid:88) s =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K − 1 (cid:88) t =
[ g s ( x ( r,t ) s ) − ∇ f s ( x ( r,t ) s )]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 iη 2 l E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88) t = ∇ f i ( x ( r,t ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + i − 1 (cid:88) s = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K − 1 (cid:88) t = ∇ f s ( x ( r,t ) s ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (21)
Applying Lemma 3 to the first term on the right hand side in Eq.
(21) and Jensen’s Inequality to the second term respectively, we get
[Equation]
2 iKη 2 l i (cid:88) s =
K − 1 (cid:88) t =
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f s ( x ( r,t ) s ) (cid:13)(cid:13)(cid:13) 2 (cid:21) (22) (7) ≤ 2 i 2 Kη 2 l σ 2 + 4 iKη 2 l i (cid:88) s =
K − 1 (cid:88) t =
E (cid:104) ∥∇ f s ( x r ∥ 2 (cid:105) + 4 iKη 2 l i (cid:88) s =
K − 1 (cid:88) t =
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f s ( x ( r,t ) s ) − ∇ f s ( x r ) (cid:13)(cid:13)(cid:13) 2 (cid:21) (23)
Asm. 1 ≤ 2 i 2 Kη 2 l σ 2 + 4 iKη 2 l
i (cid:88) s =
K − 1 (cid:88) t =
E (cid:104) ∥∇ f s ( x r ) ∥ 2 (cid:105) + 4 iKη 2 l Li (cid:88) s =
K − 1 (cid:88) t =
E (cid:20)(cid:13)(cid:13)(cid:13) x
( r,t ) s − x r (cid:13)(cid:13)(cid:13) 2 (cid:21) (24) ≤ 2 i 2 Kη 2 l σ 2 + 4 iK 2 η 2 l
N (cid:88) s =
E (cid:104) ∥∇ f s ( x r ) ∥ 2 (cid:105) + 4 iKη 2 l L
N (cid:88) s =
K − 1 (cid:88) t =
E (cid:20)(cid:13)(cid:13)(cid:13)
[Equation]
Summing up E (cid:104)
∥ x ( r,k ) i − x r ∥ 2 (cid:105) over i and k , we get
N
[Equation]
Term E is equivalent to (cid:80) Ni =1 (cid:80) K − 1 k =
E [ ∥ x ( r,k ) i − x r ∥ 2 ] , so we can rearrange the equation and get:
[Equation]
[Equation]
Using 4 N 2 K 2 η 2 l L 2 < 1 and dividing both sides by it yields the claim of Lemma 4.
D. P ROOF OF
T HEOREM
Proof.
Beginning with Assumption 1, we have
[Equation] [Equation]
From Algorithm 1, we know the global model update in round r can be written as:
[Equation]
For the expectation on x r , we get
E [ f ( x r +1 )] − f ( x r ) ≤ E (cid:34)(cid:42) ∇ f ( x r ) , − η g η l
N (cid:88) i =
K − 1 (cid:88) k
= g i ( x ( r,k ) i ) (cid:43)(cid:35)
+ L
E  
(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)
η g η l
N (cid:88) i =
K − 1 (cid:88) k
= g i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) 2   (32) = − Nη g η l
K − 1 (cid:88) k
=
E (cid:34)(cid:42) ∇ f ( x r ) , 1 N
N (cid:88) i = ∇ f i ( x ( r,k ) i ) (cid:43)(cid:35)
+ L
η 2 g η 2 l E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where we use E [ g i ( x )]
= ∇ f i ( x ) in the equality (see Assumption 2).
For the second term on the right hand side (RHS) of Eq.
(33), we have:
[Equation]
k =
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (7)
[Equation]
2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i =
K − 1 (cid:88) k
= (cid:104) g
i ( x ( r,k ) i ) − ∇ f i ( x ( r,k ) i ) (cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2  
[Equation]
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i =
K − 1 (cid:88) k
=
[Equation]
f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (34) (9) , Lem
[Equation] [Equation]
2 E (cid:34)
N
N (cid:88) i =
K − 1 (cid:88) k
= (cid:13)(cid:13)(cid:13) g
i ( x ( r,k ) i ) − ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:35) + 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i =
K
[Equation]
1 (cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)
[Equation]
  .
Note in Eq.
(35) that we apply Jensen’s Inequality first before Lemma 3 to the first term of RHS, since the data across clients are non-IID.
However, if the data across clients are IID, we can get a tighter bound of NKσ 2 .
Then plugging Eq.
(36) into Eq.
(33), we have:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88)
[Equation]
K −
[Equation]
(cid:88) k
=
[Equation]
r
[Equation]
k )
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   +
[Equation]
[Equation]
38) where we use the fact that 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a − b ∥ 2 in the last equation.
Note that −
[Equation]
N (cid:88) i =
K − 1 (cid:88) k
= ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (9) ≤ −
N η g η l
K − 1 (cid:88) k
=
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + Lη 2 g η 2 l K
K − 1 (cid:88) k
=
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   = − 1 2 N η g η l (1 − 2 NKLη g η l )
K − 1 (cid:88) k
=
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (39) and E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i = ∇ f i ( x ( r,k ) i )
− ∇ f ( x r ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where the first equality of Eq.
(40) results from the fact that ∇ f ( x r )
= 1 N (cid:80) Ni =1 ∇ f i ( x r ) .
By plugging Eq.
(39) and Eq.
(40) into Eq.
and
using 2 NKη g η l L ≤ 1 , we get
[Equation]
Then we use Lemma 4:
[Equation]
where D = 4 N 2 K 2 L 2 η 2 l .
Then using D ≤ 1 2 B 2 +1 and B ≥ 1 , we get
[Equation]
Taking unconditional expectation, rearranging the terms and then averaging the above equation over r = { 0 , · · · , R − 1 } , we have
[Equation]
Using the fact that E (cid:13)(cid:13)
[Equation]
R − 1 r =
E ∥∇ f
( x r ) ∥ 2 where ¯ x R = 1 R (cid:80) R − 1 r =0 x r , we get the
Eq.
Finally, we summarize the constraints:
[Equation]
where the last inequality is from Lemma 4.
The overall constraint is given as:
[Equation]
Now we complete the proof of Theorem 1.
Corollary 2.
Choose NKη g η l = 1 √ R and apply the result of Theorem 1.
For sufficiently large R , it holds that
[Equation]
where F := f ( x 0 ) − f ( x ∗ ) , R is the total rounds, ¯ x R = 1 R (cid:80) R − 1 r =
x r is the averaged global model over the R rounds.
Corollary 3.
Apply the result of Theorem 1.
There exits η l , such that
[Equation]
where F := f ( x 0 ) − f ( x ∗ ) , R is the total rounds, ¯ x R = 1 R (cid:80) R − 1 r =
x r is the averaged global model over the R rounds.
Proof.
Applying Lemma 2 (sub-linear convergence rate) of Karimireddy et al.
(2020) to Eq.
(44), we get the the claim of this corollary.
D. P ROOF OF T HEOREM
Theorem 2.
Let Assumptions 1, 2 and 3 hold.
Suppose that the local learning rate satisfies η l ≤
[Equation]
For Algorithm 1, it holds that
[Equation]
where ¯ x R = 1 R (cid:80) R − 1 r =0 x r is the averaged global model over the R rounds.
Proof.
This is almost the same as Theorem 1 of Wang et al.
(2020), we reproduce here for convenience of comparison between FL and SL.
The proof is similar to that of Theorem 1.
D. P ROOF OF T HEOREM
Here we use Assumption 4 to replace Assumption 3, one stronger assumption (than Assumption 3) used in Lian et al.
Assumption 4.
There exist constants G ≥
such that
[Equation] [Equation]
where i is uniformly sampled from { 1 , . .
. , N } .
In the IID case, G = 0 .
Theorem 3 (Progress of one client in one round) .
Let Assumptions 1, 2 and 3 hold.
Suppose that the local learning rate satisfies η l ≤
[Equation]
√ 5 KL .
For Algorithm 1, it holds that
[Equation]
where ¯ x R = 1 R (cid:80) R − 1 r =0 x r is the averaged global model over the R rounds.
Proof.
Different from Theorem
[Equation]
we bound the progeress of one client in one round.
Beginning with Assumption
[Equation]
we have:
[Equation]
From Algorithm
[Equation]
we know the local update of client i in round r can be written as:
[Equation]
For the expectation on x ( r, 0) i , we get
[Equation]
k = g i ( x ( r,k ) i ) (cid:43)(cid:35) +
[Equation]
 (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
η l
K − 1 (cid:88) k
= g i ( x ( r,k ) i ) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   (56) =
− η l
K − 1 (cid:88) k
=
[Equation]
(cid:104)(cid:68)
[Equation] [Equation] [Equation] [Equation]
[Equation]
[Equation]
[Equation]
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
−
[Equation]
(cid:88)
[Equation]
[Equation]
[Equation]
( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
  , (57) where we use E [ g i ( x )] = ∇ f i ( x ) in the equality (see Assumption 2).
For the second term on the right hand side (RHS) of Eq.
(57), we have:
[Equation]
[Equation] [Equation]
2 
[Equation] [Equation]
2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
−
[Table]
=
[Table]
g
[Table]
x
[Table]
,k
[Table]
i ) − ∇ f
[Table]
( x
[Table] [Table]
i ) (cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
 + 2 E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K − 1 (cid:88) k
=
[Equation]
f
[Equation] [Equation]
( r
[Equation]
) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (58)
Lem.
3 ≤ 2 E (cid:34) K − 1 (cid:88) k
= (cid:13)(cid:13)(cid:13) g
i ( x ( r,k ) i ) −
[Equation]
f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:35) + 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K − 1 (cid:88) k
= ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
  (59)
[Equation]
(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)
K
[Equation]
1 (cid:88) k
=
[Equation]
f i ( x ( r,k ) i ) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) 2   .
Then plugging Eq.
(60) into Eq.
(57), we have:
[Equation]

[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
[Equation]
(cid:88)
[Equation]
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]

[Equation] [Equation]
[Equation]
) − ∇ f (
[Equation]
[Equation]
) (cid:13)(cid:13)(cid:13)
[Equation]
(cid:21) + Lη 2 l E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K
[Equation]
(cid:88) k
=
[Equation] [Equation]
( r
[Equation] [Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
  +
[Equation]
, (62) where we use the fact that 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a − b ∥ 2 in the last equation.
Note that −
[Equation]
K −
[Equation]
(cid:88) k
=
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) + Lη 2 l E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K − 1 (cid:88) k
= ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (9) ≤ −
η l
K − 1 (cid:88) k
=
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) + Lη 2 l K K − 1 (cid:88) k
=
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) = − 1 2 η l (1 − 2 KLη l )
K − 1 (cid:88) k
=
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r,k ) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) (63) and E (cid:20)(cid:13)(cid:13)(cid:13)
∇ f i ( x ( r,k ) i )
− ∇ f ( x ( r, 0) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) =
2 E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r,k ) i )
− ∇ f i ( x ( r, 0) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) + 2 E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i ( x ( r, 0) i )
− ∇ f ( x ( r, 0) i ) (cid:13)(cid:13)(cid:13) 2 (cid:21) (64)
Asm. 1 ≤ 2 L 2 E (cid:20)(cid:13)(cid:13)(cid:13) x
( r,k ) i − x ( r, 0) i (cid:13)(cid:13)(cid:13) 2 (cid:21) + 2 G 2 .
By plugging Eq.
(63) and Eq.
(65) into Eq.
(62) and using 2 Kη l L ≤ 1 , we get
E (cid:104) f
( x ( r, 0) i +1 )
− f ( x ( r, 0) i ) (cid:105) ≤ − 1 2 Kη l (cid:13)(cid:13)(cid:13) ∇ f ( x ( r, 0) i )
(cid:13)(cid:13)(cid:13) 2 + L 2 η l
K − 1 (cid:88) k
[Equation] [Equation]
Then using
[Equation]
we can get
[Equation]
where D = 4 K 2 L 2 η 2 l .
Then using D ≤ 15 ( 1 1 − D ≤ 54 ) and B ≥
[Equation]
we get
[Equation]
Taking unconditional expectation
[Equation]
rearranging the terms and then averaging the above equation over i = { 1 , · · · , N } , r = { 0 , · · · , R − 1 } , we have
[Equation]
Using the fact that E (cid:13)(cid:13) ∇ f (¯ x R ) (cid:13)(cid:13) 2 ≤ 1 R (cid:80) R − 1 r =
E ∥∇ f
( x r ) ∥ 2 where ¯ x R = 1 R (cid:80) R − 1 r =0 x r , we get the
Eq.
Finally, we summarize the constraints:
[Equation]
The overall constraint is given as:
[Equation]
Now we complete the proof of Theorem
[Equation]
D. E XTREME CASES
Theorem 1 recovers the convergence of SGD when N = 1 and K = 1 .
Let us focus on the proof of Lemma 4.
When N = 1 and K = 1 , the client drift will reduce to:
[Equation]
where x ( r, 0)= x r (see Algorithm 1).
Thus the client drift error of Eq.
(5) will be removed, which recovers the result of SGD (Bottou et al., 2018).
E M ORE EXPERIMENTAL DETAILS
Platform.
We train LeNet-5 on MNIST and Fashion-MNIST with Nvidia GeForce RTX
Ti,
VGG-11 on CIFAR-10 with Nvidia 3090 Ti.
The algorithms are implemented by PyTorch.
We use the random seed “1234” by default.
We use vanilla SGD algorithm with momentem = 0.9 and weight decay = 1e-4 as He et al.
The detailed information of the models and other information can be found in our code.
E. M ORE RESULTS OF SL
This section is complimentary to Section 5.2 to study the factors that affects the performance of SL.
We report the details of the experiments, such as η l , η g , b .
Effect of data heterogeneity.
[Figure]
Effect of K .
[Figure]
Effect of η g .
The experimental results on Fashion-MNIST and CIFAR-10 are shown in Figure 7.
We can see that η g can be helpful in some cases, especially when η l is small.
However, there is still a big gap between theory and practice.
Further research is required.
E. M ORE COMPARISONS BETWEEN FL AND SL IN CROSS - DEVICE SETTING
The learning rates of FL and SL are selected from { 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1 } .
We report the overall results of comparisons between FL and SL on MNIST, Fashion-MNIST and CIFARdatasets with different learning rates in Table 7, Table 8 and Table 9 respectively.
Table 2 in the main body are based on these three tables here.
“L-M”, “L-M” and “V-10” denote LeNet-5 on MNIST,
LeNet-5 on Fashion-MNIST and VGG-11 on CIFAR-10 respectively.
We highlight the “best” test accuracy among all chosen learning rates with blue for FL and red for SL.
We underline the test accuracy of the “threshold” learning rate with blue for FL and red for SL.
[Table]
[Figure] [Figure]
[Table] [Table]
[Table]