P ROG F ED : E FFECTIVE , C OMMUNICATION , AND C OMPUTATION E FFICIENT F EDERATED L EARNING BY P ROGRESSIVE T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data.
However, training is resource-intensive for edge devices, and limited network bandwidth is often the main bottleneck.
Prior work often overcomes the constraints by condensing the models or messages into compact formats, e.g., by gradient compression or distillation.
In contrast, we propose ProgFed, the ﬁrst progressive training framework for efﬁcient and effective federated learning.
It inherently reduces computation and two-way communication costs while maintaining the strong performance of the ﬁnal models.
We theoretically prove that ProgFed converges at the same asymptotic rate as standard training on full models.
Extensive results on a broad range of architectures, including CNNs (VGG, ResNet, ConvNets) and U-nets, and diverse tasks from simple classiﬁcation to medical image segmentation show that our highly effective training approach saves up to 20% computation and up to 63% communication costs for converged models.
As our approach is also complimentary to prior work on compression, we can achieve a wide range of trade-offs, showing reduced communication of up to 50 × at only 0 .
1% loss in utility.
I NTRODUCTION
Federated Learning (FL) (McMahan et al., 2017) has led to remarkable advances in the development of extremely large machine learning systems.
Federated training methods allow multiple clients (edge devices) to jointly train a global model without sharing their private data with others.
Despite the progress, training methods in FL still suffer from high communication and computational costs, as edge devices are often equipped with limited hardware resources and limited network bandwidth.
Prior literature has studied various compression techniques to address the computation and communication bottlenecks.
We can divide these methods into three main categories (see also Table 1):
(i) Compression techniques that represent gradients (or parameters) with fewer bits to reduce communication costs.
Prominent examples are quantization (Alistarh et al., 2017; Lin et al., 2018; Fu et al., 2020) or sparsiﬁcation (Stich et al., 2018; Koneˇcn`y et al., 2016).
(ii)
Model pruning techniques that identify (much smaller) sub-networks within the original models to reduce computational cost at inference (Li & Wang, 2019; Lin et al., 2020).
And (iii) knowledge distillation (Hinton et al., 2015) techniques, that allow the server to distill the knowledge from the clients with hold-out datasets (Li & Wang, 2019; Lin et al., 2020).
However, these methods can sometimes degrade performance.
Progressive learning (Karras et al., 2018) is a well-known technique that has been used in image generation to stabilize the training.
The primary idea is to ﬁrst train the shallower layers on simpler tasks (e.g., images with lower resolution) and gradually grow the network to tackle more complicated tasks (e.g., images with higher resolution).
By applying progressive training in a federated learning setting, we can beneﬁt from highly reduced resource demands (the computation and communication load is drastically reduced when training shallow models).
By growing the models to their original size, we can recover (and often surpass) the accuracies reached by standard training.
Moreover, we observe that progressive training facilitates the learning process and stabilizes training.
Despite the appealing features, no previous study has systematically investigated exploiting progressive learning to reduce the costs in federated learning.
[Table]
In this work, we propose ProgFed, the ﬁrst progressive learning framework that reduces both communication and computation costs in FL.
In our approach, we divide the model into several overlapping partitions and introduce lightweight local monitoring heads to guide the training of the submodels.
During training, the model capacity is gradually increased, until it reaches the full model of interest.
This, of course, reduces the computation and communication costs, since the shallow sub models have much fewer parameters than the full model.
We show theoretically that
ProgFed training converges at the same asymptotic rate as standard training on full models.
Due to the nature of progressive learning, our method can reduce computational overheads and provides two-way communication savings (both from server-to-client and client-to-server directions).
We demonstrate that our method can reach impressive performance (matching the baselines) using much fewer communication.
ProgFed is compatible with classical compression—including quantization and sparsiﬁcation—which can be applied on top of progressive training.
We experimentally show that combing these features enables a higher compression ratio and may motivate more advanced compression schemes based on progressive learning.
We summarize our main contributions as follows.
• We propose ProgFed, the ﬁrst progressive learning framework to reduce the training resource demands (computation and two-way communication).
We theoretically prove that ProgFed converges at the same asymptotic rate as standard training the full model.
• We conduct extensive experiments on various datasets (CIFAR-10/100, EMNIST and BraTS) and architectures (VGG-16/19, ResNet-18/152, ConvNets, 3D-Unet) to show that with the same number of epochs, our method can save around 25% computation cost, up to 32% two-way communication costs in federated classiﬁcation, and 63% in federated segmentation without sacriﬁcing performance.
• In addition, our method helps the models save around 2 × fewer communication costs in classiﬁcation and 6 .
5 × fewer costs in U-net segmentation to achieve practicable performance ( ≥ 98% of the best baseline).
This is beneﬁcial for combating limited training budgets in federated learning.
Lastly, we show that our method complements classical compression and appears robust against compression errors.
It permits a higher compression ratio and may motivate more advanced compression schemes based on progressive learning.
With these combined techniques, we are able to reduce communication of up to 50 × at only 0 .
1% loss in utility.
R ELATED W ORK
In this section, we review work on cost reduction and leave progressive learning in Section D.
Message compression.
Fruitful literature has studied message compression (e.g. on gradients or model weights) to reduce the communication costs in distributed learning.
The ﬁrst category focuses solely on client-to-server compression (Alistarh et al., 2017; Wen et al., 2017; Lin et al., 2018;
Bernstein et al., 2018; Stich et al., 2018; Koneˇcn`y et al., 2016; Karimireddy et al., 2019; Fu et al., 2020; Stich, 2020).
To name a few, Koneˇcn`y et al.
(2016) reduce the costs by sending sparsiﬁed gradients and compressing them with probabilistic quantization.
Alistarh et al.
Wen et al.
(2017) prove the convergence of probabilistic quantized SGD.
SignSGD
(Bernstein et al., 2018) signiﬁcantly compresses the gradients with only one bit, while Karimireddy et al.
ﬁx the biased nature of SignSGD with an error-feedback mechanism and generalize to other compression schemes.
Compression on server-to-client communication has been shown non-trivial and attracted many recent focuses (Yu et al., 2019;
Tang et al., 2019; Liu et al., 2020; Philippenko & Dieuleveut, 2020).
Instead of dedicated designs, our method inherently reduces two-way communication costs and complements existing methods as we will show in Section 4.
[Figure]
Model pruning.
Model pruning removes redundant weights to address the resource constraints (Mozer & Smolensky, 1989; LeCun et al., 1990; Frankle & Carbin, 2019; Lin et al., 2019).
There are two categories: unstructured and structured pruning.
Unstructured methods prune individual model weights according to certain criteria such as Hessian of the loss function (LeCun et al., 1990; Hassibi & Stork, 1993) and small magnitudes (Han et al., 2015).
However, these methods cannot fully accelerate without dedicated hardware since they often result in sparse weights.
In contrast, structured pruning methods prune channels or even layers to alleviate the issue.
They often learn importance weights for different components and only keep relatively important ones (Liu et al., 2017; Yu et al., 2018;
Mohtashami et al., 2021; Li et al., 2021).
Despite the efﬁciency, model pruning usually happens at inference and does not reduce training costs while our method achieves computation- and communication-efﬁciency even during training.
Model distillation.
Another line of research for model compression is model distillation (Bucilu ˇ a et al., 2006; Hinton et al., 2015), which requires a student model to mimic the behavior of a teacher model (Polino et al., 2018; Sun et al., 2019).
Recent work has investigated transmitting logits rather than gradients (Li & Wang, 2019; Lin et al., 2020; He & Annavaram, 2020; Choquette-Choo et al., 2020), which signiﬁcantly reduces communication costs.
However, these methods either require additional query datasets (Li & Wang, 2019; Lin et al., 2020) or cannot enjoy the merit of datasets from different sources (Choquette-Choo et al., 2020).
In contrast, our work reduces the costs while retaining the dataset efﬁciency.
P ROG F ED
Federated learning is a distributed learning framework in which a huge amount of clients train local models independently, and a central server aggregates the client updates to learn a global model.
Motivated by progressive learning, we propose ProgFed that progressively expands the network from a shallower one to the complete model.
Our method is effective and efﬁcient for multi-client training while we consider only one client in theoretical analysis for conciseness.
The proposed model splitting and progressive growing are illustrated in Figure 1, and the federated optimization scheme is summarized in Algorithm 1.
We now present the proposed method in detail below.
3. P ROPOSED M ETHOD
Model Structure.
We now describe the proposed training method.
For a given a machine learning model M , i.e. a function M ( · , x ): R n → R k that maps n -dimensional input to k logits for parameters (weights) x ∈ R d , we assume that the network can be written as a composition of blocks (feature extractors)
E i along with a task head G S , namely,
[Equation]
Note that the E i ’s could denote e.g., a stack of residual blocks or simply a single layer.
The learning task is solved by minimizing a loss function of interest L : R k
→ R (e.g., cross-entropy) that maps the predicted logits of M to a real value, i.e. minimization of f ( x ) :
= L ◦ M ( x ) .
Progressive Model Splitting.
To achieve progressive learning, we ﬁrst divide the network M into
S stages, denoted by M s , for s ∈ { 1 , . . .
, S } associated with the split indices.
We additionally introduce local supervision heads for providing supervision signals.
Formally, we deﬁne
[Equation]
where G s is a newly introduced head.
Each head G s , for s < S consists of only a pooling layer and a fully-connected layer in our experiments for feed-forward networks.
The motivation is that simpler heads may encourage the feature extractors E i to learn more meaningful representations.
Note that the sub-model M s : R n → R k produces the same output size as the desired model M ; therefore, its corresponding loss f s ( x s ) :
= L ◦ M s ( x s ) can be trained with the same loss criterion
L as the full model.
Training of Progressive Models.
We propose to train each sub-model M s for T s iterations (a certain fraction of the total training budget) and gradually grow the network from M 1 to M S = M .
When growing the network from stage s to s
+ 1 , we pass the corresponding parameters of the pretrained blocks E i , i ≤ s , to the next model M s +1 and initialize its blocks E s +1 and G s +1 with random weights.
Once the progressive training phase is completed, we continue training the full model M in an end-to-end manner for the remaining iterations.
The length T s of each progressive training phase is a parameter that could be ﬁne-tuned individually for each stage (depending on the application) for best performance.
However, as a practical guideline that we adopted for all experiments in this paper, we found that denoting roughly half of the total number of training iterations
T to progressive training, and setting T s = T 2 S for s < S , T S = T ( S +1)2 S , such that T = (cid:80) Ss =1 T s , works well across all considered training tasks.
Extension to U-nets.
In addition to feed-forward networks (Figure 1(a)), we show that our method can generalize to U-net (Figure 1(b)).
U-net typically consists of an encoder and a decoder.
Unlike feed-forward networks, the encoder sends both the ﬁnal and intermediate features to the decoder.
Therefore, we propose to grow the network from outer to inner layers as shown in Figure 1(b) and retain the original output heads as G i .
We refer to the strategy as the Symmetric strategy.
In contrast, we propose another baseline, the Asymmetric strategy, which adopts the full encoder at the beginning and gradually grows until it reaches the full decoder.
For this strategy, we also adopt several temporal heads for earlier training stages.
As we will show in Section 4.3, the Symmetric strategy signiﬁcantly outperforms the Asymmetric strategy, which supports the notion of progressive learning.
Practical considerations.
We empirically observe that learning rate restart (Loshchilov & Hutter, 2017) clearly facilitates training in the centralized setting.
This is because sub-models may overﬁt the local supervision while learning rate restart helps the sub-models with the newly added layers escape from the local minima and quickly converge to a better minima.
On the other hand, warmup (Goyal et al., 2017) for the new layers plays an important role in federated learning.
Model weights often take a longer time to converge in federated learning, which makes the newly added layers introduce more noise to the previous layers.
With warm-up, the new layers recover the performance without affecting previous layers.
3. T HEORETICAL A NALYSIS
In this section we prove that progressing training converges to a stationary point at the rate O (cid:0) 1 (cid:15) 2 (cid:1) , i.e. with the same asymptotic rate as SGD training on the full network.
For this, we extend the analysis from (Mohtashami et al., 2021) that analyzed the training of partial subnetworks.
However, in our case the networks are not subnetworks, M s (cid:54)⊂ M s +1 (as the head is not shared), and we need to extend their analysis to progressive training with different heads.
Notation.
We denote by x
s the parameters of f s , s ∈ { 1 , . .
. , S } and abbreviate x S = x ∈ R d for convenience.
For s ≤ i ≤ S let x i | E s and ∇ f i ( x i ) | E s denote the projection of the parameter x i and gradient ∇ f i ( x i ) to the dimensions corresponding to the parameters of E 1 to E s (without parameter of E s +1 to E i and without head G i ).
In iteration t , the progressive training procedure updates the parameters of model f s , s = min( S, (cid:100)
tT s (cid:101) ) .
For convenience, we do not explicitly write the dependency of s on t below, and use the shorthand x st to denote the corresponding model at timestep t .
We further deﬁne x t such that x t | E s = x
t s | E s and x t | E (cid:123) s
= x 0 | E (cid:123) s on the complement.
[Equation] [Equation]
∆
[Equation]
21: end for
Assumption 1 ( L -smoothness) .
The function f : R d → R is differentiable and there exists a constant L > 0 such that
[Equation]
We assume that for every input x s , we can query an unbiased stochastic gradient g s ( x s ) with E [ g s ( x s )]
= ∇ f s ( x s ) .
We assume that the stochastic noise is bounded.
Assumption 2 (bounded noise) .
There exist a parameter σ 2 ≥ 0 such that for any s ∈ { 1 , . . .
, S } :
[Equation]
The progressive training updates x st +1 = x
ts − γ t g s ( x st ) with a SGD update on the model x st .
With the two assumptions above, which are standard in the literature, we prove the convergence of submodels M s as well as the model of interest M .
Theorem 1.
Let Assumptions 1 and 2 hold, and let the stepsize in iteration t be γ t = α t γ with
[Equation]
(cid:111) .
Then it holds for any (cid:15) > 0 ,
• 1 T (cid:80) T − 1 t =
α 2 t (cid:13)(cid:13) ∇ f s ( x st ) |
E s (cid:13)(cid:13) 2 < (cid:15) ,
after at most the following number of iterations T:
[Equation]
5) • Let q :=
max t ∈ [ T ] (cid:16) q t :=
(cid:107)∇ f ( x t ) (cid:107) α t (cid:107)∇ f s ( x st ) | Es (cid:107) (cid:17) , then 1 T (cid:80) T − 1 t =0 (cid:107)∇ f ( x t ) (cid:107) 2 < (cid:15) after at most the following iterations T :
[Equation]
where F 0 :
= f
( x 0 ) − (min x f ( x )) .
Theorem 1 shows the convergence of the full model M .
The convergence is controlled by two factors, the alignment factor α t and the norm discrepancy q t .
The former term measures the similarity between the corresponding parts of the gradients computed from the sub-models and the full model (note that α t ≡ 1 in the last phase of training on f S = f ).
The latter term q measures the magnitude discrepancy (in Figure 7 we display the evolution of q t during training for one example task, note that q t = 1 in the last phase of training).
We would like to highlight that the convergence criterion in the ﬁrst statement is lower bounded by the average gradient in the last phase of the training, 12 · 12 T (cid:80) T − 1 t = T/
(cid:107)∇ f ( x t ) (cid:107) 2 ≤ 1 T (cid:80) T − 1 t =
α 2 t (cid:107)∇ f s ( x st ) | E s (cid:107) 2 (this is due to our choice of the length of the phases, with T S ≥ T/ 2 ).
This means, that progressive training will provably require at most twice as many iterations but can reach the performance of SGD training on the full model with much cheaper per-iteration costs.
E XPERIMENTS 4. S ETUP
We now describe the main implementation details and provide supplementary details in Section B.
Datasets, tasks
and models.
We consider four datasets: CIFAR-10 (Krizhevsky et al., 2009),
CIFAR-100 (Krizhevsky et al., 2009), EMNIST (Cohen et al., 2017), and BraTS (Menze et al., 2014; Bakas et al., 2017; 2018).
The former three are for image classiﬁcation, while the last one is a medical image dataset for tumor segmentation.
We conduct centralized experiments for analyzing the basic properties of our method while considering practical applications in federated settings.
For the centralized settings, we train VGG-16, VGG-19 (Simonyan & Zisserman, 2014), ResNet-18, and ResNet-152 (He et al., 2016) on CIFAR-100 (100 clients, IID).
For the federated settings, we train ConvNets on CIFAR-10 and EMNIST (3400 clients, non-IID), ResNet-18 on CIFAR-100 ( clients, non-IID), and 3D-Unet (Sheller et al., 2020) on the BraTS dataset (10 clients, IID).
Note that we follow Hsieh et al.
(2020) to replace batch normalization in ResNet-18 with group normalization.
Implementation.
We implement all settings with Pytorch (Paszke et al., 2019).
For the experiments in the centralized settings, we implement models based on DeVries & Taylor (2017), where we run all experiments for 200 epochs and decay the learning rates in { 60, 120, 160 } epochs by a factor of 0.1.
We additionally adopt warm-up (Goyal et al., 2017) and learning rate restart (Loshchilov & Hutter, 2017) in our method to better ﬁt in progressive learning.
For federated classiﬁcation, we follow federated learning benchmarks in (McMahan et al., 2017; Reddi et al., 2021) to implement
CIFAR-10, CIFAR-100, and EMNIST, respectively.
For federated tumor segmentation, we follow
Sheller et al.
(2020) for the settings and data splits.
We set S = 3 for EMNIST and S = 4 for all the other datasets and T s as the practical guideline described in Section 3.
We adopt 5 andwarm-up epochs for federated EMNIST and federated CIFAR-100, respectively.
We summarize the parameters (and number of clients) in Table 6.
4. C OMPUTATION E FFICIENCY
We ﬁrst analyze the computation efﬁciency of our method in the centralized setting (where all data is available on a single device) to study the effect of the progressive training in isolation, before moving to the federated use cases.
We average the outcomes over three random seeds and we consider four architectures on CIFAR-100, including VGG-16, VGG-19, ResNet-18, and
ResNet-152.
As shown in Table 2, our method performs comparably to the baselines (that train on the full model) after 200 epochs while consuming fewer ﬂoating-point operations per second (FLOPs) and training wall-clock time.
To analyze the efﬁciency, we report the performance when consuming different levels of costs.
Figure 2 shows that our method (orange lines) consistently lies above end-to-end training on the full model (blue lines), meaning that our method consumes fewer computation resources to improve the models.
Moreover, we visualize { 98% , 99% , 99 .
95% , best } of the performance of the converged baseline (analysis with a larger range is presented in Figure 12).
Figure 3 indicates that our method improves computation efﬁciency across architectures.
In the best case, our method can accelerate training up to 7 × faster when considering limited computation budgets.
We also observe that VGG models improve more than ResNets.
A possible reason might be that due to local supervision, sub-models enjoy larger gradients compared to end-to-end training, while it rarely beneﬁts ResNets since skip-connections could partially avoid the problem
[Table]
4. C OMMUNICATION E FFICIENCY
[Table]
We experiment in the federated setting to verify the communication efﬁciency of our method.
In particular, we consider classiﬁcation tasks on three datasets, EMNIST, CIFAR-10, and CIFAR-100, and tumor segmentation tasks on the BraTS dataset.
We follow the standard protocol as described in Section 4.1 to train the models and average the results over three random seeds.
Results in Table 3 indicate that our method achieves comparable results on EMNIST and outperforms the baselines on all the other datasets.
In addition, our method saves 20% to 30% two-way communication costs in classiﬁcation and up to 63% costs in segmentation.
The result simultaneously conﬁrms the effectiveness and efﬁciency of our method.
We compare the performance at different communication costs in Figure 5.
We observe that our method is communication-efﬁcient over every cost budget, especially when the model parameters are not evenly distributed across sub-models.
For instance, 3D-Unet has most of its parameters in the middle part of the model, making our Symmetric update strategy extremely efﬁcient.
On the other hand, the Asymmetric strategy shows marginal improvement since it starts from the heaviest portion of the model.
The ﬁnding aligns with the motivation of progressive learning: learning from simpler models might facilitate training.
More analysis on segmentation is in Section C.4.
[Table]
[Figure]
[Table]
Lastly, we analyze the cost reduction when achieving { 98% , 99% , 99 .
5% , best } of the performance of the converged baseline.
This experiment studies the behavior of our method when only granted limited budgets.
Figure 4 (and Figure 13 in the appendix that displays a larger range) presents that except for the Asymmetric strategy, our method improves communication efﬁciency across all datasets.
In particular, it achieves practicable performance with 2x fewer costs in classiﬁcation and up to 6.5x fewer costs in tumor segmentation.
We also observe that the communication efﬁciency improves more when considering lower budgets.
This property beneﬁts when the time and communication budgets are limited (McMahan et al., 2017).
4. C OMPATIBILITY
We show that our method complements classical compression techniques including quantization and sparsiﬁcation.
We train several ResNet-18 on CIFAR-100 in the federated setting and apply linear quantization and sparsiﬁcation following Koneˇcn`y et al.
Speciﬁcally, we consider 8 bits,bits, and 2 bits for quantization (denoted by LQ-X), 25% and 10% for sparsiﬁcation (denoted by
SP-X), and their combinations.
Table 4 demonstrates the results of our method and the baseline equipped with various compression techniques.
Our method clearly outperforms the baselines in all settings.
It indicates that our method is more robust against compression errors, compatible with classical techniques, and thus permits a higher compression ratio.
In addition, we visualize { 50% , 80% , 90% , 99% , 99 .
5% , best } of relative performance against communication cost reduction in Figure 6.
We observe that our method is more efﬁcient across all percentages in every pair (Ours vs. Baseline, plotted in the same color).
Besides, the baseline fails to achieve comparable performance in many settings, e.g., the ones with quantization, while our
[Table]
method retains comparable performance even with high compression ratios.
Interestingly, even with additional compression, our method still facilitates learning at earlier stages.
For example, OursLQ8+SP25 achieves comparable performance around 50x faster than the baseline, 60x faster to achieve 80%, and more than 200x faster to achieve 50% of performance.
Overall, these properties grant our method to adequately approach limited network bandwidth and open up the possibility of more advanced compression schemes.
4. A NALYSIS OF P ROG F ED
Effect of norm discrepancy.
As discussed in Section 3.2, the convergence rate of the full model is controlled by norm discrepancy, namely q .
As q t approaches 1, the convergence rate will be closer to the convergence speed of the sub-models.
We empirically evaluate the norm discrepancy on CIFAR-100 with ResNet-18 in the centralized setting.
Figure 7 shows that the norm discrepancy decreases as the sub-models gradually recover the full model.
It suggests that spending too much time on earlier stages may hurt the convergence speed while it offers a higher compression ratio.
This outlines the trade-off between communication and training efﬁciency.
Comparison between update strategies.
As described in Section 3, ProgFed progressively trains the network from the shallowest sub-model M 1 to the full model M .
We verify our update strategy by comparing it with various baselines in the centralized setting.
Baseline : end-to-end training; Layerwise only updates the latest layer E i while still passing the input through the whole model M ; Partial is similar to our method but acquires supervision signals from the last head G S ; Mixed combines Partial and Ours , trained on supervision from both G i and G S ; Random randomly chooses a sub-model M s to update, rather than follows progressive learning.
Table 5 presents the performance and the computation cost ratio.
We make several crucial observations.
(1) Ours and Random do not pass the input through the whole network, making them consume fewer computation costs.
(2) Layerwise greedily trains the network but achieves the worst performance, which highlights the importance of end-to-end ﬁne-tuning.
(3) Ours outperforms Random in both costs and accuracy, verifying the necessity of progressive learning.
We also note that our method does not require additional memory space (compared to Random ) and is easy to implement.
C ONCLUSION
Beyond prior work on expressing models in compact formats, we show a novel approach to modifying the training pipeline to reduce the training costs.
We propose ProgFed and show that progressive learning can be seamlessly applied to federated learning for communication and computation cost reduction.
Extensive results on different architectures from small CNNs to U-Net and different tasks from simple classiﬁcation to medical image segmentation show that our method is communicationand computation-efﬁcient, especially when the training budgets are limited.
Interestingly, we found that a progressive learning scheme has even led to improved performance in vanilla learning and more robust results when learning is perturbed e.g. in the case of gradient compression, which highlights progressive learning as a promising technique in itself with future application domains such as privacy-preserving learning and advanced compression schemes.
R EPRODUCIBILITY S TATEMENT
We have provided all assumptions in the main paper and attached the proof of Theorem 1 in the appendix.
All implementation details and datasets have been discussed in the main paper and the appendix.
We attached the source code as a supplementary material and will publicly release the source code upon acceptance
A A PPENDIX A. P ROOF OF T HEOREM
In this section we prove Theorem 1.
The proof builds on (Mohtashami et al., 2021) that considered training of subnetworks [Equation] but not the progressive learning case [Equation]
Lemma 2.
Let x t denote the weights of the full model [Equation] and x st the weights of the model that is active in iteration t .
Note that it holds x t | E s = x st | E s as per the deﬁnition in the main text.
It holds,
[Equation]
Proof.
Let’s abbreviate g st = g s ( x st ) .
By the update equation x st +1 = x
st − γ t g st it holds x t +1 | E s =
x t | E s − γ t g st | E s .
With the L -smoothness assumption and the deﬁnition of α t ,
[Equation]
Where in the last equation we used the facts that α t ≤ 1 and γ t = α t γ .
We now prove Theorem
[Equation]
Proof.
We ﬁrst deﬁne F t := E f ( x t ) − (min x f ( x )) .
By rearranging Lemma 2, we have
[Equation]
Next, with telescoping summation
[Equation]
we have
[Equation]
We now can prove the ﬁrst of part Theorem 1 by setting the step size γ to be O (min { 1 L , ( F 0 σ 2 T ) 12 } as in (Mohtashami et al., 2021).
To prove the convergence of the model of interest (the second part),
[Equation]
where q
[Equation]
and
[Equation]
[Table]
By deﬁnition q ≥ q t for all t ∈
[ T ] , we reach the last inequality and combine it with the ﬁrst part of the theorem.
[Equation]
Using (cid:15)q 2 as the new threshold, we immediately prove the second part.
B I MPLEMENTATION DETAILS
We describe details of the datasets used in Section 4 and present the hyper-parameters in Table 6.
CIFAR-10.
We conduct experiments on CIFAR-10 datasets for federated learning, following the setup of previous work (McMahan et al., 2017).
The dataset is divided into 100 clients randomly, namely iid distributions for every client.
We adopt the same CNN architecture with 122,570 parameters.
CIFAR-100.
We follow the federated learning benchmark of CIFAR-100 proposed in (Reddi et al., 2021) to conduct the experiments on CIFAR-100.
We use ResNet-18/-152 (batch norm are replaced with group norm (Hsieh et al., 2020)) and
VGG-16/-19 in the centralized setting, while only considering ResNet-18 in the federated experiments.
This setup allows us to evaluate the federated learning systems on non-IID distributions, where we use the splits as suggested in (Reddi et al., 2021).
[Table]
We follow the benchmark setting in (Reddi et al., 2021) to experiment.
There are 3, clients and 671,585 training examples distributed in a non-iid fashion.
The models are eventually evaluated on 77,483 examples, resulting in a challenging task.
BraTS.
In addition to image classiﬁcation, we conduct experiments on brain tumor segmentation based on (Sheller et al., 2020).
We train a 3D-Unet on the BraTS2018 dataset, which includes 285 MRI scans annotated by ﬁve classes of labels.
The network has 9,451,567 parameters.
The training set is randomly partitioned into ten clients.
All clients participate in every training round and locally train their models for three local epochs.
This setting matches the practical medical applications.
Institutions often own relatively stable network conditions, and the data are rare and of high resolution. Architectures.
ConvNets for EMNIST and MNIST consist of two convolution layers, termed
Conv1 and Conv2, followed by two fully connected layers, termed FC1 and FC2.
To apply progressive learning with S = 3 , we set Conv1, Conv2, FC1 to be the three stages, namely E i , and FC2 to the ﬁnal head, namely G S . As for VGGs, we divide the whole networks into ﬁve components according to the max-pooling layers. We combine the ﬁrst two to be E 1 and set the others to be the remaining E i under the setting S = 4 .
To apply ProgFed to ResNets He et al.
(2016), we ﬁrst replace the batch normalization layers with group normalization.
By convention, ResNets have ﬁve convolution components, i.e. Conv1, Conv2 x, Conv3 x, Conv4 x, and Conv5 x. We combine
Conv1 and Conv2 x to be E 1 and all the other components to be the remaining E i .
It thus matches
S = 4 in our setting.
C M ORE R ESULTS C. C OMPARISON BETWEEN UPDATE STRATEGIES
As described in Section 4.5, we compare ProgFed to other baselines.
We additionally report the performance vs. computation costs and performance vs. epochs in Figure 8, where Ours reaches comparable performance while consuming the least cost.
C. C OMPUTATION E FFICIENCY
We present more experiments in the centralized setting to prove the computation efﬁciency of our method.
Figure 9 presents accuracy vs. epochs with four architectures.
The result indicates that our method converges comparably faster to end-to-end training in practice.
Figure 11 presents Figurein bar charts.
Similar to Figure 3, our method improves across architectures while VGGs beneﬁt even more from our method.
Figure 10 presents the computation costs of 3D-Unets on the BraTS dataset.
We make the ﬁrst observation that tumor segmentation requires heavy computation.
Interestingly, even though the earlier stages of Symmetric consume much fewer communication costs (Figure 5), they require more computation costs than Asymmetric .
It might root from the higher resolution of feature maps that Symmetric keeps and thus lead to a trade-off between communication and computation costs.
Figure 11 extend Figure 3 to a larger range { 50% , 60% , 70% , 80% , 90% 98% , 99% , 99 .
95% , best } .
The result shows that our method beneﬁts across models and is especially efﬁcient when training budgets are limited.
C. C OMMUNICATION E FFICIENCY
We present more experiments in the federated setting to prove the communication efﬁciency of our method.
To complement Figure 5, we additionally visualize performance vs. communication costs and performance vs. epochs in Figure 14.
Although our method causes performance ﬂuctuation in some datasets, the performance recovers very quickly.
Figure 15 presents Figure 4 in bar charts.
The results show that our method saves considerable costs in almost all settings except for EMNIST at 95%.
It is because both baseline and our method improve fast at the beginning while our method stands out in the latter phase of training (e.g. after 98%).
The result also supports that our method improves across datasets.
We present Figure 4 in a region that models are applicable.
Here, we plot the ﬁgure with a larger range { 50% , 60% , 70% , 80% , 90% 98% , 99% , 99 .
95% , best } in Figure 13.
The result is consistent, showing that our method beneﬁts across datasets, and is efﬁcient when granted limited training budgets.
[Figure]
C. V ISUALIZATION OF F EDERATED S EGMENTATION
We visualize the outputs of 3D-Unet (see Section 4.3) in Figure 16.
The result matches the number reported in Table 3 that the models perform similarly after they have converged.
However, the communication cost consumption is disparate.
Symmetric only consumes 36 .
40% while the others either do not save any cost or marginally improve it.
We visualize the results in Figure 17 when granted limited communication budgets.
Note that even with the same cost, Symmetric and Asymmetric may stay in different stages because Symmetric starts from the outer part, consisting of signiﬁcantly fewer parameters.
We ﬁnd that Baseline and Asymmetric fail to compress the models with 0 .
18% since their models are of size 34 MB (i.e. 0 .
908% ), while only Symmetric achieve it.
Interestingly, Symmetric has produced promising results at 0 .
18% of costs.
It suggests that our method signiﬁcantly facilitates learning even given limited communication budgets.
Meanwhile, we hope these ﬁndings could inspire more further work on medical learning problems. D M ORE R ELATED WORK We discuss more related work in this section. [Table]
ResNet77.
VGG16 (bn) 71.
VGG19 (bn) 69. [Table] [Table] [Table]
Progressive Learning.
The core idea of progressive learning (PL) is to train the model from easier tasks (e.g., low-resolution outputs or shallower models) to difﬁcult but desired tasks (e.g., highresolution outputs or deeper models).
It was originally proposed to stabilize the training process and has been widely considered in vision tasks such as image synthesis (Karras et al., 2018), image super-resolution
(Wang et al., 2018), facial attribute editing (Wu et al., 2020) and representation learning (Li et al., 2019).
One of the most representative methods is PGGAN (Karras et al., 2018), which trains GANs ﬁrst for generating 4x4 images with a shallow network and progressively extends the network for generating images of size 1024x1024.
In addition to stabilizing the training process,
PL naturally reduces the computation demands since it typically employs smaller models before it reaches the full model.
This even beneﬁts federated learning since the message size also decreases.
Despite the beneﬁts, little work has investigated applying PL to federated learning. We note that although some works (He et al., 2021; Belilovsky et al., 2020) also consider partially training, they often seek solutions to ﬁxing layers rather than considering training difﬁculty, and the application in general federated tasks remain unexplored. [Figure] [Figure]