Resource-Adaptive Federated Learning with All-In-One Neural Composition
Anonymous Author(s)
AffiliationAddress email
Abstract
Conventional Federated Learning (FL) systems inherently assume a uniform processing capacity among clients for deployed models.
However, diverse clienthardware often leads to varying computation resources in practice.
Such systemheterogeneity results in an inevitable trade-off between model complexity anddata accessibility as a bottleneck.
To avoid such a dilemma and achieve resourceadaptive federated learning , we introduce a simple yet effective mechanism, termed
All-In-One Neural Composition , to systematically support training complexityadjustable models with flexible resource adaption.
It is able to efficiently constructmodels at various complexities using one unified neural basis shared among clients,instead of pruning the global model into local ones.
The proposed mechanismendows the system with unhindered access to the full range of knowledge scatteredacross clients and generalizes existing pruning-based solutions by allowing softand learnable extraction of low footprint models.
Extensive experiment results onpopular FL benchmarks demonstrate the effectiveness of our approach.
The resulting FL system empowered by our All-In-One Neural Composition, called FLANC,manifests consistent performance gains across diverse system/data heterogeneoussetups while keeping high efficiency in computation and communication.
Introduction
The success of deep learning is greatly empowered by the large datasets [1, 2].
However, when data isscattered across a vast number of edge devices and subject to privacy regulations, deep learning will be unrealistic.
To this end, Federated Learning (FL) [3, 4] provides a decentralized solution, enablinga number of participants to jointly train a model without exchanging any raw data.
Due to such privacypreserving property, FL is favorable and widely adopted in many applications [5, 6, 7], including butnot limited to face recognition [8], autonomous driving [9] and next-word prediction [10].
When designing federated learning systems [11, 12, 13, 3, 14, 15], it is commonly assumed thatthe local models share the same architecture as the global one.
However, real-world deploymentscan rarely satisfy this ideal assumption.
Instead, participants in the wild often equip with diversedevices ( e.g. , Internet of Things (IoT) devices, mobile phones, tablets and personal computers), whichgreatly vary in computation resource budgets (including computation capacity, memory, storage, and network bandwidth).
Further, even if participants equip with identical devices, the resourcebudgets may still vary due to e.g. , other irrelevant programs consuming an arbitrary portion of the resource, or reduced CPU frequency due to power-saving mode.
This ubiquitous presence of system heterogeneity becomes a major bottleneck for real-world FL applications, as it leads to an inevitable trade-off between model complexity and data accessibility – service providers have to either sacrifice model capacity to enable training on indigent devices, or exclude them along with their unique data
Predictably, in such a dilemma, both choices will eventually harm the performance of the resulting model
[Figure]
O u t p u t c h a nn e l
[Figure]
Local Model Parameters
Parameters
Local Model Parameters
T o c l i e n t T o c l i e n t
T o c l i e n t
T o c l i e n t ( p r u n e )
Device type A
Device type B
Clients
Device type A
Device type B 1.
[Figure]
1.0X
[Figure] [Figure]
The existing FL methods with heterogeneous clients, e.g. ,HeteroFL [16] and FjORD [17] proposeto modify the width of deep neural networks, i.e. , to extract low-footprint sub-models by pruningexcessive channels.
However, we find their performance improvement is inconsistent, which could beeven worse than naively deploying the weakest network with the vanilla FedAvg [12].
As shown in
Figure 1, we use an FL setting with two types of devices A and B, where the high-end device type
A takes up 25% clients and can run the full model (1.0 × in terms of the model width), while thelow-end device type B takes up the majority (75%) of participants but can only afford 0.25 × of thefull model.
A counter-intuitive result is observed with standard ResNet-18 [18] and CIFAR
[19]dataset: the Top-1 accuracy of both methods decrease from 86% to 84% when increasing modelcapacity from 0.25 × to 1.0 × .
Based on this, we conjecture that the extra 0 .
75 × network capacitydoes not benefit from all participants’ knowledge.
Even if all participants contribute to the sameglobal model, pruning in fact limits the contribution of weakest devices to merely the 0 .
25 × network,leaving the majority of parameters under-optimized.
This can be even more severe when there is dataheterogeneity , where inadequately trained parameters may lead to poor generalization against unseenclasses, and hence worse the overall performance.
Given heterogeneous devices, how to unleash the full potential of resource capacity without sacrificingdata accessibility?
In this paper, we propose All-In-One Neural Composition, a new scheme forrun-time resource-adaptive model construction, which formulates networks at different capacities(thus footprint and data transfer cost) as linear combinations of one unified set of parameters, namelyneural basis .
It is designed to be compact, and regularized to be linearly independent, and hence isefficient in training and communication, as well as more faithful model constructions.
With suchscheme incorporated, the resulting FL framework (called FLANC) provides systematic supportfor training complexity-adjustable neural networks, and thus enables efficient and more effectiveresource-adaptive federated learning .
Different from pruning-based methods [16, 17], it makesknowledge contribution from clients to model parameters unhindered from both computationalresource constraints and model architectural complexity.
Our method can be seen as a generalizationof existing solutions by extending hard-coded weight masks to soft and learnable extractions oflower footprint models.
In addition, similar to previous approaches [16, 17], original models can bepre-composed for inference and executed as normal networks without any run-time overhead.
To validate our approach, we conduct extensive experiments under both statistical data heterogeneity(IID and non-IID distribution) and system heterogeneity (static and dynamic) settings.
We evaluateour method on commonly used datasets, i.e. , Fashion-MNIST [20], CIFAR10 and CIFAR100 [19] forimage classification, as well as Shakespeare [21] for next-character prediction.
Although the proposedmechanism is simple in its formulation, it is stable in training, while consistently and significantlyoutperforming the previous system-heterogeneous strategies [16, 17] across different architectures.
The Proposed Method2.1 System Heterogeneity
As aforementioned, conventional FL systems assume a uniform processing capacity for all participants.
Nevertheless, edge devices in the wild often equip with diverse hardware and may have very differentcomputational resources and data transfer budgets.
Thus, it is crucial to properly model systemheterogeneity.
When reflecting this concept to the deep networks, resource constraints can be
[Figure]
expressed in multiple aspects, such as depth (number of layers) and width (number of hiddenchannels).
In this paper, we follow [16, 17] and choose to adjust width for resource adaptation.
Compared to reducing depth, a reduction in width can more effectively reduce parameters andmemory footprint during inference, which is beneficial to the edge devices.
Meanwhile, the resultingnetworks belong to the same model class and share similar intrinsic characteristics, which is preferredfor stable training and model aggregation [16, 17, 22, 23].
We denote the ratio p as a rescaling factor of the number of active channels in a layer W ∈ R S × T .
Both the model size and computational cost of a slimmer p -width model will be reduced by p 2 .
Tomodel device in the wild, one can assign each device k with a maximum affordable capacity ratiop k according to its hardware configuration.
Since it is impractical to exhaustively list all possiblevalues of p due to the diversity of edge devices 1 , we consider a set of representative capacities P(e.g. { 0 . 25 , 0 .
75 , 1 } ) by pre-clustering participating devices into |P| resource groups accordingto their capacity.
2.2 All-In-One Neural Composition
Previous research suggests that the weights of fully-connected and convolution layers are usuallyover-parameterized, and lie on a low-rank subspace [24].
This makes it possible to express theselayers in the low-rank tensor format using decomposition techniques [25, 26].
Given a convolutionweight W k 2 × S × T with filter size k , input channel number S and output channel number T , we canapproximate it as W = V · U with V ∈ R k 2 × S × R , U ∈ R R × T , and R ≤ T .
The factorization forfully-connected layer is equivalent as the k = 1 case.
Such low-rank approximation is effective formodel compression in order to reduce model size and computation cost [27, 28, 29, 30].
Similarly,we speculate federated learning with system heterogeneity could benefit from tensor decomposition.
Thus, we introduce All-In-One Neural Composition, that employs low-rank approximation to represent networks in different widths with a unified expression.
For an arbitrary p -width network, wedecouple its weight W p ∈ R k 2 ×
Sp ×
Tp as a shared tensor V share and capacity-specific tensor U p , i.e. ,
[Equation]
However, Eq.
(1) cannot be directly used as varying p not only changes the output channel butalso rescales the input channel S by p , which makes sharing V share of default shape k 2 × S × Rinapplicable.
We address this issue by adjusting V share to be more fine-grained.
We demonstratethis process in Figure 2.
Specifically, given V share with shape k 2 × R 1 × R 2 , we select R 1 from thecommon divisors of all possible input channel Sp with p ∈ P .
Correspondingly, the capacity-specifictensor U p has the shape R 2 × ( Sp/R 1 × Tp ) .
Their multiplication result is a reshaped weight matrix
W p ∈
R k 2 ×
( R 1 × Sp/R 1 ) × T p and can be used for the original p -width network.
Eq.
(1) in fact means to compose W p with a shared basis V share and coefficients U p through linearcombination.
Specifically, every weight fragment W i,p ∈ R k 2 × R 1 of W p is explicitly written as theweighted sum of R 2 bases V share =
{ V j | j ∈ { 1 , . . .
, R 2 }} with coefficients
U i,p , i.e. ,
[Equation]
where U i,p is the i -th column of U p , and V j ∈ R
k
×
R 1 is the j
[Table]
Eq.
(2) reveals an outstanding property that the knowledge learned by V share can be propagated to allparameters.
In the context of FL, this enables every parameter to access the full range of knowledgeby training the basis V share on all devices.
In addition, our formulation also reduces both computationand communication cost as the low-rank tensor format is inherently more efficient.
Comparison to low-rank model compression.
Low-rank factorization is generally applied as apost-processing step to the pre-trained weights for model compression [31, 25, 27, 26].
Most methodsproduce a single compressed model and require access to the labelled data for fine-tuning.
Hence,in the context of FL, the standard compression techniques neither address system heterogeneity norreduce the footprint and data transfer cost, because they are only applied after the training process.
Moreover, the requirement of fine-tuning is often infeasible for privacy-preserving applications.
Incontrast, our method is applied during the training process, and is able to flexibly construct multiplenetworks and can be stably learned from scratch without post-calibration.
Comparison to pruning-based FL strategies.
Current system-heterogeneous strategies [16, 17]are pruning-based and can be viewed as a specific instantiation of Eq.
(1), by setting V share to thelargest weight W p max and the coefficients U p to be a hard-coded mask with 0 and 1 entries, indicatingwhether a parameter is pruned or not.
From this viewpoint, our method generalizes existing solutionby allowing (1) a more compact basis V share , which ensures high efficiency and reduces data transfercost; and (2) learnable soft coefficients U p , which makes both access to the full range of knowledgeand adaptive utilization possible.
Namely, unlike pruning-based methods [16, 17] where the excludedparameters cannot benefit from data on smaller devices, our approach enables every parameter at allwidth p ∈ P to leverage the full range of knowledge from all clients, no matter whether it can betrained on that device or not.
2.3 Orthogonal Regularization for Enhancing Representation Capacity
The low-rank tensor format of Eq.
(2) can also be interpreted as adaptively searching for parametersfrom a low-rank subspace S = span { V share } .
Spontaneously, if we enforce these basis vectors to belinearly independent, they can form a more expressive subspace and are capable to cover a broaderrange of knowledge.
Nevertheless, according to our observation, this preferable property cannot beobtained implicitly with standard training objectives alone.
To this end, we introduce an extra orthogonal regularizer, which enforces the columns of V T share tobe mutually orthogonal with unit norm.
We achieve this by reducing the L 2 norm of the differencebetween Gram matrix 2 and identity matrix.
To train a p -width network, one can jointly optimize theorthogonal regularizer and the original classification objective, which can be expressed as:
[Equation]
Here for simplicity, we use V share and U p to denote the collections of basis and coefficients over alllayers.
L cls is the classification loss.
l is the layer index and λ is a balancing parameter.
2.4 Federated Learning with All-In-One Neural Composition (FLANC)
In this subsection, we employ our composition technique into the FL framework to enable resourceadaptive federated training.
We abbreviate our framework as “FLANC” for convenience.
As aforementioned, the first step of our framework is to obtain a feasible set P of device capacityby clustering participants into |P| groups based on their available resource.
Meanwhile, the serveralso assigns each client k with a corresponding capacity p k based on the clustering results.
Next, theserver initializes the shared basis V 0 share as well as all capacity-specific coefficients U 0 p with p ∈ P .
It is worth noting that our server does not need to explicitly maintain the global networks duringtraining.
The data transfer cost is further reduced with the efficient tensor format.
A communication round t starts by determining a set of joining devices K t .
After that, the serverbroadcasts the basis V t share to each client k ∈ K t along with the associated resource-specific coefficients U tp k for ad-hoc adaption.
On the client side, each client updates the model with their privatedata for N local iterations.
For each training step, p k -width network is adaptively constructed using
Eq.
Therefore, the basis V share can be optimized by all clients with their local data.
At the end oftraining, the clients send the basis and coefficients back to the server.
The server finishes a communication round by updating the shared basis and coefficients using localtraining results from clients.
In our approach, the basis is fully trained on all participating devices.
On the other hand, capacity-specific coefficients are updated by the devices with the correspondingresource group.
To account for this difference, we provide the following aggregation rule:
[Equation]
where D K t = ∪ k ∈K t D k is the union of datasets of current participants.
And K t,p = { k | k ∈
K t , p k = p } is a subset of K t with the device capacity equals p .
The complete workflow of ourmethod is summarized in Algorithm 1.
Experiments
Datasets.
In this paper, we evaluate our method for image classification tasks on three populardatasets with increasing complexity: Fashion-MNIST [20], CIFAR10 and CIFAR100 [19].
FashionMNIST is a relatively simple dataset containing 60,000 examples of 10 classes.
CIFAR10 and
CIFAR100 are the common classification benchmarks with 50,000 training images with 10 and 100classes respectively.
CIFAR100 is the most challenging dataset as each class only has 500 images.
Our approach is generally applicable to not only vision tasks, but also natural language processingtasks.
To show this, we further conduct experiments on Shakespeare, which is a text dataset builtfrom Shakespeare Dialogues [32], and the task is next-character prediction.
More details about thesedatasets can be found in the supplementary material.
Data Partition.
In our experiment, we follow [16] and consider both cases of IID and non-IID datapartition, in order to comprehensively showcase the performance.
For IID partition, we uniformlysample a same number of data for each client.
For non-IID case, we assume label shifts and distributea subset of classes for each client.
Specifically, the number of classes is set to 3 for Fashion-MNISTand CIFAR10, 30 for CIFAR100.
For Shakespeare, we follow the partition method in [21].
We referthe underlying distribution of raw data (each speaking role) as the non-IID partition and each datapoint is equally likely to be sampled in the IID partition.
Model Architecture.
Our approach can seamlessly work with both convolutional and fully-connectedlayers.
In this paper, we evaluate it across several different architectures.
Specifically, we adaptthe standard ResNet-18 [18] for CIFAR10 and CIFAR100.
For Fashion-MNIST, we adopt a simple4-layer CNN.
For next-character prediction, we apply our approach upon a 1-layer RNN.
Furtherdetails can be found in the supplementary file.
Client Partition.
For all experiments, we follow previous conventions [12] and assume totally 100clients with 10% of them being active for each communication round.
To model system heterogeneity,
[Figure]
we construct 4 different complexity levels with P = { 0 . 25 , 0 . 5 , 0 .
To fully demonstrate theeffectiveness of our approach, we consider both the static and dynamic capacity distribution.
Forstatic setting, we uniformly assign a capacity level for each client, and then client keeps this budgetthroughout the training.
For dynamic setting, we allow each client to randomly increase/decreaseits capacity at every communication round.
This is also a realistic scenario as devices’ capacity canvary drastically due to other irrelevant system processes or CPU frequency changes led by the batterystatus.
Baseline Methods and Implementation Details.
We compare our method with two state-of-the-artsystem-heterogeneous solutions – HeteroFL[16] and FjORD[17].
Both approaches are similar indesign and are based on model pruning.
Other generic FL algorithms [12, 11, 33] are inapplicablewith the presence of heterogeneous devices due to broken assumptions.
Since the official codefor FjORD [17] is absent, we re-implement this method following the author’s descriptions anduse the default parameter settings including the uniform dropout rate.
For hyper-parameter tuning,we split a subset of 10% training examples as the validation set.
After selecting the parameters,validation data are merged back to the training set, and then we retrain the model for final performanceevaluation.
For our method, the selection of λ , R 1 and R 2 depends on architecture and tasks.
Weimplement the proposed approach using PyTorch 1.8.2 on Nvidia A5000 GPUs.
Detailed descriptionsof hyper-parameters and training can be found in the Appendix.
3.1 Main Performance Evaluation
Here we compare our approach with HeteroFL [16] and FjORD[17] on standard benchmarks.
Allresults with static setup are shown in Figure 3.
Results with dynamic setting are reported in Table 1.
In both system-heterogeneous settings, our approach consistently outperforms both HeteroFL and
FjORD on all entries.
The improvements is further magnified when dealing with heterogeneousdata.
Specifically, we found the performance of HeteroFL [16] and FjORD [17] very close in manycases.
This is consistent with our expectation as both methods are based on parameter pruning, evenif FjORD [17] in addition allows large devices to randomly switch to a smaller sub-model throughordered dropout.
However, we do not observe consistent improvements with this dropout technique,and in some cases it further harms the performance of the larger models.
And both HeteroFL and
FLANC are compatible with this technique.
Furthermore, when switching from IID to non-IID datapartition, both approaches suffer from significant performance drop.
For example, on CIFAR10,the overall performance of both FjORD and HeteroFL drops to 60% even with their largest model.
This is mainly because pruning restricts parameters to have a limited access to the full range of dataand thus may not transfer to unseen distributions.
In contrast, our approach allows every parameterto leverage a full range of knowledge through the shared basis, and hence is more robust to dataheterogeneity.
As a result, our FLANC achieves around 13% and 9% performance gain compared to
HeteroFL and FjORD respectively.
[Table]
In terms of efficiency
[Table]
all three methods are able to straightforwardly reduce the amount of data totransfer and the cost of computation by switching to smaller p -width models.
However, benefitingfrom the efficient tensor decomposition format
[Table]
our approach can further reduce the model size,resulting in the best communication efficiency.
3.2 Robustness with Imbalanced Device Distribution
In practice, there is usually no guarantee that different types of devices will be equally distributed.
Here we conduct experiments to compare the robustness towards imbalanced device distributions.
To make things more challenging, we also assume there is a large capacity gap between clients.
Specifically, we consider two types of clients A and B, where A can run the full model with p = 1 and
B has a limited support of p = 0 .
We consider two scenarios where A accounts for 25% and 75%of total devices respectively.0.
1.0 Capacity (p) 84.0 84.5 85.
A cc u r a c y
HeteroFLFjORDFLANC (a) A:25% - B:75% 0.
1.0 Capacity (p)
A cc u r a c y
HeteroFLFjORDFLANC (b) A: [Figure] - B:25%
The are illustrated in Figure 4.
It is notedthat, with a limited number of available devicesof type A, HeteroFL and FjORD cannot effectively improve the accuracy by increasing thecapacity, but instead suffer from performancedegradation.
This is not surprising as the majority of parameters (>90%) is under-trained withonly 1 / 4 portion of all data available.
In contrast, our approach is still able to obtain performance gain with additional parameters.
Thisproves that our approach indeed is more favorable for addressing system heterogeneity issuesfor FL.
On the other hand, when the majority ofdevices can run the original model, the performance gap between ours and HeteroFL shrinksas expected, since in this case most of the parameters can directly access all data.
In contrast,
[Figure]
FjORD squeezes its performance into some intermediate points due to dropout.
While it improvesthe performance of the smaller model, the benefits from larger capacity become marginal.
3.3 Ablation Study
Orthogonality.
Orthogonal regularizer reduces the redundancy and dependency within neural basesand improves the expressiveness of neural composition.
To demonstrate its effectiveness, we removethe orthogonality regularization and compare the performance on CIFAR10.
As shown in Figure 6a,one can observe that our regularizer constantly improves performance across all model capacities.
We further visualize the Gram matrix of the learned basis in Figure 5.
It can be seen that the Grammatrices of our approach are very close to the diagonal matrices, demonstrating the basis is indeedorthogonal.
It is worth noting that such property cannot be learned naturally with purely classificationloss.
[Figure]
Results are reported on CIFAR100 with IID data partition and dynamic system heterogeneity.
Basis Size R 1 .
In our implementation, we setthe basis size R 1 as the common divisor of theinput channels over all models.
Thus, we cancompose all networks with one unified basis.
Tostudy our model’s sensitivity to the choice of
R 1 , we set it to 25%, 50%, 100% of the smallestincoming channel, respectively.
As shown in
Figure 6b, using fine-grained basis can slightlyimproves the accuracy.
The performance peaksat 50% of the smallest incoming channel.
Number of Bases R 2 .
Increasing the numberof bases inherently enlarges the dimensionalityof the subspace and may improve performanceby providing better representational capacity.
We study its effect and consider R 2 to be 25%, 50%and 75% width of the original model, respectively.
As shown in Figure 6c Results suggest that alarger number of bases can improve the performance but the benefit becomes marginal when furtherincreasing from 50% to 75%.
In addition, using more bases also linearly increases the model size.
For our architectures, we select R 2 with the best trade-off between the performance and the numberof parameters.
Related Works
Data Heterogeneity in FL.
In the de facto FL algorithm (FedAvg [12]), clients conduct local trainingwith the received parameters from a central server.
Then the server collects all the locally trainedmodels and aggregates them into a new global model.
The design of FedAvg [12] relies on the assumption that data is uniformly distributed across clients.
However, the assumption does not hold inreal-world scenarios where the underlying data distribution is unknown and very likely non-IID.
Thenon-IID distribution of clients’ local data is one notorious trap in federated learning and results in thenon-negligible client drift issue, which can jeopardize the convergence rate and model performancewhen the data similarity decreases [34].
Tackling the statistical data heterogeneity attracts extensiveattention in recent years.
Pioneer works, like FedProx [35], introduce additional regularizers onthe local training objective to prevent the local models from diverging due to the non-IID datadistribution.
Later on, several works propose inter-client variance reduction techniques [33, 13, 36]by amending the client drift issue with the predicted model updating direction.
Personalized FLis another strategy to allow a model to better fit the local data distribution on a specific client.
One can fine-tune the global model on the local data [37], perform MAML-based personalizedapproaches [38, 39], or achieve the personalization by local batch normalization layers [11].
Thereare also many other emerging explorations dealing with FL data heterogeneity, such as heterogeneousoptimization [40, 41, 42], robust aggregation [15], etc.
Our proposed method is perpendicular to theabove studies and potentially can be combined with them for further improvement.
System Heterogeneity in FL.
In contrast to the substantial effort devoted to data heterogeneity,addressing FL system heterogeneity is largely under-explored.
The majority of existing methodsonly focus on reducing communication cost, so that the data transfer cost can be affordable bythose edge devices with constrained bandwidth and energy consumption.
For example, severaladvanced FL optimizers [13, 33, 41, 43] have been proposed to improve the convergence rate, andthus reduce data transfer cost by decreasing the number of required communication rounds.
Othermethods [44, 45, 46, 47] improve communication efficiency by combining quantization [44, 46]and sparse training [45, 48] techniques.
However, all aforementioned approaches still assume auniform processing capacity among all participants, which cannot be guaranteed for real-worldscenarios.
One promising way to remove this assumption is to leverage knowledge distillation(KD) for learning the central model instead of conventional coordinate-wise averaging.
While suchmethods [49, 50, 51, 52, 53] allow different local models, applying KD often requires additionalaccess to some common public datasets to alleviate the difficulty in knowledge transmission [54],which is impractical in FL.
Our method is most related to HeteroFL [16] and FjORD [17], whichallow federated training of local models in different widths adaptively “pruned” from a global model.
Nevertheless, as discussed before, they cannot fundamentally resolve the challenges and still sufferfrom degraded performance similar to many conventional FL methods.
Compression Techniques.
Both our approach and previous heterogeneous-system solutions arerelated to model compression techniques [31, 27, 26, 55, 56, 57], a classic research field to reduce therun-time storage and the latency of deep neural networks.
Some of the most representative techniquesinclude quantization [57, 58], neural pruning [55, 56], and tensor decomposition [31, 27, 26].
In thecontext of FL, these techniques are first studied and demonstrated to be effective in reducing thecommunication cost [44, 45, 46, 47].
To handle system heterogeneity, current approaches [16, 17]extend model pruning techniques to extract sub-models to fulfill the resource requirements.
However,as discussed, pruning simply drops model parameters, preventing the dropped ones from leveragingcomplete knowledge scattered across various devices.
On the other hand, our method can beconnected to tensor decomposition, where we factorize layers into a unified neural basis representingshared knowledge and capacity-specific coefficients for linear composition.
In this way, it not onlyallows resource-adaptive model training through composition, but also naturally ensures the completeknowledge effectively propagates to models at various complexities.
Conclusion
The assumption of uniform processing capacity in conventional federated learning contradicts thevarying computation resources in practice.
As a result, these methods suffer from an inevitabledilemma between model complexity and data accessibility.
To overcome this problem, we proposea simple yet effective mechanism, termed All-In-One Neural Composition to systematically support resource-adaptive federated learning.
Unlike previous methods, our method enables flexibleconstructions of models in different complexities, and allows unhindered access to the full rangeof knowledge scattered across clients.
Comprehensive experimental results on vision and languagetasks demonstrate the effectiveness of our method, manifesting consistent performance gain acrossdifferent setups.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Experiment results validate the main claims.
(b) Did you describe the limitations of your work?
[Yes] Please see the supplementary
material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please seethe supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Code will be
released after the reviewing process.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] See Sec. 3.
More training details are provided in the supplementary
material.
(c) Did you report error bars (e g., with respect to the random seed after running exper-
iments multiple times)?
[No] Error bars are not reported because it would be too
computationally expensive (since the main experiments involve training 192 individualmodels).
However, we do consider random effects by setting all seeds to 1 and using
PyTorch deterministic algorithms 3 in all experiments for fair comparison.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Sec.
3 and the supplementary
material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Included the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Included in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] Included in the supplementary material.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]