M OCO SFL: ENABLING CROSS - CLIENT COLLABORATIVE SELF - SUPERVISED LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing collaborative self-supervised learning (SSL) schemes are not suitable for cross-client applications because of their expensive computation and large local data requirements.
To address these issues, we propose MocoSFL, a collaborative SSL framework based on Split Federated Learning (SFL) and Momentum
Contrast (MoCo).
In MocoSFL, the large backbone model is split into a small client-side model and a large server-side model, and only the small client-side model is processed locally on the client’s local devices.
MocoSFL has three key components: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and volume of local data; (iii) frequent synchronization that helps achieve better non-IID performance because of smaller local model divergence.
For a 1,000-client case with non-IID data (each client only has data from 2 random classes of CIFAR-10),
MocoSFL can achieve over 84% accuracy with ResNet-18 model.
Next we present
TAResSFL module that significantly improves the resistance to privacy threats and communication overhead with small sacrifice in accuracy for a MocoSFL system.
On a Raspberry Pi 4B device, the MocoSFL-based scheme requires less than 1 MB of memory and less than 40 MB of communication, and consumes less than 5W power.
Thus, compared to the state-of-the-art FL-based approach, MocoSFL has significant advantages in both accuracy and practicality for cross-client applications.
I NTRODUCTION
Collaborative learning schemes have become increasingly popular, as clients can train their own local models without sharing their private local data.
Current collaborative learning applications mostly focus on supervised learning applications where labels are available (Hard et al., 2018; Roth et al., 2020).
However, availability of fully-labeled data may not be practical since labeling requires expertise and can be difficult to execute, especially for the average client.
Federated learning (FL) (McMahan et al., 2017) is the most popular collaborative learning framework.
One representative algorithm is “FedAvg” , where clients send their local copies of the model to the server and the server performs a weighted average operation (weight depends on the amount of data) to get a new global model.
FL has achieved great success in supervised learning, and has been used successfully in a wide range of applications, such as next word prediction McMahan et al.
visual object detection for safety Liu et al.
(2020), recommendation Wu et al.
(2022a;b), graph-based analysis Chen et al.
; Wu et al.
(2022c), etc.
For collaborative learning on unlabeled data, prior works (Zhang et al., 2020; Zhuang et al., 2021; 2022) combine FL scheme with classic self-supervised learning (SSL) methods such as BYOL (Grill et al., 2020) and Moco (He et al., 2020).
These methods can all achieve good performance when clients’ data is Independent and Identically Distributed (IID) but suffer from poor performance in non-IID cases.
Recently, Zhuang et al.
(2022) mitigated the non-IID performance drop with divergence-aware aggregation technique and provided state-of-the-art (SoTA) accuracy performance using a combination of FL and different SSL methods.
However, these SoTA FL-SSL schemes are not practical for cross-client applications.
First , FLSSL imposes a significant computation overhead and large memory requirement on clients.
This is
[Figure]
because SSL requires a large backbone architecture (Chen et al., 2020a) together with a large batch size to ensure good performance.
As shown in Fig. 1(a), accuracy drops dramatically when batch size is low for both FL-SSL methods.
Second , FL-SSL schemes fail to maintain a good accuracy when the number of clients is large (cross-client cases), as shown in Fig. 1(b).
For a dataset with fixed size, when the number of clients increases, data per client decreases, resulting in accuracy degradation.
The drop in accuracy is mainly because of the failure to meet data size requirement in performing contrastive learning.
Zhang et al.
; Wu et al.
(2021) attempt to address this issue in
FL-SSL with remote feature sharing.
However, this introduces a high communication overhead due to client-to-client feature memory synchronization; for a 100-client system, it costs around 2.46GB per synchronization per client.
To solve these challenges, we propose MocoSFL, a scheme based on Split Federated Learning (SFL) Thapa et al.
(2020) that incorporates the feature memory bank and momentum model designs of MoCo (He et al., 2020).
We adopt the SFL scheme for three reasons: (i) SFL utilizes a smaller client-side model and so reduces the computation overhead and has lower memory consumption and model parameters; (ii) SFL’s latent vector concatenation enables a large equivalent batch size for the centralized server-side model, making micro-batch training possible for clients and thus reducing client’s local memory; (iii) When combined with MoCo’s key-storing mechanism,
SFL’s shared server-side model enables effective feature sharing, which removes the requirement of large amount of local data and makes the scheme possible for cross-client applications.
As a result,
MocoSFL achieves good accuracy with ultra-low memory requirements and computation overhead, and can support a very large number of clients.
MocoSFL shows better non-IID performance since local model divergence is smaller.
However, the use of SFL brings extra communication overhead as well as data privacy concerns.
Thus, we present target-aware ResSFL (TAResSFL) module as an effective solution to mitigate these issues with small accuracy drop.
Our main contributions are:
• We identify two major challenges in deriving high accuracy in FL-SSL schemes for crossclient applications.
These are availability of a large amount of data that is required for contrastive learning and the ability to process them in clients who may not have sufficient hardware resources.
• We propose MocoSFL, an SFL-based scheme to address the two challenges.
MocoSFL uses a small client-side model, latent vector concatenation, and feature sharing.
For cross-client case, MocoSFL is the only viable and practical solution.
For cross-silo case, MocoSFL can achieve even better performance than SoTA FL-SSL schemes under non-IID setting because of smaller model divergence.
• To address communication overhead and privacy issues that are inherent to SFL-based schemes, we propose target-domain-aware ResSFL (TAResSFL) that effectively reduces the communication cost and mitigates model inversion attacks.
B ACKGROUND 2. S ELF -S UPERVISED L EARNING
To learn from unlabeled data, SSL schemes based on contrastive learning such as SimCLR Chen et al.
(2020a), BYOL Grill et al.
Simsiam Chen and He (2021) and MoCo He et al.
(2020) have achieved great performance on popular benchmarks.
Unlike other schemes (BYOL, SimCLR, etc.)
that use other samples in the current data-batch as negative keys, MoCo uses previously computed positive keys as negative keys, and stores current positive keys in the feature memory for future iterations.
The key-storing mechanism results in a relatively smaller batch size requirement that is beneficial for reducing device memory and also makes it easy for feature sharing implementations for our proposed scheme.
For the loss function, MoCo relies on InfoNCE loss Oord et al.
as the contrastive mechanism to update its model parameters:
[Equation]
where query key Q and positive key K + are the output vectors of server-side momentum model and the momentum model, respectively, obtained by processing two augmented views of the image.
N denotes all negative keys in the feature memory of size M .
Importantly, the success of MoCo scheme highly depends on the “hardness” of its negative keys (Kalantidis et al., 2020; Robinson et al., 2020).
The “hardness” of a negative key N in the feature memory bank, can be determined by the similarity (inner-product) between Q t (at step t ) and N ; the smaller the similarity, the better the “hardness”.
We notice the “hardness” of negative key N reduces quickly because model updates are in the direction of minimizing the InfoNCE loss in Eq.
As a result, MoCo adopts a slow-changing momentum model to produce consistent negative keys to add to the feature memory at the end of every training step and thereby maintains their hardness.
[Equation]
L EARNING
Split Federated Learning (SFL) Thapa et al.
(2020) is a recent collaborative learning scheme that focuses on high computation efficiency at the client side.
It splits the original model architecture into two parts, the client-side model that contains all layers upto the “cut-layer” and the server-side model that contains the remaining layers.
We distribute copies of client-side model C i to client- i ’s local devices and instantiate the server-side model S in a cloud server.
To complete each training step, clients need to send the latent vectors (the output of client-side model) to the server, and the server processes latent vectors, computes the loss, performs backward propagation and returns the corresponding gradients to clients.
Thapa et al.
(2020) present two possible ways for server to process latent vectors sent by clients.
In this paper, we use SFL-V1 where the server concatenates all clients’ latent vectors and processes them altogether, which makes the equivalent batch size larger at the server and benefits contrastive learning.
In contrast, in SFL-V2, client’s latent vectors are processed sequentially in a first-come-first-serve manner and thus does not benefit from the “large batch”.
We provide details of SFL-V1 in Appendix A.2.
M OTIVATION
As mentioned in Section 1, the two challenges in extending FL-SSL to cross-client applications are high computing resource requirement and large data requirement.
3. H IGH COMPUTING RESOURCE REQUIREMENT
The first challenge is the computing resource requirement of training an SSL model locally.
Using a compact backbone model may be accurate for supervised learning, but is not suitable for SSL as it has a much higher requirement on the model capacity.
(Shi et al., 2021; Fang et al., 2021) show that compact architectures like Mobilenet-V3 (Howard et al., 2019), EfficientNet Tan and Le (2019) suffer from over 10% accuracy degradation compared to a larger ResNet-18 architecture, while an even larger ResNet-50 model has over 15% better accuracy compared to ResNet-18 on ImageNet dataset (Deng et al., 2009).
This means memory requirement for training an SSL model with high accuracy is very high.
Using a smaller batch size reduces accuracy dramatically, as shown in Fig. 1(a), and is thus not an option.
A FL-SSL scheme (ResNet-18 with a batch size of 128) costs 590.
MFLOPs per image and over 1100MB of memory per client, which is not practical.
3. L ARGE DATA REQUIREMENT
The other major difficulty for FL-SSL to generalize to the cross-client case is the large data requirement.
For cross-client applications, the amount of data available to each client can be very limited.
For example, in a cross-silo medical application, a client can be a hospital with tons of data.
In comparison, in a cross-client application, a client can be a patient who has limited amount of data.
The root of the problem lies in the difficulty to find hard negative samples when clients do not have enough local data.
When the amount of data is larger, the chance for hard negative samples to be present becomes much higher.
As a result, existing FL-SSL can only be successful for cross-silo applications where clients have large amount of data and can perform effective contrastive learning locally.
As demonstrated in Fig. 1(b), we observe high accuracy when clients have 10K samples of data, while the accuracy drops quickly to around 30% when clients have only 500 samples.
Unfortunately, feature sharing Zhang et al.
; Wu et al.
(2021) in FL-SSL schemes cannot solve the large data requirement.
These schemes update the shared feature memory less frequently because of significant communication overhead with each update.
Since clients must synchronize their local copies 1 of the shared feature memory each time a minor change happens, in Zhang et al.
, new negative keys are added to the feature memory only once per epoch.
Unfortunately, they cannot maintain a good “hardness” through the entire epoch and thus feature sharing mechanism brings very limited ( ∼ 1 %) improvement in accuracy.
Also, the fact that total synchronization overhead scales quadratically with the number of clients makes the method not practical for cross-client use case.
M ETHOD 4. P ROPOSED M OCO SFL
Our proposed MocoSFL is an innovative combination of SFL-V1 and MoCo-V2 (Chen et al., 2020b) as shown in Fig.
There are three key components.
First , in each training step, the latent vectors sent by all clients are concatenated before being processed by the server-side model.
This helps achieve a large equivalent batch size in order to support mini-batch training.
Second , we use a shared feature memory which is updated by positive keys contributed by all clients in every training step.
Third , we improve the non-IID performance by using a higher synchronization frequency.
Next, we will elaborate on how these three components in the proposed MocoSFL address the two challenges in Section 4.2 and Section 4.3.
Section 4.4 describes how MocoSFL achieves better nonIID performance and Section 4.5 addresses the privacy and communication issues of the proposed scheme.
4. R EDUCE H ARDWARE R ESOURCE R EQUIREMENT
Choice of SFL helps reduce computational overhead and memory consumption at the client-end because of the much smaller client-side model.
For example, on a CIFAR-10 ResNet-18 model with a batch size of 128, a client-side model with 3 layers only costs 13.7% of the FLOPs compared to the entire model, and its memory cost is 227MB, merely one fourth of the entire model.
Furthermore, we reduce the batch size to 1 (also known as “micro-batch”), to further reduce the memory consumption.
The use of micro-batch in local model training is only possible thanks to the latent vector concatenation mechanism which basically aggregates latent vectors sent by all clients into a big batch before sending it to the server.
In addition, in a micro-batch setting, we replace the batch normalization layer by group normalization Wu and He (2018) and weight standardization (Qiao et al., 2019) to gain better accuracy performance.
In Fig. 3(b), we compare the computation and memory consumption of the proposed MocoSFL with the FL-SSL scheme.
MocoSFL with cut-layer of 3 achieves ∼ 288 × reduction in memory consumption than FL-SSL and has 2%-10% higher accuracy.
Details of accuracy evaluation are included in Section 5.1.
Wireless communication
Server momentum
InfoNCE
Loss
MocoSFL Scheme
Server online 𝑲 + 𝑸
Shared Feature Memory 𝑵
DeQueue
EnQueue 𝑲 +
Server
Client 𝑗 momentum 𝑨 𝒌 + ,𝒋
Client 𝑗 online 𝑿 𝒌 + ,𝒋 𝑿 𝒒,𝒋
Client 𝒋
Augmentation 𝑨 𝒒,𝒋 𝑨 𝒌 + 𝑨 𝒒
Frequent Sync
Concatenate
Concatenate
Client 𝑖 momentum 𝑨 𝒌 + ,𝒊
Client 𝑖 online 𝑿 𝒌 + ,𝒊 𝑿 𝒒,𝒊
Client 𝒊 𝑿 𝒊
Augmentation 𝑨 𝒒,𝒊 𝑿 𝒋
[Figure]
4. M ITIGATE L ARGE D ATA R EQUIREMENT
As indicated by (Kalantidis et al., 2020; Robinson et al., 2020), the “hardness” of a negative key N heavily depends on its similarity with the current query key Q , given that N and Q have different ground-truth labels.
To evaluate the hardness of negative key N 0 residing in the feature memory, we use the similarity measure (inner-product) between N 0 and Q t , a freshly calculated query key at time t .
In FL-SSL with feature sharing, the negative key is only updated after a long period of time to reduce communication overhead.
As a result, the hardness diminishes quickly.
In contrast, MocoSFL frequently updates its feature memory to maintain a good hardness.
At every training step, a freshly calculated positive key K + is added to the tail of the queue, and the oldest one is popped out.
However, frequent updates of feature memory is not enough to ensure a high level of hardness.
We also find it is necessary to use a large batch size and a large feature memory.
This finding agrees with the study in Bulat et al.
and also explains the accuracy drop for a small batch size in
FL-MocoV2 in Fig.
1(a).
To illustrate this, we consider the total similarity measure at time t and make the following assumptions:
For the newest batch of negative keys N t = K t − 1 of size B in the feature memory at time t , we assume the similarity measure between N t and Q t is a constant η for all t .
We also assume similarity of every batch of negative keys in feature memory gets reduced by a constant factor γ ( γ < 1 ) after each model update to represent the degradation caused by model updates.
Thus, for a freshly computed query Q t , its total similarity measure with negative keys in the feature memory can be represented as:
[Equation]
where B is the batch size and M is the feature memory size.
We see that using a large batch size
B is beneficial as it helps bring more fresh negative keys and maintain better hardness.
Also, using a larger feature memory (increasing M ) can keep enough negative keys and contribute to a better total hardness.
In the cross-client case, FL-SSL schemes can hardly meet these two requirements because of the small clients’ memory.
Nonetheless, MocoSFL can easily fulfill them because (1) latent vector concatenation enables a large equivalent batch size, and (2) feature memory hosted by the cloud server can be much larger.
4. I MPROVING N ON -IID P ERFORMANCE
We found that use of SFL results in fewer model parameters at the client side and hence smaller model divergence.
Furthermore, introducing frequent synchronization in MocoSFL provides additional reduction in model divergence and greatly improves the non-IID performance.
According to Zhang et al.
; Zhuang et al.
(2021; 2022), the model divergence between two models is calculated
F L - S S L
M o c o S F L - 5 M o c o S F L - 3 M o c o S F L - 1  91
D i v e r g e n c e
S y n c F r e q = 1 S y n c F r e q =
S y n c F r e q = 1(a) 1 6 2 7 .9 2 1 .6 .0 .
M F L O P s / i m a g e
M e m o r y
( M B )4 08 01 2 01 6
M e t r i c s
F L - S S L
M o c o S F LM o c o S F L -
[Figure]
as the L2 norm of the weight difference.
Following the same idea, the total divergence measure of a cross-client system can be measured as the average weight divergence of local models w.r.t.
the global model during training: divergence =
[Equation]
[Equation]
(4) where L denotes the number of layers in the client-side model, E denotes the total number of synchronizations, N C denotes the number of clients, and l, e, i are the respective indices for L, E, N C .
W ∗ is the average of all client models W i .
MocoSFL reduces the model divergence with two orthogonal mechanisms.
The first mechanism is the reduction of client-side model size, which directly results in a lower model divergence.
As shown in Fig. 3(a), compared to FL-SSL scheme,
MocoSFL has a much lower model divergence when the client-side model has less than 5 layers.
The other mechanism is frequent model synchronization which helps reduce the model divergence.
This is only possible in SFL because of communication overhead of sending weights in a smaller client-side model, is smaller.
Fig.
3(a) also illustrates how model divergence further reduces as we increase the synchronization frequency.
4. I MPROVING P RIVACY AND C OMMUNICATION O VERHEAD OF M OCO SFL
The proposed MocoSFL scheme is based on SFL and suffers from two issues – high overall communication overhead due to transmitting and receiving latent vectors and vulnerability to Model Inversion Attack (MIA)
Fredrikson et al.
(2015), in which the server can reconstruct clients’ raw inputs from latent vectors, making clients’ data privacy questionable.
(We leave the details of its threat model and working mechanism in Appendix A.3)
To address the privacy and communication issues in MocoSFL, we propose Target-Aware-ResSFL (TAResSFL).
TAResSFL extends ResSFL Li et al.
(2022) for self-supervised learning through: (1) target-data-aware self-supervised pre-training, and (2) freezing feature extractor during SFL training.
TAResSFL also utilizes the bottleneck layer design from ResSFL to reduce the communication overhead.
In ResSFL Li et al.
, the server performs pretraining to build up the resistance to MIA using data from a different domain since it does not have access to clients’ data.
Next, the pretrained resistant client-side model is transferred to the clients and gets fine-tuned using SFL.
TAResSFL improves the pretraining step by assuming that the server can get access to a small subset ( < 1%) of training data, together with large amount of data from another domain, and perform pre-training using self-supervised learning.
Such a pretrained client-side model has better transferability, and can thus stay frozen during SFL process, thereby avoiding the expensive fine-tuning.
As shown in Fig.
4, we blend the source dataset X s with a tiny subset of target dataset X t during the feature extractor training.
The attacker-aware training has the min-max form:
[Equation]
where R in the inner maximization denotes a similarity measure, for which we use the structural similarity index (SSIM) score Zhao et al.
The inner maximization is used to train the simulated attack model G , whose function is reconstructing the activation back to a raw input state that is similar with ground-truth X q .
The outer minimization step goes in the direction of lowering contrastive loss, where the regularization term makes the model accurate as well as resistant to attack.
These two steps are done alternatively to make the feature extractor resistant, and also be able to achieve good accuracy on the target dataset.
Fig.
4 presents the Target-aware ResSFL scheme.
We use the resistant feature extractor to initialize client-side models during transfer step, as shown by the pink arrow in Fig. 4.
Unlike ResSFL, here we freeze its parameter completely to maintain the resistance since any parameter change can cause resistance drop.
However, freezing brings a noticeable accuracy drop of larger than 3%, even with the use of CIFAR-100 as source dataset if we perform pre-training by only using the source dataset (see Table 4).
But if we blend the source data with a small portion of target data during pre-training, the accuracy can be greatly improved.
Since the model accuracy validation is done by the server for monitoring purpose (Bhagoji et al., 2019), we believe that the server can separate out a small proportion of validation data to meet the target data availability assumption.
The freezing also greatly benefits hardware resource requirement since: (1) clients only need to transmit the latent vectors to the server and do not need to perform backward propagation using gradients from the server; (2) client-side model synchronization is not needed.
As a result, MocoSFL with TAResSFL component achieves a ∼ 128 × (=5001.4/39.1) overall communication reduction compared to the original MocoSFL, as shown in Table 1.
Here MocoSFL methods undergomodel synchronizations for 200 epochs while FL-SSL methods need 100 synchronizations (using the same setting in Zhuang et al.
v
MocoSFL-TAResSFL Scheme
[Figure]
[Figure]
E XPERIMENTAL R ESULT
Experimental Setting.
We simulate the multi-client MocoSFL scheme on a Linux machine, where we use different CPU threads to simulate different clients and a single RTX-3090 GPU to simulate the cloud server.
We use ResNet-18 (He et al., 2016) for the majority of the experiments to better compare with existing SoTA (Zhuang et al., 2022).
We use CIFAR-10 as the main dataset and also present results on CIFAR-100 and ImageNet 12-class subset as in Li et al.
For the IID case, we assume the entire dataset is divided randomly and equally among all clients.
For non-IID experiments, we mainly consider the pathological (aka.
class-wise) non-IID distribution as in McMahan et al.
(2017); Zhuang et al.
(2022) where we assign 2 classes of CIFAR-10/ImageNet-12 data or 20 classes of CIFAR-100 data randomly to each client.
We perform MocoSFL training for a total of 200 epochs, using SGD as the optimizer with an initial learning rate of 0.06.
For accuracy performance evaluation, we adopt similar linear probe methods as in Grill et al.
; Zhuang et al.
(2022), where we train a new linear classifier on the outputs generated by the MocoSFL backbone model.
We include details of hyper-parameter choices and evaluations in Appendix A.1.
5. A CCURACY P ERFORMANCE
Improved non-IID performance.
Fig.
5 shows how the increased synchronization frequency can significantly improve the non-IID accuracy.
We present results for the cut-layer choices of 1 and 3 convolutional layers in the client-side model, represented by “MocoSFL-1” and “MocoSFL-3”, respectively.
We attribute the improved accuracy to the reduction in model divergence.
Comparison with FL-SSL.
When synchronization frequency of the MocoSFL is set to 10 (per epoch) for the 5-client cases, MocoSFL achieves significantly better non-IID accuracy performance than Zhuang et al.
(2022) on CIFAR-10 dataset due to lower model divergence (see Table 2).
On
CIFAR-100 dataset, with N C
= 5 , we observe our method has lower accuracy than FL-SSL methods.
We hypothesize that the performance of MocoSFL is more sensitive to the model complexity, and thus has limited performance for a more complex task like CIFAR-100.
When model complexity is high enough, for instance, on a larger ResNet-50 model, accuracy of our method is ∼ 4% higher than Zhuang et al.
(2022) as shown in Appendix B.7.
Furthermore, our methods outperform FL-SSL methods by a large margin in 20-client cases thanks to the feature sharing aspect.
[Figure]
NN
A cc u r a c y
[Figure]
A cc u r a c y ( % )
[Figure]
[Figure]
[Table]
Cross-client Performance.
Our proposed MocoSFL can generalize from a cross-silo application (with upto 20 clients) to a cross-client application with 100, 200, and 1000 clients.
Note that none of the previous FL-SSL methods can scale to such a large number of clients.
For the hyper-parameter choices, we follow two principles introduced in Appendix A.1 – we let each client use a batch size of 1 and use the synchronization frequency of f S =
(1000 /N C ) /epoch,
and we set the client sampling ratio to 100 /N C to keep the same equivalent batch size at the server end.
The results are shown in Table 3.
Note that each client has only 50 data samples in the 1000-client case.
MocoSFL’s accuracy for IID case is high when N C increases from 100 to 1,000, though its accuracy drop by 1% for non-IID case.
This small drop is because model divergence scales with number of clients as described in Section 4.4.
[Table]
5. P RIVACY E VALUATION
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
We use G to perform MIA attack; the architecture information of G is given in Appendix A.3.
We assume the real-time attacker also uses an attack model with the same architecture as G .
For the pre-training step, we use a fixed hyper-parameter choice of λ = 2 . 0 and a target SSIM level of 0.6 to limit the strength of regularization.
We assume 0.0%, 0.5% and 1.0% of the target dataset CIFARis accessible [Figure] and we use CIFAR-100 dataset as source dataset to assist the pre-training.
For the bottleneck layers, we use a pair of convolutional layers of 4 channels with stride of 2 (C4S2).
We set the cut-layer choices to 1 and 3 and set the #clients to 100 for the training process.
Successful mitigation of MIA.
As shown in Table 4, applying TAResSFL can achieve good accuracy performance ( > 81%) as well as high enough MIA resistance ( > 0.020) for most cases.
Fig.
6 shows the visualization of MocoSFL-3 when 1.0% target data is available.
We observe the reconstructed images are much more noisy and blurry such that the subject can be successfully hidden.
Larger cut-layer allows a better resistance-accuracy tradeoff.
Using a smaller cut-layer ofseems a better choice in terms of accuracy and hardware requirement.
However, as shown in Table 4, the accuracy and resistance tradeoff seems much better by using a cut-layer of 3.
We believe that the extra client-side model complexity helps in the optimization of both accuracy and resistance objectives.
So, applying TAResSFL with a larger cut-layer is a more favorable option.
5. H ARDWARE D EMONSTRATION
Finally, we compare the total hardware resource cost of the proposed MocoSFL and “MocoSFL+TAResSFL” with synchronization frequency of 1/epoch for 200 epochs, and FL-SSL (Zhuang et al., 2022) with 500 local epochs per client and synchronization frequency set to 1 per 5 local epochs (original setting).
For MocoSFL, we use 1,000 clients with batch size of 1, and cut-layer of 3.
For FL-SSL, to achieve similar accuracy, we use 5 clients with batch size of 128.
And we assume the data follows the default 2-class non-IID setting.
To evaluate overhead, we use commonly-used libraries such as ‘fvcore’ for FLOPs and ‘torch.cuda.memory allocated’ for memory (please see Appendix A. for detail.)
For power measurement, we use a Raspberry Pi 4B equipped with 1GB memory as one real client and simulate all other clients on the PC.
[Equation]
TAResSFL Scheme v
Feature Extractor 𝑨 𝒕,𝒌+ 𝑨 𝒔,𝒌+
Tail model + Contrastive head 𝑸 𝒕 ,𝑲 𝒕+ 𝑸 𝒔 ,𝑲 𝒔+ 𝑨 𝒕,𝒒 𝑨 𝒔,𝒒 𝑿 𝒕,𝒌+ 𝑿 𝒔,𝒌+ 𝑿 𝒕,𝒒 𝑿 𝒔,𝒒
v
Simulated Attacker 𝑿 𝒕,𝒒∗ 𝑨 𝒕,𝒒 𝐼𝑛𝑓𝑜𝑁𝐶𝐸 𝑙𝑜𝑠𝑠 v
Resistant Feature Extractor initialize ClientsOnline models Freeze ❷ ❶ 𝑿 𝒔,𝒒∗ 𝑨 𝒔,𝒒
Server Battery
TCP
Client
[Figure]
v
MocoSFL Scheme v
Feature Extractor 𝑨 𝒕,𝒌+ 𝑨 𝒔,𝒌+
Tail model + Contrastive head 𝑸 𝒕 ,𝑲 𝒕+ 𝑸 𝒔 ,𝑲 𝒔+ 𝑨 𝒕,𝒒 𝑨 𝒔,𝒒 𝑿 𝒕,𝒌+ 𝑿 𝒔,𝒌+ 𝑿 𝒕,𝒒 𝑿 𝒔,𝒒
v
Simulated Attacker 𝑿 𝒕,𝒒∗ 𝑨 𝒕,𝒒 𝐼𝑛𝑓𝑜𝑁𝐶𝐸 𝑙𝑜𝑠𝑠 v
Resistant Feature Extractor initialize ClientsOnline models freeze ❷ ❶ 𝑿 𝒔,𝒒∗ 𝑨 𝒔,𝒒   3.
Computation (GFLOPs)
Communication (MB)
Memory (MB)
MocoSFL-TAResSFL
FL-SSL
Server Battery
TCP
Client
[Figure]
via a measurement tool, and communicates wirelessly with the server using websocket (TCP).
As shown in Fig.
7, FL-SSL requires 7,949 TFLOPs for the entire training session as each client needs to perform 500 local epochs on 10K data and a memory consumption of 921 MB 2 .
The communication overhead due to synchronization has a high cost of around 8,269 MB.
Thus, FL-SSL is only suitable for cross-silo situation.
For “MocoSFL+TAResSFL”, hardware requirements are reasonable since computation is only 734 GFLOPs and communication is around 39 MB in total.
The memory consumption is a tiny 0.8 MB mostly due to the weight parameters since TAResSFL does not require local training.
As shown in Fig.
8, our measurement using a USB multimeter shows the proposed MocoSFL running on the Raspberry Pi only draws power of 2.26W, in average, and consumes around 9,100 mAh on a 5V battery.
C ONCLUSION
We propose MocoSFL, a collaborative SSL framework based on SFL.
The proposed framework addresses hardware resource requirement at client-side by enabling small batch size training and computation offloading.
It also relieves the large data requirement of local contrastive learning by enabling effective feature sharing.
The proposed scheme is the only one that can support a large number of clients.
In combination with a ResSFL-based module, it addresses privacy concerns of MocoSFL.
Finally, it achieves even better IID/non-IID performance with much lower hardware requirement than the SoTA FL-based SSL methods.
E THICS S TATEMENT
In this work, we address two practical issues of hardware (compute and memory) resources and large data requirement in collaborative SSL by using a SFL-based scheme.
Compared to the conventional approaches, the resulting MocoSFL scheme has significant advantages in affordability and is much easier to deploy.
Apart from being environmental-friendly, MocoSFL makes it possible to reach more clients, especially for those in poverty or under-represented minorities, and thus eliminate potential human-bias and unfairness issues of the ML service.
We address the privacy issues in the proposed MocoSFL by proposing TAResSFL module to use in sensitive applications.
Fig.
6 shows that the subject of the raw image can be successfully hidden.
More discussion on the privacy of “MocoSFL+TAResSFL”.
We notice a line of works known as instance encoding (Huang et al., 2020; Yala et al., 2021), which try to protect users’ data by transforming the original dataset to a distorted version such that they cannot be recognized by humans while an arbitrary deep learning model can still learn useful information from it (i.e. achieve high accuracy on a classification task).
MocoSFL, especially with the TAResSFL, where the client-side model is frozen, have some similarities since the frozen client-side model can be seen as a transformation, and the collection of latent vectors can be regarded as the transformed dataset.
However, “MocoSFL+TAResSFL” scheme has two favorable properties that make it distinct from instance encoding methods.
Domain dependency.
First, SFL only finds a transformation method for a given task.
Since the transformation method itself (the client-side model) heavily depends on the target domain information (as we need access to the target domain data to train the client-side model), it cannot work on data from another domain.
As the output of the “client-side model transformation”, latent vectors are only useful for current problem without any transferability guarantee.
However, instance encoding methods intend to derive a general transformation method that can work across domains, targeting a harder problem.
Risk Control.
Instance encoding methods publish the transformed dataset to the wide public which is risky.
As pointed out by Carlini et al.
(2021): all the raw data will be leaked if a successful decryption method is invented in the future even if it does not exist now.
While in our proposed scheme, the latent vectors from clients will only be accessible to the server party, thereby reducing the risk.
Since we already provide TAResSFL to mitigate the MIA attack that can be possibly launched from the honest-but-curious server, the risk is minimized.
To eliminate possibility of future advanced attack, we can introduce a protocol that requests the server party to regularly delete 3 these latent vectors immediately after the training is done.
R EPRODUCIBILITY S TATEMENT
To make it easier for readers to reproduce the results in this work, we have provided detailed explanation on the MocoSFL training and evaluation hyperparameters, collaborative learning system hyperparameters in Appendix A.1, and TAResSFL module hyperparameters in Appendix A.4.
Moreover, we plan to release our code upon acceptance.
A T ECHNICAL D ETAILS A. H YPER - PARAMETER C HOICES
Training Hyper-parameters.
For data preprocessing, we split each dataset into 50K training data and 10K validation data.
We use the training data directly as query data X q , and its augmented version as positive data X k + .
We use the same augmentation pipeline as Chen et al.
(2020b).
For the backbone model, we remove the last fully-connected layer and replace it with a MLP head consisting of two larger fully connected layers of 1024 neurons.
To fit well with small batch size training, we replace the batch normalization layers by group normalization layers and set the number of channels per group to 4 while keeping the minimum number of groups to 1 if the number of channels is less than 4.
We also use the weight standardization technique (Qiao et al., 2019) in combination with group normalization for better performance (Kolesnikov et al., 2020).
At the server-side, we use a feature memory of 6,000 negative keys that is implemented as a First-InFirst-Out (FIFO) queue, and we update it by pushing newly generated positive keys at every training step.
For contrastive loss calculation, we use the symmetric contrastive loss calculation introduced in
Moco-V3 (Chen et al., 2021), where the position of query and positive keys are swapped in Eq.
(1), and the new loss takes the form of ( L Q , K, N + L K , Q, N ) / 2 .
For model updating, we update the online model of both client-side and server-side model with the SGD optimizer with 0.06 initial learning rate, 0.9 momentum and 0.0005 weight decay.
We schedule the learning rate using the cosine annealing trick for both client-side and server-side models.
The momentum model update is done through the moving average mechanism in He et al.
According to previous study Chen et al.
(2020b), the training accuracy converges with extensive long training (i.e. > 800 epochs), however, such long training session is impractical for collaborative learning setting.
Thus, we set the number of epochs to 200 to maximize time and performance tradeoff.
At the end of each epoch, we save the model and perform a k-nearest-neighbor (knn) validation using the 20% validation dataset.
Finally, we use the saved model that has the best knn accuracy to perform the final linear probe evaluation.
Evaluation Hyper-parameters.
For the knn validation, we feed the validation dataset and use the normalized feature output of the backbone (together with the MLP head) and follow the implementation in Wu et al.
(2018) to get the accuracy.
For the linear probe evaluation, we follow exactly the same procedure as in Zhuang et al.
(2022), where we replace the MLP head with a randomly initialized fully-connected layer, fine-tune the new model on the labeled dataset while keeping the backbone parameters frozen.
We train the single fully-connected layer for a total of 100 epochs on the labeled training dataset and use the accuracy on the labeled validation dataset as final accuracy.
To optimize the single FC layer, we use a batch size of 128 and Adam optimizer with default learning rate 0.001 and use cosine annealing to schedule its learning rate.
Collaborative Learning Hyper-parameters: Synchronization Frequency and Client Sampling
Ratio.
In our proposed MocoSFL scheme, we use an auto-adjust mechanism for adjusting the synchronization frequency, the client-sampling ratio, and the local batch size according to number of clients.
The auto-adjust mechanism aims to strike a balance between model divergence and sample hardness and is based on two principles (also supported by empirical results shown in Tableand Table 8).
The first principle is to always keep the server-side batch size around 100, so that we can keep a good hardness (indicated by Eq.
For example, for 5-client MocoSFL, we force each client to have a batch size of 20 to keep the equivalent batch size to 100.
For 1,000-client
MocoSFL, we use a client sampling ratio of 0.1 to let 100 clients join in each round.
The second principle is that the synchronization frequency must be adjusted according to the number of local training steps.
The reason is that more local updates leads to higher model divergence, which needs to be mitigated using more frequent synchronization.
For example, for a 100-client MocoSFL where each client performs 500 local training steps, accuracy drops for a synchronization frequency of “1/epoch”, and accuracy increases until “10/epoch”.
However, for a 1,000-client MocoSFL where each client has 50 data and performs 50 local training steps in each epoch (with batch size of 1), setting synchronization frequency to “2/epoch” is sufficient.
We provide further evidence of the above two principles in Appendix B.3.
TAResSFL Hyper-parameters.
For the Hyperparameter choices in TAResSFL, please refer to Appendix A.4.
A. SFL-V1 P ROCESS
Require:
For N C clients, instantiate private training data ( X i , Y i ) for 1 , 2 , ..., N C .
Each clientside model C i has L layers and server-side model S contains the remaining layers.
1: for epoch t ← 1 to num epochs do 2:
C ∗ = 1 N C (cid:80) N C i =
1 C i ; C
i ← C ∗ for all i { Model Synchronization } 3: for step s ← 1 to num batches do 4: for client i ← 1 to N C in Parallel do 5: data batch ( x i , y i )
← ( X i , Y i ) 6:
A i =
C i ( W C i ; x i ) { Send A i to Server } 7: end for 8:
A = cat ( A 1 , A 2 , ..., A N C ) ; y = cat ( y 1 , y 2 , ..., y N C ) 9:
L = L CE ( S ( W S ; A ) , y ) 10: ∇ A L ← back-propagation { Partition ∇ A L , Send ∇ A i L Correspondingly to Client i } 11:
Update W S ; 12: for client i ← 1 to N C in Parallel do 13: ∇ x i L ← back-propagation 14:
Update W C i ; 15: end for 16: end for 17: end for
The training process of SFL-V1 in supervised learning is shown in Algorithm 1.
Since a large equivalent batch size at server side is essential for the success of contrastive learning, we adopt
SFL-V1.
Here a large equivalent batch size (equal to the number of clients times clients’ batch size) can be achieved due to latent vector concatenation as shown in line 8 of Algorithm 1.
We choose
SFL-V1 over SFL-V2 because: V1 performs concatenation of clients’ latent vectors and feeds it to
S and updates the model only one time (call “optimizer.step() for one time” in pytorch), while V sequentially feeds them and updates S for a total of N C times, where N C is the number of clients.
In
PyTorch, this means calling “optimizer.step() for N C time”.
A. D ATA P RIVACY : M ODEL INVERSION ATTACK
The MIA workflow is shown in Fig.
It follows the same procedure as previous works on the data privacy of SFL (Vepakomma et al., 2020; Li et al., 2022).
Threat Model.
The basic assumption is that the server follows the honest-but-curious assumption, that is, it follows the protocol strictly, but can gather information during this process to breach the privacy.
We also assume the server party can access the 10K validation dataset since validation needs to be done by the server for monitoring purpose (Bhagoji et al., 2019).
Attack Preparation.
To perform MIA, the server needs to instantiate an inversion model.
We assume that the server use the L3 inversion model (top-tier inversion model in Li et al.
as shown in Fig.
Then, the server queries the client-side model using the 10K validation dataset.
to gather enough input-output pairs.
These pairs are used to train the initialized inversion model, where we use mean-squre-error (MSE) loss with Adam optimizer with 0.001 learning rate for 50 epochs and set the batch size to 32.
Attack Evaluation.
Once the inversion model is ready, server can now use it on the latent vectors sent by the target client to generate the reconstructed image.
We use a fixed set of 128 images that are randomly sampled from the training dataset to represent the private data of the target client, and use the MSE metric between the ground-truth image and reconstructed image to evaluate the data privacy (higher the MSE, higher the data privacy).
MocoSFL + TAResSFL
MocoSFL
Ground-truth
[Figure]
Client’s View:
Server’s View: (Honest-butcurious)
Teaser
[Figure]
A. TAR ES SFL MODULE
The MocoSFL+TAResSFL scheme is depicted in Fig.
It consists of a pretraining step and a transfer step.
The default architecture is modified by introducing a pair of bottleneck layers (the last layer of “C-model” and the first layer of the tail model).
The pretraining step is done offline at
Tail momentum
InfoNCE
Loss
Tail online 𝑲 + 𝑸 𝑵 Server
C-model momentum
C-model online 𝑿 𝑺,𝒌 + ∗ 𝑿 𝑺,𝒒∗
Augmentation
Server momentum
InfoNCE
Loss
Server online 𝑲 + 𝑸 𝑵
DeQueue
EnQueue 𝑲 +
Server 𝑨 𝒌 + 𝑨 𝒒 𝑨 𝒌 + ,𝒊
Pretrained C-model 𝑿 𝒌 + ,𝒊 𝑿 𝒒,𝒊
Client 𝒊 𝑿 𝒊 Augmentation 𝑨 𝒒,𝒊
Wireless communication 𝑨 𝒌 + ,𝒋
Pretrained C-model 𝑿 𝒌 + ,𝒋 𝑿 𝒒,𝒋
Client 𝒋 𝑿 𝒋 Augmentation 𝑨 𝒒,𝒋 +
Regularization 𝑿 𝑺 * = { 𝑿 𝑺 }∪ {𝒑𝑿 𝒕 } (𝒑 ≪ 𝟏)
No Sync is needed
Freeze
Freeze
L3 Inversion Model MSE 𝑨 𝑺,𝒒 ∗ 𝑨 𝑺,𝒌 + ∗ 𝑿
[Figure]
the server and the transfer step is the actual MocoSFL collaborative learning step, where clients use the pretrained “C-model” as the initial client-side model and participate in the SFL-based training to protect data privacy from MIA attacks.
In the pretraining step in Fig.
11 (a), the input X ∗ S is the mixture of source dataset X S (an out-of-distribution dataset/auxiliary dataset) and a small proportion p of target dataset X S .
The optimization process follows Eq.
(5), whose objective is to make “Cmodel” have enough resistance to MIA attack while keeping a high accuracy.
In the transfer step in Fig.
11 (b), all client-side models are initialized using the pretrained “C-model” and kept frozen.
All clients feed augmented images to “C-model” locally and send latent vectors to the server to perform the MocoSFL training process.
This design reduces communication overhead dramatically thanks to bottleneck layers.
It also reduces computation and memory since client-side model’s backward computation and synchronization costs are saved.
TAResSFL hyperparameters.
For the pretraining step, we use a single-client MocoSFL scheme to simulate the offline server pretraining process.
We use batch size of 128 with learning rate of 0.
that is scheduled by cosine annealing.
We add a pair of bottleneck layers, both with 3x3 kernel size.
The first bottleneck layer is a Conv2D layer that has output channel size of 4 and stride of 2 and is placed at the end of client-side model as part of “C-model”, the second one is a Transpose Conv2d layer (“torch.nn.ConvTranspose2d” in pytorch), having a stride of 2 that is placed at the front end of the tail model.
The training data is a combination of source data and a small proportion of target data.
For example, we use CIFAR-100 as the source data and combine it with 0.0%, 0.5%, and 1.0% randomly sampled target data to derive the training dataset of the pretraining step.
Then, we use MSE loss to optimize the inversion model and use Structural Similarity Index (SSIM) loss of the inversion model as the regularization term of the contrastive loss.
We set λ to 2.0 and SSIM threshold to 0.6.
If
SSIM is lower than 0.6, then we disable the regularization term temporarily, until it is higher than 0.
again.
to stabilize the training.
As described earlier, we perform one inversion model training step and one classification model (C-model + tail model) training step for every source data batch.
The number of epochs for the pretraining step is set to 200.
For the transfer step, we initialize the client-side model using the “C-model” derived in the pretraining step and initialize the server-side model using the “tail model”.
During training, the client-side model update is disabled (keep client-side model frozen), and the server-side model is optimized using SGD with a small learning rate of 0.001, to be able to achieve high accuracy on the target data.
We also remove client-side momentum models, the client-side model backward operation and the client-side models’ synchronization step since they are not required for frozen client-side models.
The number of epochs for the transfer step is set to 200.
A. C OMMUNICATION , C OMPUTATION AND M EMORY O VERHEAD C ALCULATION
For communication overhead calculation, we divide it into two parts, which are latent vector size and weight size.
The latent vector size is determined by the output size of client-side model, and the weight size is determined by the parameter size of client-side model.
In each epoch, weight communication is weight size multiplies by the number of synchronization times per epoch, while latent vector communication is latent vector size multiplied by the total number of data sent in this epoch.
For FL, we only need to consider the weight communication.
For MocoSFL without TAResSFL module, we need to consider the weight communication and also double the latent vector communication since clients receive the gradients sent back by the server.
For MocoSFL with TAResSFL module, we do not double the latent vector communication because gradients are not needed, also, the weight communication is zero since synchronization is not needed.
Finally, we multiply communication overhead per epoch by the total number of epochs.
For computation overhead, we use FLOPs to represent since floating-point operation dominates the computation.
The FLOPs of the models were counted by the ‘fvcore’ library developed by Facebook
AI Research.
To count the inference FLOPs, we multiply the output of ‘fvcore.nn.
FlopCountAnalysis’ on a single image by the number of data per client and then multiply it with the total number of epochs.
Due to symmetry, the FLOPs of model backpropagation are approximately equal to that of the model inference.
And we also need to count the momentum model inference.
Thus, the FLOPs number we show for training is three times of the model inference FLOPs.
For memory overhead, we use ‘torch.cuda.memory allocated’ API to measure the allocated memory for training and inference.
B E XTENSIVE E MPIRICAL R ESULTS
In this section, we perform extensive empirical evidence of the proposed MocoSFL scheme on different architecture (Appendix B.1), evidence of two hyperparameter choice principles (Appendix B.3), performance evaluation of TAResSFL with different hyperparameters (Appendix B.4), performance evaluation of MocoSFL in a semi-supervised learning application (Appendix B.6), and extensive comparison with other schemes.
B. R ES N ET -50 RESULTS
We follow the same setup as in Table 2 and Table 3 on ResNet-50 architecture for cross-silo (5 andclients) and cross-client settings (100 clients).
Table 5 shows that compared with ResNet-18 model,
ResNet-50 has ∼ 3% better accuracy performance.
[Table]
B. M OBILE N ET V2 AND VGG-13 CROSS - CLIENT RESULTS
We follow the same setup as in Table 2 and Table 3 on MobileNet-V2, VGG-13 architecture for cross-client settings (100 clients).
Table 6 shows that compared with ResNet-18 model, ResNethas ∼ 3% better accuracy performance
[Table]
E FFECT OF BATCH SIZE , SYNCHRONIZATION FREQUENCY AND FEATURE SHARING
We provide some empirical evidence to support the two principles presented in Appendix A.1.
The first is keeping the batch size to above 100.
Table 7 provides the accuracy performance when the number of clients is set to 5 for different server-side batch size B (after vector concatenation).
So
B = 50 corresponds to local batch size of 10 since there are 5 clients.
A small batch size has much worse performance (around 5% lower), while a large batch size (i.e., 100, 200) helps keep a good hardness and maintains good accuracy.
However, when batch size increases further to 400, we notice small degradation in accuracy.
Thus, as stated in the first principle [Table] when the number of clients is larger than 200 and each client locally uses a batch size of 1 (resulting in a batch size of 200), we have to apply client sampling to force a smaller total batch size.
[Table]
We also investigate the necessity of synchronization frequency when the number of clients increases.
While previous experiments (Fig. 5) demonstrate significant benefit of increasing synchronization frequency, we find that for a cross-client application where each client has very small amount of data, increasing synchronization frequency does not have much impact on the accuracy.
Tableillustrates this finding.
The accuracy performance quickly saturates at “2/epoch” and shows no further improvement when frequency is increased further.
This is because the number of local training steps becomes less as stated in second principle (see Appendix A.1).
[Table]
Next, we investigate the effectiveness of feature sharing in MocoSFL.
In MocoSFL, the negative keys from different clients are shared.
If such sharing is disabled and server stores keys contributed from different clients separately in different feature memories, the accuracy is significantly lower.
As shown in Fig.
12, without feature sharing, both IID and non-IID accuracy drop significantly.
B. D IFFERENT TAR ES SFL SETTINGS
We vary the hyperparameter settings of the proposed “MocoSFL + TAResSFL” scheme on ResNeton CIFAR-10 dataset.
In Table 9, we change the source dataset to SVHN.
Here, we observe that using SVHN as source data gets worse accuracy performance than using CIFAR-100.
This is because the domain difference between SVHN and CIFAR-10 (target dataset) is much larger than CIFARand CIFAR-10.
However, even with such domain difference, the proposed method still works.
For
MocoSFL with cut-layer of 3, we can still achieve over 80% accuracy with 0.034 MSE.
In Table 10, we vary the bottleneck layer settings, where we set the bottleneck layer size to C 2 S 2 ,
C 4 S 2 and C 8 S 2 .
A smaller channel size leads to better “bottlenecking” effect that is, lower communication overhead, better resistance to attack, but lower accuracy performance.
For MocoSFL3, we observe that C 2 S 2 achieves the highest resistance with lowest accuracy, and using C 8 Scannot meet the required resistance.
We find that using C 4 S 2 is the best choice in our specific case.
In Table 11, we vary the regularization strength λ .
The λ affects the gradient magnitude contributed by the regularization; using higher λ can boost the resistance but may lead to accuracy drop.
However, we observe resistance almost saturates when λ = 2 . 0 and hence use this setting across our experiments.
Batch Size Online Momentum Q
K +
N hard N hard N easy N easy Feature Space
[Table] [Table] [Figure]
[Table] [Figure] Number of Clients [Table]
NN
[-Batch Size FL-BYOL FL-MoCoV
-]
[Figure]
cc
[Figure]
c y
[Table]
[Figure] [Table] [Figure]
[Table]
A cc u r a c y (
[Figure]
[Figure] [Figure]
[Table] [Table] [Table]
as training data, with different regularization strength λ . B. E XTENSIVE I MAGE Q UALITY M ETRICS FOR TAR ES SFL In Appendix B.
[Table]
we report other image quality metrics Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR) metrics for Table
[Table]
B. S EMI - SUPERVISED L EARNING E VALUATION
We report the performance of using the proposed MocoSFL for semi-supervised learning applications for both cross-silo (5 and 20 clients) and cross-client setting (100 clients).
As shown in Table 12, the proposed MocoSFL achieves competitive accuracy performance.
For ResNet-18 model, we achieve 10% higher accuracy than FL-BYOL (Zhuang et al., 2022) for the semi-supervised learning setting where only 1% label is given on a CIFAR-10 dataset
[Table]
B. E XTENSIVE C OMPARISON WITH O THER S CHEMES
We provide a comprehensive comparison with prior SoTA works (Zhuang et al., 2021; 2022) in Table 13.
Our proposed MocoSFL scheme shows significantly better accuracy performance, especially for ResNet-50 model, where we achieve 5% higher accuracy than Zhuang et al.
(2022) on all metrics (linear probe and semi-supervised with 1% and 10% labels).
[Table]