M OCO SFL: ENABLING CROSS - CLIENT COLLABORATIVE SELF - SUPERVISED LEARNING
Daisuke Sony michael spranger
A BSTRACT
Existing collaborative self-supervised learning (SSL) schemes are not suitable for cross-client applications because of their expensive computation and large local data requirements.
To address these issues, we propose MocoSFL, a collaborative SSL framework based on Split Federated Learning (SFL) and Momentum
Contrast (MoCo).
In MocoSFL, the large backbone model is split into a small client-side model and a large server-side model, and only the small client-side model is processed locally on the client’s local devices.
MocoSFL has three key components: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and volume of local data; (iii) frequent synchronization that helps achieve better non-IID performance because of smaller local model divergence.
For a 1,000-client case with non-IID data (each client only has data from 2 random classes of CIFAR-10),
MocoSFL can achieve over 84% accuracy with ResNet-18 model.
Next we present
TAResSFL module that significantly improves the resistance to privacy threats and communication overhead with small sacrifice in accuracy for a MocoSFL system.
On a Raspberry Pi 4B device, the MocoSFL-based scheme requires less than 1 MB of memory and less than 40 MB of communication, and consumes less than 5W power.
The code is available at https: github. com/ SonyAI/MocoSFL .
I NTRODUCTION
Collaborative learning schemes have become increasingly popular, as clients can train their own local models without sharing their private local data.
Current collaborative learning applications mostly focus on supervised learning applications where labels are available (Hard et al., 2018; Roth et al., 2020).
However, availability of fully-labeled data may not be practical since labeling requires expertise and can be difficult to execute, especially for the average client.
Federated learning (FL) (McMahan et al., 2017) is the most popular collaborative learning framework.
One representative algorithm is “FedAvg” , where clients send their local copies of the model to the server and the server performs a weighted average operation (weight depends on the amount of data) to get a new global model.
FL has achieved great success in supervised learning, and has been used successfully in a wide range of applications, such as next word prediction McMahan et al.
visual object detection for safety Liu et al.
(2020), recommendation Wu et al.
(2022a;b), graph-based analysis Chen et al.
; Wu et al.
(2022c), etc.
For collaborative learning on unlabeled data, prior works (Zhang et al., 2020; Zhuang et al., 2021; 2022) combine FL scheme with classic self-supervised learning (SSL) methods such as BYOL (Grill et al., 2020) and Moco (He et al., 2020).
These methods can all achieve good performance when clients’ data is Independent and Identically Distributed (IID) but suffer from poor performance [Figure] cc [Figure]
a
[Figure]
y [Figure] in non-IID cases. Recently, Zhuang et al.
(2022) mitigated the non-IID performance drop with divergence-aware aggregation technique and provided state-of-the-art (SoTA) accuracy performance using a combination of FL and different SSL methods.
However, these SoTA FL-SSL schemes are not practical for cross-client applications.
First , FLSSL imposes a significant computation overhead and large memory requirement on clients.
This is because SSL requires a large backbone architecture (Chen et al., 2020a) together with a large batch size to ensure good performance.
As shown in Fig. 1(a), accuracy drops dramatically when batch size is low for both FL-SSL methods.
Second , FL-SSL schemes fail to maintain a good accuracy when the number of clients is large (cross-client cases), as shown in Fig. 1(b).
For a dataset with fixed size, when the number of clients increases, data per client decreases, resulting in accuracy degradation.
The drop in accuracy is mainly because of the failure to meet data size requirement in performing contrastive learning.
Zhang et al.
; Wu et al.
(2021) attempt to address this issue in
FL-SSL with remote feature sharing.
However, this introduces a high communication overhead due to client-to-client feature memory synchronization; for a 100-client system, it costs around 2.46GB per synchronization per client.
To solve these challenges, we propose MocoSFL, a scheme based on Split Federated Learning (SFL) Thapa et al.
(2020) that incorporates the feature memory bank and momentum model designs of MoCo (He et al., 2020).
We adopt the SFL scheme for three reasons: (i) SFL utilizes a smaller client-side model and so reduces the computation overhead and has lower memory consumption and model parameters; (ii) SFL’s latent vector concatenation enables a large equivalent batch size for the centralized server-side model, making micro-batch training possible for clients and thus reducing client’s local memory; (iii) When combined with MoCo’s key-storing mechanism,
SFL’s shared server-side model enables effective feature sharing, which removes the requirement of large amount of local data and makes the scheme possible for cross-client applications.
As a result,
MocoSFL achieves good accuracy with ultra-low memory requirements and computation overhead, and can support a very large number of clients.
MocoSFL shows better non-IID performance since local model divergence is smaller.
However, the use of SFL brings extra communication overhead as well as data privacy concerns.
Thus, we present target-aware ResSFL (TAResSFL) module as an effective solution to mitigate these issues with small accuracy drop.
Our main contributions are:
• We identify two major challenges in deriving high accuracy in FL-SSL schemes for crossclient applications.
These are availability of a large amount of data that is required for contrastive learning and the ability to process them in clients who may not have sufficient hardware resources.
• We propose MocoSFL, an SFL-based scheme to address the two challenges.
MocoSFL uses a small client-side model, latent vector concatenation and feature sharing.
For cross-client case, MocoSFL is the only viable and practical solution.
For cross-silo case, MocoSFL can achieve even better performance than SoTA FL-SSL schemes under non-IID setting because of smaller model divergence.
• To address communication overhead and privacy issues that are inherent to SFL-based schemes, we propose target-domain-aware ResSFL (TAResSFL) that effectively reduces the communication cost and mitigates model inversion attacks.
B ACKGROUND 2. S ELF -S UPERVISED L EARNING
To learn from unlabeled data, SSL schemes based on contrastive learning such as SimCLR Chen et al.
(2020a), BYOL Grill et al.
Simsiam Chen and He (2021) and MoCo He et al.
(2020) have achieved great performance on popular benchmarks.
Unlike other schemes (BYOL, SimCLR, etc.)
that use other samples in the current data-batch as negative keys, MoCo uses previously computed positive keys as negative keys, and stores current positive keys in the feature memory for future iterations.
The key-storing mechanism results in a relatively smaller batch size requirement that is beneficial for reducing device memory and also makes it easy for feature sharing implementations for our proposed scheme.
For the loss function, MoCo relies on InfoNCE loss Oord et al.
as the contrastive mechanism to update its model parameters:
[Equation]
(1) where query key Q and positive key K + are the output vectors of server-side momentum model and the momentum model, respectively, obtained by processing two augmented views of the image.
N denotes all negative keys in the feature memory of size M .
Importantly, the success of MoCo scheme highly depends on the “hardness” of its negative keys (Kalantidis et al., 2020; Robinson et al., 2020).
The “hardness” of a negative key N in the feature memory bank, can be determined by the similarity (inner-product) between Q t (at step t ) and N ; the smaller the similarity, the better the “hardness”.
We notice the “hardness” of negative key N reduces quickly because model updates are in the direction of minimizing the InfoNCE loss in Eq.
As a result, MoCo adopts a slow-changing momentum model to produce consistent negative keys to add to the feature memory at the end of every training step and thereby maintains their hardness.
2. S PLIT F EDERATED L EARNING
Split Federated Learning (SFL) Thapa et al.
(2020) is a recent collaborative learning scheme that focuses on high computation efficiency at the client side.
It splits the original model architecture into two parts, the client-side model that contains all layers upto the “cut-layer” and the server-side model that contains the remaining layers.
We distribute copies of client-side model C i to client- i ’s local devices and instantiate the server-side model S in a cloud server.
To complete each training step, clients need to send the latent vectors (the output of client-side model) to the server, and the server processes latent vectors, computes the loss, performs backward propagation and returns the corresponding gradients to clients.
Thapa et al.
(2020) present two possible ways for server to process latent vectors sent by clients.
In this paper, we use SFL-V1 where the server concatenates all clients’ latent vectors and processes them altogether, which makes the equivalent batch size larger at the server and benefits contrastive learning.
In contrast, in SFL-V2, client’s latent vectors are processed sequentially in a first-come-first-serve manner and thus does not benefit from the “large batch”.
We provide details of SFL-V1 in Appendix A.2.
M OTIVATION
As mentioned in Section 1, the two challenges in extending FL-SSL to cross-client applications are high computing resource requirement and large data requirement.
3. H IGH COMPUTING RESOURCE REQUIREMENT
The first challenge is the computing resource requirement of training an SSL model locally.
Using a compact backbone model may be accurate for supervised learning, but is not suitable for SSL as it has a much higher requirement on the model capacity.
(Shi et al., 2021; Fang et al., 2021) show that ) suffer from over 10% accuracy degradation compared to a larger ResNet-18 architecture, while an even larger ResNet-50 model has over 15% better accuracy compared to ResNet-18 on ImageNet dataset (Deng et al., 2009).
This means memory requirement for training an SSL model with high accuracy is very high.
Using a smaller batch size reduces accuracy dramatically, as shown in Fig. 1(a), and is thus not an option.
A FL-SSL scheme (ResNet-18 with a batch size of 128) costs 590.
MFLOPs per image and over 1100MB of memory per client, which is not practical.
3. L ARGE DATA REQUIREMENT
The other major difficulty for FL-SSL to generalize to the cross-client case is the large data requirement.
For cross-client applications, the amount of data available to each client can be very limited.
For example, in a cross-silo medical application, a client can be a hospital with tons of data.
In comparison, in a cross-client application, a client can be a patient who has limited amount of data.
The root of the problem lies in the difficulty to find hard negative samples when clients do not have enough local data.
When the amount of data is larger, the chance for hard negative samples to be present becomes much higher.
As a result, existing FL-SSL can only be successful for cross-silo applications where clients have large amount of data and can perform effective contrastive learning locally.
As demonstrated in Fig. 1(b), we observe high accuracy when clients have 10K samples of data, while the accuracy drops quickly to around 30% when clients have only 500 samples.
Unfortunately, remote feature sharing Zhang et al.
; Wu et al.
(2021; 2022d) in FL-SSL schemes cannot solve the large data requirement.
These schemes update the shared feature memory less frequently because of significant communication overhead with each update.
Since clients must synchronize their local copies of the shared feature memory each time a minor change happens, in Zhang et al.
; Wu et al. (2022d), new negative keys are added to the feature memory only once per epoch.
Even so, the total communication overhead of remote feature sharing scales quadratically with the number of clients makes it not practical for cross-client case.
M ETHOD 4. P ROPOSED M OCO SFL
Our proposed MocoSFL is an innovative combination of SFL-V1 and MoCo-V2 (Chen et al., 2020b) as shown in Fig.
There are three key components.
First , in each training step, the latent vectors sent by all clients are concatenated before being processed by the server-side model.
This helps achieve a large equivalent batch size in order to support mini-batch training.
Second , we use a shared feature memory which is updated by positive keys contributed by all clients in every training step.
Third , we improve the non-IID performance by using a higher synchronization frequency.
Next, we will elaborate on how these three components in the proposed MocoSFL address the two challenges in Section 4.2 and Section 4.3.
Section 4.4 describes how MocoSFL achieves better nonIID performance and Section 4.5 addresses the privacy and communication issues of the proposed scheme.
4. R EDUCE H ARDWARE R ESOURCE R EQUIREMENT
Choice of SFL helps reduce computational overhead and memory consumption at the client-end because of the much smaller client-side model.
For example, on a CIFAR-10 ResNet-18 model with a batch size of 128, a client-side model with 3 layers only costs 13.7% of the FLOPs compared to the entire model, and its memory cost is 227MB, merely one fourth of the entire model.
Furthermore, we reduce the batch size to 1 (also known as “micro-batch”), to further reduce the memory consumption.
The use of micro-batch in local model training is only possible thanks to the latent vector concatenation mechanism which basically aggregates latent vectors sent by all clients into a big batch before sending it to the server.
In addition, in a micro-batch setting, we replace the batch normalization layer by group normalization Wu and He (2018) and weight standardization (Qiao et al., 2019) to gain better accuracy performance.
In Fig. 3(b), we compare the computation and memory consumption of the proposed MocoSFL with the FL-SSL scheme.
MocoSFL with cut-layer of 3 achieves ∼ 288 ×
Wireless communication
Server momentum
InfoNCE
Loss
MocoSFL Scheme
Server online 𝑲 + 𝑸
Shared Feature Memory 𝑵
DeQueue
EnQueue 𝑲 +
Server
Client 𝑗 momentum 𝑨 𝒌 + ,𝒋
Client 𝑗 online 𝑿 𝒌 + ,𝒋 𝑿 𝒒,𝒋
Client 𝒋
Augmentation 𝑨 𝒒,𝒋 𝑨 𝒌 + 𝑨 𝒒
Frequent Sync
Concatenate
Concatenate
Client 𝑖 momentum 𝑨 𝒌 + ,𝒊
Client 𝑖 online 𝑿 𝒌 + ,𝒊 𝑿 𝒒,𝒊
Client 𝒊 𝑿 𝒊
Augmentation 𝑨 𝒒,𝒊 𝑿 𝒋
[Figure]
reduction in memory consumption than FL-SSL and has 2%-10% higher accuracy.
Details of accuracy evaluation are included in Section 5.1.
4. M ITIGATE L ARGE D ATA R EQUIREMENT
As indicated by (Kalantidis et al., 2020; Robinson et al., 2020), the “hardness” of a negative key N heavily depends on its similarity with the current query key Q , given that N and Q have different ground-truth labels.
To evaluate the hardness of negative key N 0 residing in the feature memory, we use the similarity measure (inner-product) between N 0 and Q t , a freshly calculated query key at time t .
In FL-SSL with feature sharing, the negative key is only updated after a long period of time to reduce communication overhead.
As a result, the hardness diminishes quickly.
In contrast, MocoSFL frequently updates its feature memory to maintain a good hardness.
At every training step, a freshly calculated positive key K + is added to the tail of the queue, and the oldest one is popped out.
However, frequent updates of feature memory is not enough to ensure a high level of hardness.
We also find it is necessary to use a large batch size and a large feature memory.
This finding agrees with the study in Bulat et al.
and also explains the accuracy drop for a small batch size in
FL-MocoV2 in Fig.
1(a).
To illustrate this, we consider the total similarity measure at time t and make the following assumptions:
For the newest batch of negative keys N t = K t − 1 of size B in the feature memory at time t , we assume the similarity measure between N t and Q t is a constant η for all t .
We also assume similarity of every batch of negative keys in feature memory gets reduced by a constant factor γ ( γ < 1 ) after each model update to represent the degradation caused by model updates.
Thus, for a freshly computed query Q t , its total similarity measure with negative keys in the feature memory can be represented as:
[Equation]
where B is the batch size and M is the feature memory size.
We see that using a large batch size
B is beneficial as it helps bring more fresh negative keys and maintain better hardness.
Also, using a larger feature memory (increasing M ) can keep enough negative keys and contribute to a better total hardness.
In the cross-client case, FL-SSL schemes can hardly meet these two requirements because of the small clients’ memory.
Nonetheless, MocoSFL can easily fulfill them because (1) latent vector concatenation enables a large equivalent batch size, and (2) feature memory hosted by the cloud server can be much larger.
4. I MPROVING N ON -IID P ERFORMANCE
We found that use of SFL results in fewer model parameters at the client side and hence smaller model divergence.
Furthermore, introducing frequent synchronization in MocoSFL provides additional
F L - S S L
M o c o S F L - 5 M o c o S F L - 3 M o c o S F L - 1  91
D i v e r g e n c e
S y n c F r e q = 1 S y n c F r e q =
S y n c F r e q = 1(a) 1 6 2 7 .9 2 1 .6 .0 .
M F L O P s / i m a g e
M e m o r y
( M B )4 08 01 2 01 6
M e t r i c s
F L - S S L
M o c o S F LM o c o S F L -
[Figure]
reduction in model divergence and greatly improves the non-IID performance. According to Zhang et al. ; Zhuang et al. (2021; 2022), the model divergence between two models is calculated as the L2 norm of the weight difference.
Following the same idea, the total divergence measure of a cross-client system can be measured as the average weight divergence of local models w.r.t.
the global model during training:
[Equation]
where L denotes the number of layers in the client-side model, E denotes the total number of synchronizations, N C denotes the number of clients, and l, e, i are the respective indices for L, E, N C .
W ∗ is the average of all client models W i .
MocoSFL reduces the model divergence with two orthogonal mechanisms.
The first mechanism is the reduction of client-side model size, which directly results in a lower model divergence.
As shown in Fig. 3(a), compared to FL-SSL scheme,
MocoSFL has a much lower model divergence when the client-side model has less than 5 layers.
The other mechanism is frequent model synchronization which helps reduce the model divergence.
This is only possible in SFL because of communication overhead of sending weights in a smaller client-side model, is smaller.
Fig.
3(a) also illustrates how model divergence further reduces as we increase the synchronization frequency.
4. I MPROVING P RIVACY AND C OMMUNICATION O VERHEAD OF M OCO SFL
The proposed MocoSFL scheme is based on SFL and suffers from two issues – high overall communication overhead due to transmitting and receiving latent vectors and vulnerability to Model Inversion Attack (MIA)
Fredrikson et al.
(2015), in which the server can reconstruct clients’ raw inputs from latent vectors, making clients’ data privacy questionable.
(We leave the details of its threat model and working mechanism in Appendix A.3)
To address the privacy and communication issues in MocoSFL, we propose Target-Aware-ResSFL (TAResSFL).
TAResSFL extends ResSFL Li et al.
(2022) for self-supervised learning through: (1) target-data-aware self-supervised pre-training, and (2) freezing feature extractor during SFL training.
TAResSFL also utilizes the bottleneck layer design from ResSFL to reduce the communication overhead.
In ResSFL Li et al.
, the server performs pretraining to build up the resistance to MIA using data from a different domain since it does not have access to clients’ data.
Next, the pretrained resistant client-side model is transferred to the clients and gets fine-tuned using SFL.
TAResSFL improves the pretraining step by assuming that the server can get access to a small subset ( < 1%) of training data, together with large amount of data from another domain, and perform pre-training using self-supervised learning.
Such a pretrained client-side model has better transferability, and can thus stay frozen during SFL process, thereby avoiding the expensive fine-tuning.
As shown in Fig.
4, we blend the source dataset X s with a tiny subset of target dataset X t during the feature extractor training.
The attacker-aware training has the min-max form:
[Equation]
where R in the inner maximization denotes a similarity measure, for which we use the structural similarity index (SSIM) score Zhao et al.
The inner maximization is used to train the simulated attack model G , whose function is reconstructing the activation back to a raw input state that is similar with ground-truth X q .
The outer minimization step goes in the direction of lowering contrastive loss, where the regularization term makes the model accurate as well as resistant to attack.
These two steps are done alternatively to make the feature extractor resistant, and also be able to achieve good accuracy on the target dataset.
Fig.
4 presents the Target-aware ResSFL scheme.
We use the resistant feature extractor to initialize client-side models during transfer step, as shown by the pink arrow in Fig. 4.
Unlike ResSFL, here we freeze its parameter completely to maintain the resistance since any parameter change can cause resistance drop.
However, freezing brings a noticeable accuracy drop of larger than 3%, even with the use of CIFAR-100 as source dataset if we perform pre-training by only using the source dataset (see Table 4).
But if we blend the source data with a small portion of target data during pre-training, the accuracy can be greatly improved.
Since the model accuracy validation is done by the server for monitoring purpose (Bhagoji et al., 2019), we believe that the server can separate out a small proportion of validation data to meet the target data availability assumption.
The freezing also greatly benefits hardware resource requirement since: (1) clients only need to transmit the latent vectors to the server and do not need to perform backward propagation using gradients from the server; (2) client-side model synchronization is not needed.
As a result, MocoSFL with TAResSFL component achieves a ∼ 128 × (=5001.4/39.1) overall communication reduction compared to the original MocoSFL, as shown in Table 1.
Here MocoSFL methods undergomodel synchronizations for 200 epochs while FL-SSL methods need 100 synchronizations (using the same setting in Zhuang et al. [Equation] MocoSFL-TAResSFL Scheme [Equation]
Feature Extractor 𝑨 𝒕,𝒌+ 𝑨 𝒔,𝒌+
Tail model + Contrastive head 𝑸 𝒕 ,𝑲 𝒕 + 𝑸 𝒔 ,𝑲 𝒔+ 𝑨 𝒕,𝒒 𝑨 𝒔,𝒒 𝑿 𝒕,𝒌+ 𝑿 𝒔,𝒌+ 𝑿 𝒕,𝒒 𝑿 𝒔,𝒒
v
Simulated Attacker 𝑿 𝒕,𝒒∗ 𝑨 𝒕,𝒒 𝐼𝑛𝑓𝑜𝑁𝐶𝐸𝑙𝑜𝑠𝑠 v
Resistant Feature Extractor initialize ClientsOnline models Freeze ❷ ❶ 𝑿 𝒔,𝒒∗ 𝑨 𝒔,𝒒 Server Battery TCP Client     3.
Computation(GFLOPs)
[Figure]
(MB)
[Figure]
FL
[Figure]
SSL
[Figure]
MocoSFL+TAResSFL
Mix
[Figure]
E XPERIMENTAL R ESULT
Experimental Setting.
We simulate the multi-client MocoSFL scheme on a Linux machine, where we use different CPU threads to simulate different clients and a single RTX-3090 GPU to simulate the cloud server.
We use ResNet-18 (He et al., 2016) for the majority of the experiments to better compare with existing SoTA (Zhuang et al., 2022).
We use CIFAR-10 as the main dataset and also present results on CIFAR-100 and ImageNet 12-class subset as in Li et al.
For the IID case, we assume the entire dataset is divided randomly and equally among all clients.
For non-IID experiments, we mainly consider the pathological (aka.
class-wise) non-IID distribution as in McMahan et al.
(2017); Zhuang et al.
(2022) where we assign 2 classes of CIFAR-10/ImageNet-12 data or 20 classes of CIFAR-100 data randomly to each client.
We perform MocoSFL training for a total of 200 epochs, using SGD as the optimizer with an initial learning rate of 0.06.
For accuracy performance evaluation, we adopt similar linear probe methods as in Grill et al.
; Zhuang et al.
(2022), where we train a new linear classifier on the outputs generated by the MocoSFL backbone model.
We include details of hyper-parameter choices and evaluations in Appendix A.1.
5. A CCURACY P ERFORMANCE
Improved non-IID performance.
Fig.
5 shows how the increased synchronization frequency can significantly improve the non-IID accuracy.
We present results for the cut-layer choices of 1 and 3 convolutional layers in the client-side model, represented by “MocoSFL-1” and “MocoSFL-3”, respectively.
We attribute the improved accuracy to the reduction in model divergence.
Comparison with FL-SSL.
When synchronization frequency of the MocoSFL is set to 10 (per epoch) for the 5-client cases, MocoSFL achieves significantly better non-IID accuracy performance than Zhuang et al.
(2022) on CIFAR-10 dataset due to lower model divergence (see Table 2).
On
CIFAR-100 dataset, with N C
= 5 , we observe our method has lower accuracy than FL-SSL methods.
We hypothesize that the performance of MocoSFL is more sensitive to the model complexity, and thus has limited performance for a more complex task like CIFAR-100.
When model complexity is high enough, for instance, on a larger ResNet-50 model, accuracy of our method is ∼ 4% higher than Zhuang et al.
(2022) as shown in Appendix B.7.
Furthermore, our methods outperform FL-SSL methods by a large margin in 20-client cases thanks to the feature sharing aspect.
[Figure]
NN
A cc u r a c y
[Figure]
A cc u r a c y ( % )
[Figure] [Figure]
[Table]
Cross-client Performance.
Our proposed MocoSFL can generalize from a cross-silo application (with upto 20 clients) to a cross-client application with 100, 200, and 1000 clients.
Note that none of the previous FL-SSL methods can scale to such a large number of clients.
For the hyper-parameter choices, we follow two principles introduced in Appendix A.1 – we let each client use a batch size of 1 and use the synchronization frequency of f S =
(1000 /N C ) /epoch,
and we set the client sampling ratio to 100 /N C to keep the same equivalent batch size at the server end.
The results are shown in Table 3.
Note that each client has only 50 data samples in the 1000-client case.
MocoSFL’s accuracy for IID case is high when N C increases from 100 to 1,000, though its accuracy drop by 1% for non-IID case.
This small drop is because model divergence scales with number of clients as described in Section 4.4.
[Table]
5. P RIVACY E VALUATION
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
We use G to perform MIA attack; the architecture information of G is given in Appendix A.3.
We assume the real-time attacker also uses an attack model with the same architecture as G .
For the pre-training step, we use a fixed hyper-parameter choice of λ = 2 . 0 and a target SSIM level of 0.6 to limit the strength of regularization.
We assume 0.0%, 0.5% and 1.0% of the target dataset CIFARis accessible
[Figure]
and we use CIFAR-100 dataset as source dataset to assist the pre-training.
We set the cut-layer choices to 1 and 3 and set the #clients to 100 for the training process.
Successful mitigation of MIA.
As shown in Table 4, applying TAResSFL can achieve good accuracy performance ( > 81%) as well as high enough MIA resistance ( > 0.020) for most cases.
Fig.
6 shows the visualization of MocoSFL-3 when 1.0% target data is available.
We observe the reconstructed images are much more noisy and blurry such that the subject can be successfully hidden.
Larger cut-layer allows a better resistance-accuracy tradeoff.
Using a smaller cut-layer ofseems a better choice in terms of accuracy and hardware requirement.
However, as shown in Table 4, the accuracy and resistance tradeoff seems much better by using a cut-layer of 3.
We believe that the extra client-side model complexity helps in the optimization of both accuracy and resistance objectives.
So, applying TAResSFL with a larger cut-layer is a more favorable option.
5. H ARDWARE D EMONSTRATION
Finally, we compare the total hardware resource cost of the proposed MocoSFL and “MocoSFL+TAResSFL” with synchronization frequency of 1/epoch for 200 epochs, and FL-SSL (Zhuang et al., 2022) with 500 local epochs per client and synchronization frequency set to 1 per 5 local epochs (original setting).
For MocoSFL, we use 1,000 clients with batch size of 1, and cut-layer of 3.
For
FL-SSL, to achieve similar accuracy, we use 5 clients with batch size of 128.
And we assume the data follows the default 2-class non-IID setting.
To evaluate overhead, we use commonly-used libraries such as ‘fvcore’ for FLOPs and ‘torch.cuda.memory allocated’ for memory (please see Appendix A. for detail.)
For power measurement, we use a Raspberry Pi 4B equipped with 1GB memory as one real client and simulate all other clients on the PC. v
MocoSFL-TAResSFL Scheme v
[Table]
+ [Table] [Table]
+ [Table] [Table] +
[Table] [Table] + [Table]
[Table] [Table] 𝑿 𝒕,𝒌+ 𝑿 𝒔,𝒌+ 𝑿 𝒕,𝒒 𝑿 𝒔,𝒒 v
Simulated Attacker 𝑿 𝒕,𝒒∗ 𝑨 𝒕,𝒒 𝐼𝑛𝑓𝑜𝑁𝐶𝐸 𝑙𝑜𝑠𝑠 v
Resistant Feature Extractor initialize ClientsOnline models Freeze ❷ ❶ 𝑿 𝒔,𝒒∗ 𝑨 𝒔,𝒒
Server Battery
TCP
Client
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
Mix
[Figure]
MocoSFL Scheme v
Feature Extractor 𝑨 𝒕,𝒌+ 𝑨 𝒔,𝒌+
Tail model + Contrastive head 𝑸 𝒕 ,𝑲 𝒕+ 𝑸 𝒔 ,𝑲 𝒔+ 𝑨 𝒕,𝒒 𝑨 𝒔,𝒒 𝑿 𝒕,𝒌+ 𝑿 𝒔,𝒌+ 𝑿 𝒕,𝒒 𝑿 𝒔,𝒒
v
Simulated Attacker 𝑿 𝒕,𝒒∗ 𝑨 𝒕,𝒒 𝐼𝑛𝑓𝑜𝑁𝐶𝐸 𝑙𝑜𝑠𝑠 v
Resistant Feature Extractor initialize ClientsOnline models freeze ❷ ❶ 𝑿 𝒔,𝒒∗ 𝑨 𝒔,𝒒   3.
Computation (GFLOPs)
Communication (MB)
Memory (MB)
MocoSFL-TAResSFL
FL-SSL
Server Battery
TCP
[Figure]
As shown in Fig.
7, FL-SSL requires 7,949 TFLOPs for the entire training session as each client needs to perform 500 local epochs on 10K data and a memory consumption of 921 MB.
The communication overhead due to synchronization has a high cost of around 8,269 MB.
Thus, FL-SSL is only suitable for cross-silo situation.
For “MocoSFL+TAResSFL”, hardware requirements are reasonable since computation is only 734 GFLOPs and communication is around 39 MB in total.
The memory consumption is a tiny 0.8 MB mostly due to the weight parameters since TAResSFL does not require local training.
As shown in Fig.
8, our measurement using a USB multimeter shows the proposed MocoSFL running on the Raspberry Pi only draws power of 2.26W, in average, and consumes around 9,100 mAh on a 5V battery.
C ONCLUSION
We propose MocoSFL, a collaborative SSL framework based on SFL.
The proposed framework addresses hardware resource requirement at client-side by enabling small batch size training and computation offloading.
It also relieves the large data requirement of local contrastive learning by enabling effective feature sharing.
The proposed scheme is the only one that can support a large number of clients.
In combination with a ResSFL-based module, it addresses privacy concerns of MocoSFL.
Finally, it achieves even better IID/non-IID performance with much lower hardware requirement than the SoTA FL-based SSL methods.
E THICS S TATEMENT
In this work, we address two practical issues of hardware (compute and memory) resources and large data requirement in collaborative SSL by using a SFL-based scheme.
Compared to the conventional approaches, the resulting MocoSFL scheme has significant advantages in affordability and is much easier to deploy.
Apart from being environmental-friendly, MocoSFL makes it possible to reach more clients, especially for those in poverty or under-represented minorities, and thus eliminate potential human-bias and unfairness issues of the ML service.
We address the privacy issues in the proposed MocoSFL by proposing TAResSFL module to use in sensitive applications.
Fig.
6 shows that the subject of the raw image can be successfully hidden.
More discussion on the privacy of “MocoSFL+TAResSFL”.
We notice a line of works known as instance encoding (Huang et al., 2020; Yala et al., 2021), which try to protect users’ data by transforming the original dataset to a distorted version such that they cannot be recognized by humans while an arbitrary deep learning model can still learn useful information from it (i.e. achieve high accuracy on a classification task).
MocoSFL, especially with the TAResSFL, where the client-side model is frozen, have some similarities since the frozen client-side model can be seen as a transformation, and the collection of latent vectors can be regarded as the transformed dataset.
However, “MocoSFL+TAResSFL” scheme has two favorable properties that make it distinct from instance encoding methods.
Domain dependency.
First, SFL only finds a transformation method for a given task.
Since the transformation method itself (the client-side model) heavily depends on the target domain information (as we need access to the target domain data to train the client-side model), it cannot work on data from another domain.
As the output of the “client-side model transformation”, latent vectors are only useful for current problem without any transferability guarantee.
However, instance encoding methods intend to derive a general transformation method that can work across domains, targeting a harder problem.
Risk Control.
Instance encoding methods publish the transformed dataset to the wide public which is risky.
As pointed out by Carlini et al.
(2021): all the raw data will be leaked if a successful decryption method is invented in the future even if it does not exist now.
While in our proposed scheme, the latent vectors from clients will only be accessible to the server party, thereby reducing the risk.
Since we already provide TAResSFL to mitigate the MIA attack that can be possibly launched from the honest-but-curious server, the risk is minimized.
To eliminate possibility of future advanced attack, we can introduce a protocol that requests the server party to regularly delete 1 these latent vectors immediately after the training is done.
R EPRODUCIBILITY S TATEMENT
To make it easier for readers to reproduce the results in this work, we open-source our code at https://github.com/SonyAI/MocoSFL .
We also provide detailed explanation on the MocoSFL training and evaluation hyperparameters, collaborative learning system hyperparameters in
Appendix A.1, and TAResSFL module hyperparameters in Appendix A.4.