SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
Abstract
Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients.
Most existing results on Federated Learning (FL) assume the clients have ground-truth labels.
However, in many practical scenarios, clients may be unable to label task-speciﬁc data due to a lack of expertise or resource.
We propose SemiFL to address the problem of combining communication-efﬁcient FL such as FedAvg with SemiSupervised Learning (SSL).
In SemiFL, clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has a small amount of labeled data.
We provide a theoretical understanding of the success of data augmentation-based SSL methods to illustrate the bottleneck of a vanilla combination of communication-efﬁcient FL with SSL.
To address this issue, we propose alternate training to ‘ﬁne-tune global model with labeled data’ and ‘generate pseudo-labels with the global model.’
We conduct extensive experiments and demonstrate that our approach signiﬁcantly improves the performance of a labeled server with unlabeled clients training with multiple local epochs.
Moreover, our method outperforms many existing SSFL baselines and performs competitively with the state-of-the-art FL and SSL results. Our code is available here.
Introduction
For billions of users around the world, mobile devices and Internet of Things (IoT) devices are becoming common computing platforms [1].
These devices produce a large amount of data that can be used to improve a variety of existing applications [2].
Consequently, it has become increasingly appealing to process data and train models locally from privacy and economic standpoints.
To address this, distributed machine learning framework of Federated Learning (FL) has been proposed [3,4].
This method aggregates locally trained model parameters in order to produce a global inference model without sharing private local data.
Most existing works of FL focus on supervised learning tasks assuming that clients have groundtruth labels.
However, in many practical scenarios, most clients may not be experts in the task of interest to label their data.
In particular, the private data of each client may be completely unlabeled.
For instance, a healthcare system may involve a central hub (“server”) with domain experts and a [Figure] limited number of labeled data (such as medical records), together with many rural branches with non-experts and a massive number of unlabeled data.
As another example, an autonomous driving startup (“server”) may only afford beta-users assistance in labeling a road condition but desires to improve its modeling quality with the information provided by many decentralized vehicles that are not beta-users.
The above scenarios naturally lead to the following important question: How a server that hosts a labeled dataset can leverage clients with unlabeled data for a supervised learning task in the Federated Learning setting?
We propose a new Federated Learning framework SemiFL to address the problem of Semi-Supervised
Federated Learning (SSFL) as illustrated in Figure 1.
We discover that it is challenging to directly combine the state-of-the-art SSL methods with the communication efﬁcient federated learning methods such as FedAvg to allow local clients to train multiple epochs [4].
The key ingredient that enables SemiFL to allow unlabeled clients to train multiple local epochs is that we alternate the training of a labeled server and unlabeled clients to ensure that the quality of pseudo-labeling is highly maintained during training.
In particular, we ﬁne-tune the global model with labeled data and generate pseudo-labels only with the global model.
We perform extensive empirical experiments to evaluate and compare our method with various baselines and state-of-the-art techniques.
The results demonstrate that our method can outperform existing SSFL methods and perform close to the state-of-the-art of FL and SSL results.
In particular, we contribute the following.
• We propose SemiFL in which clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has a small amount of labeled data.
We identify the difﬁculty of combining communication efﬁcient FL method FedAvg [4] with the state-of-the-art SSL methods.
• We develop a theoretical analysis on strong data augmentation for SSL methods, the ﬁrst in the literature to our best knowledge.
We provide a theoretical understanding of the success of data augmentation-based SSL methods to illustrate the bottleneck of a vanilla combination of communication-efﬁcient FL with SSL.
• To the best of our knowledge, we propose the ﬁrst communication efﬁcient SSFL method alternate training that can improve the performance of a labeled server by allowing unlabeled clients to train multiple local epochs, i
e., from 42% to 88% with 250 labeled data, and from 77% to 93% accuracy with 4000 labeled data on the CIFAR10 dataset.
Moreover, our proposed method achieves 30% improvement over the existing SSFL methods.
Furthermore, SemiFL performs competitively with the state-of-the-art FL methods and SSL methods.
i e., only 1% and 2% away from the state-of-the-art FL and SSL results, respectively, forlabeled data on the CIFAR10 dataset.
The outline of the paper is given below.
In Section 2, we review the related work.
In Section 3, we identify the problem of combining SSL with communication-efﬁcient FL methods, develop a theoretical analysis of how strong data augmentation can signiﬁcantly improve the classiﬁcation accuracy, and present the proposed SemiFL method with some intuitive explanations.
In Section 4, we evaluate the empirical performance of the SemiFL.
We make some concluding remarks in Section 5.
Related Work
Federated Learning The goal of Federated Learning is to scale and speed up the training of distributed models [5, 6].
FedAvg [4] allows local clients to train multiple epochs to facilitate convergence.
FedProx (Li et al., 2018) performs proximal regularization against global weights.
FL counterparts of Batch Normalization [7–9] are developed to further enhance the performance.
The use of local momentum and global momentum [10] have been shown to facilitate faster convergence.
FedOpt [11] proposes federated versions of adaptive optimizers to improve performance over FedAvg.
Semi-Supervised Learning Semi-Supervised Learning (SSL) refers to the general problem of learning with partially labeled data, especially when the amount of unlabeled data is much larger than that of the labeled data [12, 13].
The idea of self-training (namely to obtain artiﬁcial labels for unlabeled data from a pre-trained model) can be traced back to decades ago [14,15].
Pseudolabeling [16], a component of many recent SSL techniques [17], is a form of entropy minimization [18] by converting model predictions into hard labels.
Consistency regularization [19] refers to training models via minimizing the distance among stochastic outputs [13, 19].
A theoretical analysis of consistency regularization was recently developed in [20].
More recently, It has been demonstrated that the technique of strong data augmentation can lead to better outcomes [21–24].
Strongly augmented examples are frequently found outside of the training data distribution, which has been shown to beneﬁt SSL [25].
Semi-Supervised Federated Learning (SSFL)
Most existing FL works focus on supervised learning tasks, with clients having ground-truth labels.
However, in many real-world scenarios, most clients are unlikely to be experts in the task of interest, an issue raised in a recent survey paper [26].
In the research line of SSFL, the work [27] splits model parameters for labeled server and unlabeled clients separately.
Another related work [28] trains and aggregates the model parameters of the labeled server and unlabeled clients in parallel with group-wise reweights.
Applications of SSFL to speciﬁc applications can be found in, e.g., [29,30].
Method 3.1 Problem
In a supervised learning classiﬁcation task, we are given a dataset D =
{ x i , y i } Ni =1 , where x i is a feature vector, y i is an one-hot vector representing the class label in a K -class classiﬁcation problem, and N is the number of training examples.
In a Semi-Supervised Learning classiﬁcation task, we have two datasets, namely a supervised dataset S and an unsupervised dataset U .
Let S
=
{ x is , y is } N S i
= be a set of N S labeled data observations, and U = { x iu } N U i
=1 be a set of N U unlabeled observations (without the corresponding true label y iu ).
It is often interesting to study the case where N S (cid:28) N U .
In this work, we focus on Semi-Supervised Federated Learning (SSFL) with unlabeled clients, as illustrated in Figure 1.
Assume M clients and let x u,m denote the set of unsupervised data available at client m = 1 , 2 , · · · , M .
Similarly, let ( x s , y s ) denote the set of labeled data available at the server.
The server model is parameterized by model parameters W s .
The client models are parameterized respectively by model parameters { W u, 1 , . .
, W u,M } .
We assume that all models share the same model architecture, denoted by f : ( x, w ) (cid:55)→ f
( x, w ) , which maps an input x and parameters W to a vector on the K -dimensional simplex, e.g., using softmax function applied to model outputs.
Communication Efﬁcient FL with SSL In the standard communication efﬁcient FL scenario where clients can train multiple local epochs before model aggregation (i.e., FedAvg [4]), existing SSFL methods have difﬁculty in performing close to the state-of-the-art centralized SSL methods [27,28,31].
In fact, we will demonstrate in Table 1 that existing SSFL methods cannot outperform the case of training with only labeled data.
This is somewhat surprising given that their underlying methods of training unlabeled data are similar.
As shown in Figure 2, SSL methods such as FixMatch can only work with FedSGD, which requires batch-wise gradient aggregation and thus is not communication efﬁcient.
This is because SSL methods, such as FixMatch and MixMatch, sample from both labeled and unlabeled datasets for every batch of training data with a carefully tuned ratio [23,32].
Thus, it is not straightforward how we can combine the SSL method in a communication-efﬁcient FL scenario where we train multiple local epochs.
To understand the bottleneck of this vanilla combination, we need to understand better [Figure] why the state-of-the-art centralized SSL methods work.
In section 3.2, we analyze the strong data augmentation for SSL and demonstrate that the success of FixMatch is due to using data augmentation on pseudo-labeled data of high quality.
3.2 Theoretical Analysis of Strong Data augmentation for SSL
Pseudo-labeling is widely used for labeling unlabeled data in SSL methods [16,22,32].
However, the quality (accuracy) of those pseudo-labels can be low, especially at the beginning of the training.
In this light, several papers [22,32] propose to hard-threshold or sharpen the pseudo-labels to improve the quantity of accurately labeled pseudo-labels.
The problem with hard thresholding is that the data samples satisfying the conﬁdence threshold have a small training loss.
Therefore, the model cannot be signiﬁcantly improved as it already performs well on the data above the threshold.
To address this issue, we will use strong data augmentation [25,32] to generate data samples that have larger training loss.
The main idea is to construct a pseudo-labeling mechanism whereby our SSL method can generate more and more high-quality pseudo-labels during training.
Meanwhile, the augmented data for model training can produce a more considerable drop in training loss than the original data.
To provide further insights into SSL, we develop a theoretical analysis of the strong data augmentation, which is a critical component of the state-of-the-art SSL method FixMatch [32] and SemiFL, and can be interesting in its own right.
Intuitively, strong augmentation is a process that maps a data point (e.g., an image) from high quality to relatively low grade unilaterally.
The low-quality data and their high-conﬁdence pseudo-labels are then used for training so that there are sufﬁcient “observations” in the data regime insufﬁciently covered by labeled data.
Our theory is based on an intuitive “adequate transmission” assumption, which means that the distribution of augmented data from high-conﬁdence unlabeled data can adequately cover the data regime of interest during prediction.
Consequently, reliable information exhibited from unlabeled data can be “transmitted” to data regimes that may have been insufﬁciently trained with labeled data, as illustrated in Figure 3.
Instead of studying SSL in full generality, we restrict our attention to a class of nonparametric kernel-based classiﬁcation learning [33–35] and derive analytically tractable statistical risk-rate analysis.
More detailed background and technical details are included in the Appendix.
We provide a simpliﬁed statement as follows.
Theorem 1 Under suitable assumptions, an SSL classiﬁer ˆ C ssl trained from n u unlabeled data and the strong data augmentation technique has a statistical risk bound at the order of R ( ˆ C ssl ) ∼ n − q ( α +1) /
{ q ( α +3+ ρ )+ d } u where d , q , α , ρ are constants that describe the data dimension, smoothness of the conditional distribution function ( Y | X ), class separability (or task difﬁculty), and inadequacy of transmission, respectively.
The smaller ρ , the better risk bound.
Moreover, suppose that ˆ C l is the classiﬁer trained from n l labeled data, where n l ∼ n ζ u , ζ ∈ (0 , 1) .
It can be veriﬁed that the bound of
R ( ˆ C u ) is much smaller than that of R ( ˆ C l ) when ζ < q ( α +3)+ d q ( α +3+ ρ )+ d .
This provides an insight into the critical region of n u where signiﬁcant improvement can be made from unlabeled data. [Figure]
3.3 Alternate Training
As depicted in Figure 4(a), existing SSFL works follow the state-of-the-art SSL methods to synchronize the training of supervised and unsupervised data [23, 32]. For example, FedMatch [27] and FedRGD [28] adopt a vanilla combination of FedAvg and FixMatch.
They aggregate the server model trained from labeled data and clients’ models trained from unlabeled data at each communication round in parallel and generate pseudo-labels for each batch of unlabeled data with the local training model.
However, existing results [27,28] indicate that this vanilla combination has difﬁculty performing close to the state-of-the-art SSL methods, even if the unlabeled clients are trained with the same SSL methods.
[Figure] In the communication efﬁcient FL setting, we cannot guarantee an increase in the quality of pseudolabels during training because we allow local clients to train multiple epochs, potentially deteriorating the performance (see Figure 2).
Furthermore, the aggregation of a server model trained with groundtruth labels and a subset of client models trained with pseudo-labels does not constantly improve the performance of the global model over the previous communication round.
A poorly aggregated global model of the last round produces worse-quality pseudo-labels.
Subsequently, the performance of the aggregated model degrades in the next round.
To maintain and improve the quality of our generated pseudo-labels during training, we propose to train the labeled server and unlabeled clients in an alternate manner, as illustrated in Figure 4(b).
In particular, our approach consists of two important components:
• Fine-tune global model with labeled data At each round, the server will retrain the global model with the labeled data.
In this way, the server can provide a comparable or better model than the previous round for the active clients in the next round to generate pseudo-labels.
On the contrary, the vanilla method aggregates server and client models in parallel.
As a result, the quality of generated pseudo-labels will gradually degrade and thus deteriorate the performance.
• Generate pseudo-labels with global model We will label the unlabeled data once the active clients immediately receive the global model from the server.
This way, pseudo-labels’ quality will not degrade during the local training.
On the contrary, the vanilla method labels every batch of data during the training of unlabeled clients.
As a result, the quality of generated pseudo-labels will gradually degrade during local training, thus deteriorating performance.
Our proposed approach ensures that the clients can continually generate better quality pseudo-labels during training.
Our experimental studies show that the proposed method can signiﬁcantly improve the performance of the labeled server and performs competitively even with the state-of-the-art FL and centralized SSL methods.
The limitation of our approach is that we need to update the aggregated client model with labeled data from the server, which will delay the computation time.
We will conduct an ablation study on each component of alternative training in Table 2.
3.4 The SemiFL Algorithm
We summarize the pseudo-code of the proposed solution in Algorithm 1.
At each iteration t , the server will ﬁrst update the model with the standard supervised loss L s for local epochs E with data batch ( x b , y b ) of size B s randomly split from the supervised dataset D s , using
[Equation]
where α ( · ) represents a weak data augmentation, such as random horizontal ﬂipping and random cropping, that maps one image to another.
Subsequently, the server updates the static Batch Normalization (sBN) statistics [9] (which is discussed in Appendix B).
Next, the server distributes server model parameters W s to a subset of clients.
We denote the proportion of active clients at each communication round t as activity rate C t ∈ (0 , 1] .
Without loss of generality, we assume that
C t =
C is a constant over time.
After each active local client, say client m , receives the transmitted
W s , it generates pseudo-labels y u,m as follows:
[Equation]
Each local client will construct a high-conﬁdence dataset D ﬁx u,m inspired by FixMatch [32] at each iteration t , deﬁned as:
[Equation]
for a global conﬁdence threshold 0 < τ < 1 pre-selected by all clients.
If for some client m , we have
D ﬁx u,m = ∅
then it will stop and refrain from transmission to the server.
Otherwise, we will sample with replacement to construct a dataset inspired by MixMatch [23].
In other words,
[Equation]
where |D ﬁx u,m | denotes the number of elements of D ﬁx u,m .
Thus |D mix u,m | =
|D ﬁx u,m | .
Subsequently, client m trains its local model for E epoch to speed up convergence [4].
For each local training epoch of the client m , it randomly splits local data D ﬁx u,m , D mix u,m into batches B ﬁx u,m , B mix u,m of size B m .
For each batch iteration, as in [36], client m constructs Mixup data from one particular data batch ( x ﬁx b , y ﬁx b ) , ( x mix b , y mix b ) by
[Equation]
where a is the Mixup hyperparameter.
Next, client m deﬁnes the “ﬁx” loss L ﬁx [32] and “mix” loss
L mix [24] by
[Equation]
Input: Unlabeled data x u, 1: M distributed on M local clients, activity rate C , the number of communication rounds T , the number of local training epochs E , server and client respective batch sizes B s and B m , local learning rate η , server model parameterized by W s client models parameterized by { W u, 1 , . . .
, W u,M } , weak data augmentation function α ( · ) , strong data augmentation function A ( · ) , conﬁdence threshold τ , Mixup hyper-parameter a , loss hyperparameter λ , common model architecture function f ( · )
System executes: for each communication round t = 1 , 2 , . . .
T do
W ts ← ServerUpdate ( x s , y s , W ts )
Update the sBN statistics
S t ← max( (cid:98) C · M (cid:99) , 1) active clients uniformly sampled without replacement for each client m ∈ S t in parallel do
Distribute server model parameters to local client m , namely W tu,m ← W ts
W tu,m ← ClientUpdate ( x u,m , W tu,m ) end
Receive model parameters from M t clients, and calculate W ts =
M − 1 t (cid:80) M t m =1 W tu,m end
W Ts ← ServerUpdate
( x s , y s , W Ts )
Update the sBN statistics
ServerUpdate ( x s , y s , W s ) :
Construct supervised dataset D s =
( x s , y s ) for each local epoch e from 1 to E do B s ← Randomly split local data D s into batches of size B s for batch ( x b , y b )
∈ B s do
L s ← (cid:96) ( f ( α ( x b ) , W s ) , y b )
W s ← W s − η ∇ W L s end end
Return W s ClientUpdate ( x u,m , W u,m ) :
Generate pseudo-labels with weakly augmented data α ( x u,m ) , namely y u,m =
f ( α ( x u,m ) , W u,m )
Construct FixMatch dataset, namely D ﬁx u,m = { ( x u,m , y u,m ) with max ( y u,m ) ≥ τ }
If D ﬁx u,m = ∅ then Stop.
Return.
Construct an equal-size Mixup dataset, namely
D mix u,m = Sample |D ﬁx u,m | with replacement { ( x u,m , y u,m ) } for each local epoch e from 1 to E do
B ﬁx u,m , B mix u,m ← Randomly split local data D ﬁx u,m , D mix u,m into batches of size B ﬁx m , B mix m for batch ( x ﬁx b , y ﬁx b ) , ( x mix b , y mix b )
∈ B ﬁx u,m , B mix u,m do λ mix ∼ Beta( a, a ) x mix ← λ mix x ﬁx b + (1 − λ mix ) x mix b
L ﬁx ← (cid:96) ( f ( A ( x ﬁx b ) , W u,m ) , y ﬁx b )
L mix ← λ mix · (cid:96) ( f ( α ( x mix ) , W u,m ) , y ﬁx b ) + (1 − λ mix ) · (cid:96) ( f ( α ( x mix ) , W u,m ) , y mix b ) (cid:1)
W u,m ← W u,m − η ∇ W
( L ﬁx + λ · L mix ) end end
Return W u,m and send it to the server
Here, A represents a strong data augmentation mapping, e.g., the RandAugment [37] used in our experiments, and (cid:96) is often the cross entropy loss for classiﬁcation tasks.
Finally, client m performs a gradient descent step with
[Equation]
, (6) where λ > 0 is a hyperparameter set to be one in our experiments.
After training for E local epochs, client m transmits W u,m to the server.
[Figure] Without loss of generality, assume that clients 1 , 2 , · · · , M t have sent their models to the server at time t .
The server then aggregates client model parameters { W u, 1 , . .
, W u,M t } by W s =
M − 1 t (cid:80) M t m =1 W u,m [4].
This process is then repeated for multiple communication rounds T .
After the training is ﬁnished, the server will further ﬁne-tune the aggregated global model by additional training with the server’s supervised data using its supervised loss L s .
Finally, it will update the sBN statistics one ﬁnal time.
Experiments
Experimental setup To evaluate our proposed method, we conduct experiments with CIFAR10,
SVHN, and CIFAR100 datasets [38,39].
To compare our method with existing FL and SSFL methods, we follow the standard communication efﬁcient FL setting, which was originally used in FedAvg [4] and widely adopted by following works, such as [9, 40, 41].
We have 100 clients throughout our experiments, and the activity rate per communication round is C = 0 .
We uniformly assign the same number of data examples for IID data partition to each client.
For a balanced Non-IID data partition, we ensure each client has data at most from K classes and the sample size of each class is the same.
We set K = 2 because it is the most label-skewed case for classiﬁcation, and it has been evaluated in [9,40,41].
For unbalanced Non-IID data partition, we sample data for each client from a Dirichlet distribution Dir( α ) [41,42].
As α → ∞ , it reduces to IID data partition.
We perform experiments with α = { 0 . 1 , 0 . 3 } .
To compare our method with the state-of-the-art SSL methods, we follow the experimental setup in [32].
We use Wide ResNet28x2 [43] as our backbone model for CIFAR10 and SVHN datasets and
Wide ResNet28x8 for CIFAR100 datasets throughout our experiments.
The number of labeled data at the server for CIFAR10, SVHN, and CIFAR100 datasets N S are { 250 , 4000 } , { 100 , 2500 } , and { 2500 , 10000 } respectively.
We conduct four random experiments for all the datasets with different seeds, and the standard errors are shown inside the parentheses for tables and by error bars in the ﬁgures.
We demonstrate our experimental results in Table 1 and the learning curves of CIFAR10,
SVNH, and CIFAR100 datasets in Figure 5, 7, and 8.
Further details are included in the Appendix.
Comparison with SSL methods We demonstrate the results of Fully Supervised and Partially
Supervised cases and existing SSL methods for comparison in Table 1.
The Fully Supervised case refers to all data being labeled, while in the Partially Supervised case, we only train the model with the partially labeled N S data.
Our results signiﬁcantly outperform the Partially Supervised case.
In other words, SemiFL can substantially improve the performance of a labeled server with unlabeled clients in a communication-efﬁcient scenario.
Our method performs competitively with the state-of-the-art SSL methods for IID data partition.
Moreover, it is foreseeable that as the clients become more label-skewed for Non-IID data partition, the performance of our method degrades.
However, even the most label-skewed unlabeled clients can improve the performance of the labeled server using our approach.
A limitation of our work is that as the supervised data size decreases, the performance of
SemiFL degrades more than the centralized SSL methods.
We believe it is because we cannot train labeled and unlabeled data simultaneously in one data batch.
Comparison with FL and SSFL methods We compare our results with the state-of-the-art FL and SSFL methods in Table 1.
We demonstrate that SemiFL can perform competitively with the state-of-the-art FL result trained with fully supervised data.
It is worth mentioning that SSFL may [Table] outperform FL methods in the Non-IID data partition case because the server has a small set of labeled IID data.
We also demonstrate that our method signiﬁcantly outperforms existing SSFL methods.
Existing SSFL methods fail to perform closely to the state-of-the-art centralized SSL methods, even if their underlying SSL methods are the same.
Moreover, existing SSFL methods cannot outperform the Partially Supervised case, indicating that they deteriorate the performance of the labeled server.
In particular, FedMatch allocates disjoint model parameters for the server and clients, and FedRGD assigns a higher weight for the server model for aggregation.
Both methods do not directly ﬁne-tune the global model with labeled data and generate pseudo-labels with the received global model.
To our best knowledge, the proposed SemiFL is the ﬁrst SSFL method that actually improves the performance of the labeled server and performs close to the state-of-the-art FL and SSL methods.
Ablation studies We conduct ablation studies on SemiFL and demonstrate the results in Table 2.
Based on our extensive experiments, it is evident that “Fine-tune global model with labeled data” and “Generate pseudo-labels with global model” are the critical components of the proposed ‘Alternate
Training’ method for the success of our method.
We also conduct an ablation study on static Batch
Normalization(sBN), the number of local training epochs, the Mixup data augmentation, and global
SGD momentum.
The detailed results can be found in the Appendix.
4.1 Quality of Pseudo Labeling
We measure the quality of Pseudo-Labeling for Semi-Supervised Learning from three aspects, including the accuracy of pseudo-labels (Pseudo Accuracy), the accuracy of thresholded pseudolabels (Threshold Accuracy), and the ratio of pseudo-labeled data (Label Ratio) with CIFAR dataset in Figure 6. We perform ablation studies on our proposed method by measuring the quality of Pseudo-Labeling. The results demonstrate that our proposed alternative training, the combination of ‘Fine-tune global model with labeled data’ and ‘Generate pseudo-labels with global model,’ can produce pseudo labels of much better quality when clients have completely unlabeled data and train multiple local epochs.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Conclusion
In this work [Figure] we propose a new communication-efﬁcient Federated Learning (FL) framework named
SemiFL to address the problem of Semi-Supervised Federated Learning (SSFL) for unlabeled clients.
We identify the difﬁculty of combining communication-efﬁcient Federated Learning (FL) with state-of-the-art Semi-Supervised Learning (SSL).
We develop a theoretical analysis of strong data augmentation for SSL, which illustrates the bottleneck of vanilla combination.
We propose to train the labeled server and unlabeled clients in an alternate manner by ‘ﬁne-tune global model with labeled data’ and ‘generate pseudo-labels with global model.’
We utilize several training techniques and establish a strong benchmark for SSFL.
Extensive experimental studies demonstrate that our communication-efﬁcient method can signiﬁcantly improve the performance of a labeled server with unlabeled clients.
Moreover, we show that SemiFL can perform competitively with the state-of-the-art centralized SSL and fully supervised FL methods.
Our study provides a practical SSFL framework that extends the scope of FL applications. Acknowledgments
The work of Enmao Diao and Vahid Tarokh was supported by the Ofﬁce of Naval Research (ONR) under grant number N00014-18-1-2244.
The work of Jie Ding was supported by the National Science
Foundation (NSF) under grant number DMS-2134148.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See last sentences of Section 3. and 4.
, (c) Did you discuss any potential negative societal impacts of your work?
[No] We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section D. in Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section D.5 in Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide source codes in the supplementary material.
We use public available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section C.1 in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The error bars are shown in ﬁgures and the numerical standard error are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] One Nvidia 1080TI is enough for one experiment run.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the public available datasets we use.
(b) Did you mention the license of the assets?
[No] They should be available publicly.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use public available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We use public available datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Performance Goal
We outline the general performance goal of Semi-Supervised Federated Learning.
The performance ceiling is obviously that of Fully Supervised Learning (FSL) (namely, assuming that all the server’s and clients’ data are centralized and fully labeled).
For our context where clients’ data are unlabeled, a vanilla approach trains the labeled data only on the server-side, referred to as Partially Supervised
Learning (PSL).
Clearly, the PSL performance can serve as a lower bound benchmark for other approaches that employ additional unlabeled data.
When the server contains a small amount of labeled data and a substantial amount of unlabeled data (centralized), Semi-Supervised Learning (SSL) seeks to use unlabeled data to improve over the PSL.
It was shown that state-of-the-art SSL methods such as FixMatch [32] could produce similar results as FSL.
Our work focuses on Semi-Supervised Federated Learning (SSFL), where the unlabeled data are distributed among many clients.
The general goal of SSFL is to perform similarly to the state-ofthe-art SSL and signiﬁcantly outperform PSL and the existing SSFL methods.
In other words, our performance goal is to achieve FSL (cid:38) SSL (cid:38) SSFL (cid:29) PSL .
B Static Batch Normalization
We utilize a recently proposed adaptation of Batch Normalization (BN) named Static Batch Normalization (sBN) [9].
It was shown that this method greatly accelerates the convergence and improves the performance of FedAvg [4] compared with other forms of normalization, including InstanceNorm [45], GroupNorm (GN) [46], and LayerNorm [47].
During the training phase, sBN does not track the running statistics with momentum as in BN.
Instead, it simply standardizes the data batch x b and utilizes batch-wise statistics µ b and σ b in the following way.
[Equation]
In FL training, the afﬁne parameters γ and β can be aggregated as usual.
We note that FedAvg with vanilla BN is not functional because the BN statistics µ and σ used for inference are averaged from the tracked running BN statistics of local clients during training.
Let x m represents the local data of client m (with size N m ).
For a total of M local clients, sBN computes the global BN statistics µ and σ for inference by querying each local client one more time after training is ﬁnished, based on
[Equation]
In the context of SemiFL, we need to generate pseudo-labels at every communication round.
Thus, local clients need to upload BN statistics for every communication round.
Fortunately, we can utilize the server data x s to update the global statistics instead of querying each local client, where µ = E[ x s ] and σ 2 =
Var[ x s ] .
We provide experimental results of querying the sBN statistics from all the clients and include an ablation study using only the server data in Table 3.
In Table 3, we demonstrate the ablation study of the sBN statistics on the CIFAR10 dataset.
Compared with updating the sBN statistics with only the server data, updating the sBN statistics with both server and clients does not provide signiﬁcant improvements.
[Table]
C Experimental Results C.1 Experimental setup
In Table
[Table]
we provide the hyperparameters used in the experiments.
Similar to [32], we use SGD as our optimizer and a cosine learning rate decay as our scheduler [48].
We also use the same hyperparameters as [32], where the local learning rate η = 0 .
03 , the local momentum β l = 0 .
9 , and the conﬁdence threshold τ = 0 .
The Mixup hyperparameter a is set to be 0 .
75 as suggested by [36].
We use the standard supervised loss to train the labeled server.
For training the unlabeled clients, the “ﬁx” loss L ﬁx (proposed in FixMatch [32]) leverages the techniques of consistency regularization and pseudo-labeling simultaneously.
Speciﬁcally, the pseudo-labels are generated from weakly augmented data, and the model is trained with strongly augmented data.
The “mix” loss (adapted from MixMatch [23,36]) reduces the memorization of corrupted labels and increases the robustness to adversarial examples.
It was also shown to beneﬁt the SSL [24] and FL [49] methods.
We have conducted an ablation study and demonstrated that the mix loss moderately improves performance.
[Table]
C.2 SVHN and CIFAR
In Figure 7 and 9, we demonstrate the results of SVHN and CIFAR100 datasets.
[Figure] [Figure]
C.3 Ablation studies
We perform an ablation study of the training techniques adopted in our experiments.
We study the efﬁcacy of the number of local training epoch E , the Mixup data augmentation, and the global SGD momentum β g [10] as shown in Table 5.
Less local training epoch signiﬁcantly hurts the performance due to slow convergence.
The Mixup data augmentation has around 2% Accuracy improvement for the CIFAR10 dataset.
It demonstrates that it is beneﬁcial to combine strong data augmentation with
Mixup data augmentation for training unlabeled data.
The global momentum marginally improves the result.
[Table]
[Figure]
D Theoretical Analysis of Strong Data Augmentation for SSL D.1 Background of Classiﬁcation
We take the binary classiﬁcation task as an illustrating example.
Let ( Y, X ) be a random variable with values in R d × { 1 , 0 } .
For the prediction task, we look for a classiﬁer C : R d → { 1 , 0 } such that the risk P ( C ( X ) (cid:54) =
Y ) is small, where P denotes the probability measure for ( Y, X ) .
Let m ( x ) ∆ =
E ( Y = 1 | X = x ) denote the conditional probability of Y given X = x .
For example, the standard logistic regression model is in the form of m ( x ) = 1 / (1 + exp( − β T x )) for some β ∈ R d .
When the underlying m is known, the risk-optimal classiﬁer is known to be
[Equation]
) for any given x .
When the underlying m is unknown, we need to train a classiﬁer ˆ C n from observed training data ( Y i , X i ) , i = 1 , . . . , n , which are often assumed to be IID random variables following the same distribution of ( Y, X ) .
A general approach is to ﬁrst learn ˆ m n : R d → R and then let ˆ C n ( x ) ∆ = 1 { ˆ m n ( x ) − 1 / 2 } .
To evaluate the prediction performance of a learned ˆ C n , we consider its gap with the optimal classiﬁer
[Equation]
8) referred to as the classiﬁcation risk of ˆ C n .
D.2 Background of Semi-Supervised Learning
Suppose that we observe n l IID labeled data of ( Y l , X l ) , denoted by D l = { ( Y l i , X l i ) } n l i =1 , where X l has probability distribution P l and E ( Y l | X l = x ) = m ( x ) .
We also observe n u unlabeled data of ( X u ) , denoted by { X u j } n u j =1 , where each X u has probability distribution P u .
Here, P u may or may not be the same as P l .
The Semi-Supervised Learning problem of interest concerns the case n u (cid:29) n l and solutions that can properly utilize the unlabeled data to boost the performance of a classiﬁer trained from labeled data.
In other words, we look for a classiﬁer
ˆ C ssl n ( x ) trained from observations of both ( Y l , X l ) and X u , so that its risk satisﬁes
[Equation]
where ˆ C l is the classiﬁer trained from observations of ( Y l , X l ) only.
D.3 A new perspective of Semi-Supervised Learning
As we mentioned in Section 2, there has been a lot of empirical success in using new techniques such as consistency regularization and strong augmentation to improve the classiﬁcation risk of classical
Semi-Supervised Learning.
Recently, the work of [50] provides a theoretical understanding of the consistency regularization in reducing classiﬁcation risk.
Its analysis is based on an “expansion” assumption that a low-probability subset of data must expand to a large-probability neighborhood, and there is little overlap between neighborhoods of different classes.
To the best of our knowledge, the existing theories do not explain why the strong augmentation technique works so well (to achieve state-of-the-art performance) for Semi-Supervised Learning.
Intuitively, strong augmentation is a process that maps a data point (e.g., an image) from high quality to relatively low quality in a unilateral manner (illustrated in Figure 10).
Strong augmentation such as RandAugment [37] consists of a set of data augmentation strategies, e.g., rotating the image, shearing the image, translating the image, adjusting the color balance, and modifying the brightness.
The low-quality data and their high-conﬁdence pseudo-labels are then used for training so that there are sufﬁcient “observations” near the difﬁcult data regimes (e.g., near the decision boundary).
In line with the above intuition, we develop a theoretical understanding of how and when using strong augmentation can signiﬁcantly reduce the classiﬁcation risk obtained from only labeled data.
Instead of studying Semi-Supervised Learning in full generality, we restrict our attention to a class of nonparametric kernel-based classiﬁcation learning and derive analytically tractable statistical risk-rate analysis.
Our theory is based on an intuitive “ adequate transmission ” assumption, which means that the distribution of augmented data from high-conﬁdence unlabeled data can adequately cover the data regime of interest during the test.
Consequently, reliable information exhibited from unlabeled data can be “transmitted” to data regimes that may have been insufﬁciently trained with labeled data.
In addition to the notations made in Subsections D.1 and D.2, we will let ˜ X denote stronglyaugmented data from X u , and ˜ Y its corresponding label that follows the same conditional distribution, namely P ( ˜ Y = 1 | ˜ X ) = m ( ˜ X ) .
Recall that P u and P l are the probability measures of unlabeled X u and labeled X l , respectively.
We suppose that the test data distribution for evaluating the classiﬁcation performance also follows P l .
In other words, the probability measure in (8) is the product of P Y | X or P ˜ Y | ˜ X (as determined by m ( · ) ) and P l .
Let ˆ m 0 denote an initial estimate of m .
For generality, we will assume ˆ m 0 is learned from all or only part of the available labeled data.
To develop theoretical analyses, we consider the following generic SSL classiﬁer with strong augmentation.
Generic Semi-Supervised Learning with Strong Data Augmentation
• Step 1 .
From { X u i } n u i =1 , we pick up those “high-conﬁdence” x satisfying
[Equation]
for some δ (to be quantiﬁed), and denote the set as X aug .
• Step 2 .
For each X ∈ X aug , we calculate the pseudo-label ˆ
Y
= 1 { ˆ m 0 ( X ) − 1 / 2 } ; meanwhile, we generate the strongly augmented data ˜ X .
Consequently, we obtain a set of data ( ˆ Y , ˜ X ) and denote that set as D aug .
• Step 3 .
Train an estimate of m , denoted by m ssl , and the associated classiﬁer C ssl using the labeled and augmented data D ssl ∆ =
D l ∪ D aug .
Note that if ˆ m 0 is learned from data independent with D l , the data in D ssl are independent but not necessarily identically distributed (since P l and P u may not be the same).
To show how SSL with strong augmentation can potentially enhance classiﬁcation learning, we consider a classical nonparametric classiﬁer ˆ C deﬁned in the following way.
Let K : R d → R + denote the box kernel function that maps u to 1 {(cid:107) u (cid:107) ≤ 1 } , where 1 {·} denotes the indicator function.
With n labeled data ( Y i , X i ) , similarly to (7), we deﬁne
[Equation]
} , where ˆ m
[Equation]
if (cid:80) ni =1 K ( h − 1 n ( x − X i )) (cid:54)
= 0 , and ˆ m n ( x ) = [-
-]0 otherwise.
Here, ˆ m n is known as the NadarayaWatson kernel estimate [51,52] of the underlying m , and h n > 0 is the bandwidth.
In our setting, we suppose that n 0 > 0 labeled data are used to learn ˆ m 0 , and another n l ≥ 0 labeled data along with n u > 0 unlabeled data to learn ˆ m ssl and thus the subsequent classiﬁer ˆ C ssl .
Note that the n l is introduced only for generality.
Our technical analysis includes n l = 0 as a special case.
In the main result to be introduced, the risk bound will only involve n u but eliminate n l during technical derivations since we are interested in the regime of n u (cid:29) n 0 + n l .
Before starting the main result, we make the following additional technical assumptions and provide the intuitions.
(A1)
There exists positive constants c 1 and s such that P u (min { 1 − m ( X ) , m ( X ) } ≤ δ ) ≥ g s ( δ ) for all sufﬁciently small δ > 0 , where g s ( δ ) ∆ = c 1 δ s .
Explanation of (A1) :
Recall that P u is the probability measure of unlabeled data.
This condition requires a nontrivial amount of unlabeled data with high conﬁdence (or large margin) in the sense that m ( X ) is close to either zero or one.
The function g s quantiﬁes the “sufﬁciency” of data at the tail part of X .
Take logistic regression m ( x ) = 1 / (1 + exp( − β T x )) as an example.
It can be easily veriﬁed that
[Equation]
so P u (min { 1 − m ( X ) , m ( X ) } ≤ δ ) =
P u (1 − m ( X ) ≤ [Equation] + P u ( m ( X ) ≤ δ ) ≥ P u ( | β T X | ≥ − log δ ) for all δ ∈ (0 , 1 / 2) .
For example, if | β T X | follows standard Exponential, we let g s : δ (cid:55)→ δ (A2)
There exists a constant c 3 ∈ (0 , 1 / 2) such that the strong augmentation X u → ˜ X satisﬁes P ( ˜ Y = 1 | ˜ X =
˜ x, X u = x ) = m ( x ) for all x such that min { 1 − ˆ m 0 ( x ) , ˆ m 0 ( x ) } ≤ c 3 .
Explanation of (A2) : Let us think X u as a high-conﬁdence image, with m ( X u ) close to either zero or one.
Meanwhile, ˜ X is a strongly augmented version of X u , e.g., by random masking or noise injection, so m ( ˜ X ) is closer to 1 / 2 than m ( X u ) .
The condition of (A2) means that if conditioning on both images, the label ˜ Y has a distribution that is only determined by the higher-quality image, which is quite intuitive.
A mathematically equivalent way to describe (A2) is that ˜ X
→ X u → ˜ Y follows a Markov chain.
(A3)
There exist positive constants c 2 , c 4 , and a non-negative v such that for every P l -measurable ball B ⊆ R d with P l ( B ) ≤ c 4 , for the strong augmentation X u → ˜ X , we have P u ( ˜ X ∈ B | min { 1 − ˆ m 0 ( X u ) , ˆ m 0 ( X u ) } ≤ δ ) / P l ( B ) ≥ g v ( δ ) for all sufﬁciently small δ > 0 , where g v ( δ ) ∆ =
c 2 δ v .
Explanation of (A3) :
The above numerator is the probability of the augmented data ˜ X falling into B conditional on the original unlabeled data (with probability P u ) having high conﬁdence.
This assumption ensures that for every regime of signiﬁcant interest in evaluating the prediction performance (since P l is the measure for test data), there will be a sufﬁcient probability coverage of the augmented data.
This is an intuitive condition since otherwise, the augmented data cannot represent the test data of interest to boost the test performance.
In this assumption, the function g v determines the coverage as a function of tail probability δ .
For example, if v = 0 , a sufﬁciently small δ (or higher conﬁdence) gives a non-vanishing coverage.
The combination of (A2) and (A3) can be interpreted as an “adequate transmission” condition, under which a small amount of high-conﬁdence unlabeled data can induce augmented data that can accurately represent the test data regime of interest.
Such transmitted data can be basically approximated as labeled data for supervised training.
(A4)
There exist positive constants c
and α such that P l ( | m ( X l ) − 1 / 2 | ≤ t ) ≤ c 6 t α for all t > 0 .
Moreover, X l ∈ [0 , 1] d .
Explanation of (A4) :
The inequality is a margin condition that has been used in the classical learning literature (see, e.g., [34,35] and the references therein).
It determines the difﬁculty of the underlying classiﬁcation task.
Intuitively speaking, a larger α means more separability of the two classes under the probability P l .
The boundedness of X l is for technical convenience. (A5)
There exist positive constants q and c {+
+}
such that | m ( x ) − m ( x (cid:48) ) | ≤ c 7 (cid:107) x − x (cid:48) (cid:107) q for all x, x (cid:48) ∈ [0 , 1] d , where (cid:107) · (cid:107) denotes the Euclidean norm.
Explanation of (A5) :
This condition assumes a Lipschitz-type condition of m ( · ) , where q is allowed to be different from one.
Intuitively, it assumes the underlying classiﬁer to learn cannot be too bumpy.
For q ∈ (0 , 1] , a larger q means more smoothness of m ( · ) . (A6)
There exist positive constants r , c 8 , and ∆ such that | ˆ m 0 ( x ) − m ( x ) | ≤ c 8 n − rfor all x satisfying min { 1 − ˆ m 0 ( x ) , ˆ m 0 ( x ) } ≤ ∆ .
Explanation of (A6) :
This assumption requires that conditional on X falls into a large-margin area, the estimation error of the initial function ˆ m 0 is not too large. (A7)
For the constants s , v , α , q , and r deﬁned in the above assumptions, we have
[Equation]
Explanation of (A7) :
The two inequalities will be technical conditions used in the proof.
A sufﬁcient condition for (11) to hold is that α ≥ s .
Intuitively, this requires that α , which describes the separability of the decision boundary (the larger, the better), is not smaller than s , which quantiﬁes the sufﬁciency of tail samples (the smaller, the better).
The inequality (12) means that the initial classiﬁer ˆ m 0 cannot perform too poorly.
This matches our empirical observations that the SSL training in each round has to immediately follow a preceding round that uses some labeled data.
Also, the denominator in (12) favors relatively small s, d compared with α, v, q .
D.4 Main result
Our main result is provided below.
Theorem 1 : Under Assumptions (A1)-(A7), the generic SSL classiﬁer with strong augmentation (namely the above Steps 1-3) satisﬁes [Equation]
[Equation]
for some constant C that does not depend on the sample size.
Explanation of Theorem 1 : The theorem gives an explicit rate of convergence for the SSL classiﬁcation risk using unlabeled data of size n u .
It is the informal statement made in the main paper with ρ ∆ = v + s .
We interpret the power
[Equation]
as follows.
If the margin parameter α is large, the classiﬁcation is relatively easy, and the ratio can go up to one, namely R ( ˆ C ssl ) ∼ n − 1 u .
This is reminiscent of an existing result that uses labeled data and a large margin to achieve the n − 1 l rate [33].
If the tail sufﬁciency parameter s or the coverage parameter v is large, the ratio becomes approximately ( α + 1) / ( v + s ) .
Intuitively, a larger s or v indicates that there will be fewer high-conﬁdence unlabeled data to be transmitted to beneﬁt the classiﬁcation learning (on the evaluation measure P l of interest), which is in line with a slower rate of convergence n − ( α +1) /
( v + s ) u .
On the contrary, consider the other extreme that v = s = 0 .
Then, the ratio becomes q ( α +1) / { q ( α + 3) + d } , which matches an existing result in classiﬁcation learning [34].
For comparison, we deﬁne the baseline classiﬁer that only uses n l labeled data based on the kernel estimation in (10).
We denote that classiﬁer as ˆ C l .
The risk would be R ( ˆ C l ) ≤ C (cid:48)
n − q ( α +1) /
{ q ( α +3)+ d } l for some constant
C (cid:48) .
Comparing this with (13), we can determine the region where employing SSL can signiﬁcantly improve supervised learning.
To illustrate this point, let us suppose that
[Equation]
for some constant ζ ∈ (0 , 1) .
It can be veriﬁed that the bound of R ( ˆ C l ) is much larger than that of
R ( ˆ C ssl )
when
[Equation]
or equivalently,
[Equation]
The inequality (14) provides an insight into the critical region of n u where signiﬁcant improvement can be made from unlabeled data, as dependent on constants that describe the underlying function smoothness ( q ), data dimension ( d ), task difﬁculty ( α ), and “adequate transmission” parameters ( s, v ).
D.5 Proof of Theorem
We ﬁrst give a sketch of the proof.
We ﬁrst relate the risk bound of R ( ˆ C ssl ) to the estimation error of ˆ m ssl , and then decompose the error into a bias term and a variance term.
Each term is then bounded using concentration inequalities, in a way similar to the techniques used in [53, Ch. 5] and [34].
Different from the standard nonparametric analysis of classiﬁcation learning with IID data, we will use the aforementioned “adequate transmission” conditions to derive the rate of convergence from data that are contributed from both labeled and pseudo-labeled data.
The analysis involves a careful choice of the tuning parameters, e.g., the δ in Assumption (A1) and the kernel bandwidth, so that the biases introduced from pseudo-labeled data have a diminishing inﬂuence on the risk rate.
Next, we provide detailed proof.
We let n = n l + n u denote the total size of labeled and unlabeled data available to the SSL training.
For notational clarity, we sometimes put subscript n , e.g., δ n instead of δ (in Step 1), to highlight a quantity that is designed to vanish at some rate as n becomes large.
Recall that D ssl
=
D l ∪ D aug .
Let n l and n augu denote the sample sizes of D l and D aug , respectively.
Note that n augu is random since the Step 1 depends on n 0 labeled data.
We ﬁrst consider the risk conditional on a ﬁxed n augu , denoted by R n aug u ( ˆ C ssl ) .
Direct calculations show that
[Equation]
[Equation]
l
[Equation]
for an arbitrary t n > 0 to be selected.
From Assumption (A4), | m ( X ) − 1 / 2 | ≤ 1 / 2 , and 1 {| m ( X ) − 1 / 2 | > t n , ˆ C ssl ( X ) (cid:54) =
C ( X ) } ≤ 1 {| m ( X ) − ˆ m ( X ) | > t n } , we have
[Equation]
Moreover, by the triangle inequality, we have
[Equation]
where we deﬁne the function ¯ m by
[Equation]
if the denominator is nonzero [Equation] and ¯ m ( x ) = 0 otherwise.
In the sequel [Equation] we bound each term in (17).
First, we rewrite
[Equation] [Equation]
where P denotes the probability measure induced by D ssl (which is implicitly used to deﬁne ¯ m ).
For each x [Equation] we deﬁne the event
[Equation]
Then, from Assumption (A5) and the deﬁnition that K ( u ) =
1 {(cid:107) u (cid:107) ≤ 1 } , we have
[Equation]
[Equation]
Let B x,h ∆ = { u ∈ R
d : (cid:107) u − x (cid:107) ≤ h } denote the Euclidean ball of center x and radius h .
If we choose
[Equation]
the above inequality (19) implies that
[Equation]
Let c 9 ∆ = max v> 0 ve v .
Let { z i } M n i =1 be a set of points in R d such that [0 , 1] d ⊆ ∪ M n i =
1 B z i ,h n / 2 , with
M n = c 10 h − dn for some c 10 .
Taking (22) into (18) [Equation] and invoking Assumption (A3), we obtain
P l
[Equation]
| m
[Equation]
| > t n /
[Equation] (cid:1) [Equation] where we let ˜ n ∆ = n l + g v ( δ n ) n augu .
The technique of covering used in the last two inequalities was from [53, Eq. 5.1].
To bound the second term in (17), we write
[Equation]
Recall that D ssl
=
D l ∪ D aug .
For every ( Y l , X l ) ∈
D l , we have E ( Y l | X l ) = m ( X ) .
For any δ n that satisﬁes δ n ≤ min { c 3 , ∆ , 1 / 4 } , where c 3 was introduced in Assumption (A2) and ∆ was introduced in Assumption (A6), we have
[Equation] and similarly, P ( ˆ Y = 0 , ˜ Y =[-
 | ˜ X, X u ) = P ( ˆ Y =-] 1 | ˜ X, X u ) ≤ δ n + c 8 n − r 0 .
Thus,
[Equation]
Consequently, for every ( ˆ Y , ˜ X ) ∈ D aug , we have
[Equation]
25) where κ ( ˜ X ) ∆ =
E ( ˆ Y − ˜ Y | ˜ X ) ≤ 2 δ n + 2 c 8 n − r 0 .
Back in (24), let u ( Y ) =
Y if ( Y, X )
∈ D l and u ( Y ) =
˜ Y
if ( Y, X ) ∈ D aug , where ˜ Y is the pseudo-label random variable as in Assumption (A2) and equality (25).
In this way, we have E ( u ( Y ) |
X )
= m ( X ) .
We rewrite (24) as
[Equation]
¯
[Equation]
x
[Equation]
x ) [Equation] T 4 ( x ) , where
[Equation]
Let X ssl ∆ =
{ X : ( · , X ) ∈ D ssl } and X aug ∆ = { X : ( · , X ) ∈ D aug } .
Then, we can bound P (cid:0) | [Equation] m
[Equation]
| >
[Equation]
| [Equation] [Equation] ≤ P
[Equation]
>
[Equation]
|
[Equation]
+
[Equation]
provided that
[Equation]
In the above derivation
[Equation]
(27) uses the Hoeffding’s inequality
[Equation]
the fact that K 2 ( · ) =
K ( · )
[Equation]
and the triangle inequality, (28) uses the Hoeffding’s inequality again [Equation] (29) follows from (25) [Equation]
(30) is from
X aug ⊆
X ssl [Equation] and (31) is by the deﬁnition of the indicator function.
Consequently, with the choice of
[Equation]
n augu
[Equation] [Equation]
(33) we have
[Equation]
The ﬁrst term in (35), according to the Bernstein inequality
[Equation]
can be upper bounded by
[Equation]
Therefore
[Equation]
we can bound the second term in (17) by
[Equation]
The second term in (35), according to the same arguments as in (23), can be upper bounded by 8 · 256 · c 9 c 10 /
[Equation]
g
[Equation]
δ n ) t 2 n n augu h dn ) .
Therefore, we have
[Equation]
Combining inequalities (15), (16), (17), and (23), we obtain
[Equation]
Finally, we use a probabilistic lower bound of n augu to obtain the risk bound.
Let E denote the event min { 1 − ˆ m 0 ( X ) , ˆ m 0 ( X ) } ≤ δ n .
By the triangle inequality, assumptions (A1) and (A6), we have
P
[Equation]
[Equation]
Note that n aug u is a sum of n u IID Bernoulli random variables Z with probability P ( Z = 1) =
P u (min { 1 − ˆ m 0 ( X ) , ˆ m 0 ( X ) } ≤ δ n ) .
By the Hoeffding’s inequality, with probability at least 1 − 2 exp {− n u (˜ n u /n u ) 2 / 2 } , we have
[Equation]
where ˜ n u ∆
[Equation]
g [Equation] − [Equation]
[Equation]
Therefore, we have
[Equation]
provided that the choices of (20), (32), and (33) are made, namely
[Equation]
Choosing h n , t n , and δ n at the rate of
[Equation]
and invoking the assumption (A7), we can verify that the rate of convergence in (36) is at the order of
[Equation]
, which concludes the proof.