An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning
Abstract
Semi-supervised few-shot learning consists in training a classiﬁer to adapt to new tasks with limited labeled data and a ﬁxed quantity of unlabeled data.
Many sophisticated methods have been developed to address the challenges this problem comprises.
In this paper, we propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective, and then augment the extremely label-constrained support set in fewshot classiﬁcation tasks.
Our approach can be implemented in just few lines of code by only using off-the-shelf operations, yet it is able to outperform state-of-the-art methods on four benchmark datasets.
Introduction
Deep learning [16] allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction, which has already demonstrated its powerful capabilities in many computer vision tasks, e g. , object recognition [7], ﬁne-grained classiﬁcation [39], object detection [18], etc.
However, deep learning based models always require large amounts of supervised data for good generalization performance.
Few-Shot Learning (FSL) [37], as an important technique to alleviate label dependence has received great attention in recent years.
It has formed several learning paradigms including metric-based methods [29, 33, 45], optimizationbased methods [4, 25, 28] and transfer-learning based methods [3, 24].
More recently, it is intriguing to observe that there has been extensive research in FSL on exploring how to utilize unlabeled data to improve model performance under few-shot supervisions, which is Semi-Supervised Few-Shot Learning (SSFSL)
The most popular fashion of SSFSL is to predict unlabeled data with pseudo-labels by carefully devising tailored strategies, and then augment the extremely small support set of labeled data in few-shot classiﬁcation, e.g. , [9, 15, 36].
In this paper, we follow this fashion and propose a simple but quite effective approach to
SSFSL, i.e. , a Method of sUccesSIve exClusions (MUSIC), cf.
Figure 1.
As you can imagine, in such label-constrained tasks, e.g. , 1-shot classiﬁcation, it would be difﬁcult to learn a good classiﬁer, and thus cannot obtain sufﬁciently accurate pseudo-labels. Therefore, we [Figure]
think about the problem in turn, and realize the process of pseudo-labeling in SSFSL as a series of successive exclusion operations.
In concretely, since it is hard to annotate which class the unlabeled data belongs to, in turn, it should be relatively easy 2 to predict which class it does not belong to based on the lowest conﬁdence prediction score.
Thus, if we treat the predicted pseudo-labels in the previous traditional way as labeling positive labels, our exclusion operation is to assign negative pseudo-labels to unlabeled data.
In the following, we can use the negative learning paradigm [10] to update the classiﬁer parameters and continue the negative pseudo-labeling process by excluding the predicted negative label in the previous iteration, until all negative pseudo-labels are obtained.
Moreover, it is apparent to ﬁnd that when all negative labels of unlabeled data are sequentially excluded and labeled, their positive pseudo-labels are also obtained.
We can thus eventually augment the small support set with positive pseudo-labels, and fully utilize the auxiliary information from both labeled base-class data and unlabeled novel-class data in SSFSL.
Also, in our MUSIC, to further improve few-shot classiﬁcation accuracy, we equip a minimum-entropy loss into our successive exclusion operations for enhancing the predicted conﬁdence of both positive and negative labels.
In summary, the main contributions of this work are as follows:
• We propose a simple but effective approach,
i
e. , MUSIC, to deal with semi-supervised few-shot classiﬁcation tasks.
To our best knowledge, MUSIC is the ﬁrst approach to leverage negative learning as a straightforward way to provide pseudo-labels with as much conﬁdence as possible in such extremely label-constrained scenarios.
• We can implement the proposed approach using only off-the-shelf deep learning computational operations, and it can be implemented in just few lines of code.
Besides, we also provide the default value recommendations of hyper-parameters in our MUSIC, and further validate its strong practicality and generalization ability via various SSFSL tasks.
• We conduct comprehensive experiments on four few-shot benchmark datasets,
i
e. , miniImageNet , tieredImageNet , CIFAR-FS and CUB , for demonstrating our superiority over state-of-the-art FSL and SSFSL methods.
Moreover, a series of ablation studies and discussions are performed to explore working mechanism of each component in our approach.
Related Work
Few-shot learning The research of few-shot learning [4, 29, 33, 42, 45] aims to explore the possibility of endowing learning systems the ability of rapid learning for novel categories from a few examples.
In the literature, few-shot learning methods can be roughly separated into two groups: 1)
Meta-learning based methods and 2) Transfer-learning based methods.
Regarding meta-learning based methods, aka “learning-to-learn”, there are two popular learning paradigms, i.e. , metric-based methods [29, 33, 45] and optimization-based methods [4, 25, 28].
More speciﬁcally, Prototypical Networks [29] as a classical metric-based method was considered to generate an embedding in which data points cluster around a single prototype representation for each class.
DeepEMD [45] proposed to adopt the Earth Mover’s Distance as a metric to compute a structural distance between dense image representations to determine image relevance for few-shot learning.
For optimization-based methods, MAML [4] learned an optimization method to follow the fast gradient direction to rapidly learn the classiﬁer for novel classes.
In [25], it reformulated the parameter update into an LSTM and achieved this via a meta-learner.
Regarding transfer-learning based methods, they are expected to leverage techniques to pre-train a model on the large amount of data from the base classes, without using the episode training strategy.
The pre-trained model is then utilized to recognize novel classes of few-shot classiﬁcation.
In concretely, [24] proposed to directly set the ﬁnal layer weights from novel training examples during few-shot learning as a weight imprinting process.
In [3], the authors investigated and shown such transfer-learning based methods can achieve competitive performance as meta-learning methods.
Semi-supervised few-shot learning Semi-Supervised Learning (SSL) is an approach to machine learning that combines a small amount of labeled data with a large amount of unlabeled data during training [6, 46].
In the era of deep learning, SSL generally utilizes unlabeled data from the following perspectives, e.g. , considering consistency regularization [14], employing moving average strategy [30], applying adversarial perturbation regularization [22], etc.
In recent years, the use of unlabeled data to improve the accuracy of few-shot learning has received increasing attention [9, 15, 19, 23, 36, 44], which leads to the family of Semi-Supervised FewShot Learning (SSFSL) methods.
However, directly applying SSL methods to few-shot supervised scenarios usually causes inferior results due to the extreme small number of labeled data, e.g. , 1shot.
More speciﬁcally, to deal with the challenging SSFSL, Ren et al.
[26] extended Prototypical
Networks [29] to use unlabeled samples when producing prototypes.
TPN [19] was developed to propagate labels from labeled data to unlabeled data by learning a graph that exploits the manifold structure of the data.
Recently, state-of-the-art SSFSL methods, e.g. , [9, 15, 36], were proposed to predict unlabeled data by pseudo-labeling and further augment the label-constrained support set in few-shot classiﬁcation.
Distant from previous work, to our best knowledge, we are the ﬁrst to explore leveraging complementary labels ( i.e. , negative learning) to pseudo-label unlabeled data in SSFSL.
Negative learning As an indirect learning method for training CNNs, Negative Learning (NL) [10] was proposed as a novel learning paradigm w.r.t.
typical supervised learning (aka Positive Learning,
PL).
More speciﬁcally, PL indicates that “input image belongs to this label”, while NL means “input image does not belong to this complementary label”.
Compared to collecting ordinary labels in PL, it would be less laborious for collecting complementary labels in NL [10].
Therefore, NL can not only be easily combined with ordinary classiﬁcation [5, 10], but also assist various vision applications, e.g. , [12] dealing with noisy labels by applying NL, [35] using unreliable pixels for semantic segmentation with NL, etc.
In this paper, we attempt to leverage NL to augment the few-shot labeled set by predicting negative pseudo-labels from unlabeled data, and thus obtain more accurate pseudo labels to assist classiﬁer modeling under label-constrained scenarios.
Methodology 3.1 Problem Formulation
Deﬁnition In Semi-Supervised Few-Shot Learning (SSFSL), we have a large-scale dataset D base containing many-shot labeled data from each base class in C base , and a small-scale dataset D novel consisting of few-shot labeled data as a support set S from the category set C novel , as well as a certain number of unlabeled data U acquired also from C novel .
Note that, D novel is disjoint from D base for generalization test.
The task of SSFSL is to learn a robust classiﬁer f ( · ; θ ) based on both S and U for making predictions on new queries Q from D novel , where D base is utilized as auxiliary data.
Setting Regarding the basic semi-supervised few-shot classiﬁcation setting, it generally faces the
N -way- K -shot problem, where only K labeled data from S and U unlabeled data from U per class are available to learn an N -way classiﬁer.
In this setting, queries in Q are treated independently of each other, and are not observed in U .
It is referred to as inductive inference .
For another important setting in SSFSL, i.e. , transductive inference , the query set Q is observed also during training and joint with U .
3.2 MUSIC: A Simple Method of sUccesSIve exClusions for SSFSL
The basic idea of our MUSIC is to augment the few-shot labeled set (the support set) S by predicting “negative” ( i.e. , “saying not belonging to”) pseudo-labels to unlabeled data U , particularly for such label-constrained scenarios.
Given an image I , we can obtain its representation by training a deep network F ( · ; Θ) based on auxiliary data D base :
[Equation]
where Θ is the parameter of the network.
After that, F ( · ; Θ) is treated as a general feature embedding function for other images and Θ is also ﬁxed [31].
Then, considering the task of c -class classiﬁcation, the aforementioned classiﬁer f ( · ; θ ) maps the input space to a c -dimensional score space as
[Equation]
where p is indeed the predicted probability score belonging to the c -dimensional simplex ∆ c − 1 , softmax ( · ) is the softmax normalization, and θ is the parameter.
In SSFSL, θ is randomly initialized and ﬁne-tuned only by NK labeled data in S by the cross-entropy loss:
[Equation]
where y ∈ R c is a one-hot vector denoted as the ground-truth label w.r.t.
x , and y k and p k is the k -th element in y and p , respectively.
To augment the limited labeled data in S , we then propose to predict unlabeled images ( e.g. , I u ) in U with pseudo-labels from an indirect learning perspective, i.e. , excluding negative labels.
In concretely, regarding a conventional classiﬁcation task, the ground-truth y k = 1 represents that its data x belongs to class k , which can be also termed as positive learning .
In contrast, we hereby denote another one-hot vector y ∈ R c as the counterpart to be the complementary label [10, 12], where y k = 1 means that x does not belong to class k , aka negative learning .
Due to the quite limited labeled data in few-shot learning scenarios, the classiﬁer f ( · ; θ ) is inaccurate to assign correct positive labels to I u .
On the contrary, however, it could be relatively easy and accurate to give such a negative pseudo-label to describe that I u is not from class k by assigning y uk = 1 .
Therefore, we realize such an idea of “exclusion” by obtaining the most conﬁdent negative pseudo-label based on the class having the lowest probability score.
The process is formulated as:
[Equation]
where p u represents the prediction probability w.r.t.
I u , and δ is a reject option to ensure that there is sufﬁciently strong conﬁdence to assign pseudo-labels.
While if all p uk are larger than δ , no negative pseudo-labels are returned for I u in this iteration.
Thus, after obtaining samples and negative pseudo-label pairs ( I u , y u ) , f ( · ; θ ) can be updated by
[Equation]
In the next iteration, we exclude the k -th class, i.e. , the negative pseudo-label in the previous iteration, from the remaining candidate classes.
After that, the updated classiﬁer is employed to give the probability score p u \ k ∈ R c − 1 of I u , without considering class k .
The similar pseudo-labeling process is conducted in a successive exclusion manner until all negative pseudo-labels are predicted according to Eqn.
(4), or no negative pseudo-labels are able to be predicted with a strong conﬁdence.
Finally, in the last iteration, for those samples in U whose negative labels are all labeled, their positive pseudo-labels are naturally available.
We can further update the classiﬁer by following Eqn.
(3) based on the ﬁnal positive labels.
Then, the updated classiﬁer f ( · ; θ ) is ready for predicting Q as evaluation.
Moreover, to further improve the probability conﬁdence and then promote pseudo-labeling, we propose to equip a minimum-entropy loss (MinEnt) upon p u by optimizing the following objective:
[Equation]
[Table]
It could sharp the distribution of p u and discriminate the conﬁdence of both positive and negative labels.
Algorithm 3.2 provides the pseudo-code of our MUSIC.
Experiments 4.1 Datasets and Empirical Settings
We conduct experiments on four widely-used few-shot learning benchmark datasets for general object recognition and ﬁne-grained classiﬁcation, including miniImageNet [25], tieredImageNet [26],
CIFAR-FS [2] and CUB [34].
Speciﬁcally, miniImageNet consists of 100 classes with 600 samples of 84 × 84 resolution per class, which are selected from ILSVRC-2012 [27].
tieredImageNet is a larger subset from ILSVRC-2012 with 608 classes in a man-made hierarchical structure, where its samples are also of 84 × 84 image resolution.
CIFAR-FS is a variant of CIFAR-100 [13] with low resolution, which has 100 classes and each of them has 600 samples of 32 × 32 size.
Regarding CUB , it is a ﬁne-grained classiﬁcation dataset of 200 different bird species with 11,788 images in total.
For fair comparisons, we obey the protocol of data splits in [9, 15, 36] to train the feature embedding function and conduct experiments for evaluations in SSFSL.
We choose the commonly used ResNet12 [7] as the backbone network, and the network conﬁgurations are followed [9, 15, 36].
For pre-training, we just follow the same way of [38] to pre-train the network, but do not use any pseudo labels during pre-training.
For optimization, Stochastic Gradient Descent (SGD) with momentum of 0 . 9 and weight decay of 5 × 10 − 4 is adopted as the optimizer to train the feature extractor from scratch.
The initial learning rate is 0 . 1 , and decayed as 6 × 10 − 3 , 1 . 2 × 10 − 3 and 2 .
4 × 10 − 4 after 60, 70 and 80 epochs, by following [38].
Regarding the hyper-parameters in MUSIC, the reject option δ in Eqn.
(4) is set to 1 c and the trade-off parameter over Eqn.
(6) is set to 1 as default for all experiments and iterations, which shows its practicality and non-tricky.
During evaluation, the last layer of pre-trained model is replaced by an (cid:96) 2 -normalization layer and a c -dimensional fully connected layer as the classiﬁer.
We also use SGD for optimization.
Our MUSIC and all baselines are evaluated over 600 episodes with 15 test samples in each class.
All experiments are conducted by MindSpore with a GeForce RTX 3060 GPU.
4.2 Main Results
We report the empirical results in the following four setups.
All results are the average accuracy and the corresponding 95% conﬁdence interval over the 600 episodes are also conducted.
Basic semi-supervised few-shot setup We compare our MUSIC with state-of-the-art methods in the literature in Table 1. As shown, our simple approach outperforms the competing methods of both generic few-shot learning and semi-supervised few-shot learning by a large margin across different few-shot tasks over all the datasets. Beyond that, we also report the results of solely using [Table]
COSOC [21]
ResNet69.
SetFeat [1]
ResNet68.
MCL [20]
ResNet69.
STL DeepBDC [41]
ResNet67.
TPN [19] 4 CONV 52.
TransMatch [44]
WRN-2860.
LST [17]
ResNet70.
EPNet [23]
ResNet70.
ICI [36]
ResNet69.
iLPC
ResNet70.
[Table] [Table]
[Table]
PLCM [9]
ResNet72.
[Table] [Table]
ResNet
[Table]
[Table] [Table]
[Table] [Table]
Ours
[Table]
ResNet73.
[Table]
[Table]
[Table]
the pseudo-labeled negative or positive samples generated by our MUSIC, which is denoted by “Ours (only neg)” or “Ours (only pos)” in that table.
It is apparent to observe that even only using negative pseudo-labeling, MUSIC can still be superior to other existing FSL methods.
Moreover, compared with the results of only using positive pseudo-labeling, the results of only using negative are worse.
It reveals that accurate positive labels still provide more information than negative labels [10].
Transductive semi-supervised few-shot setup In the transductive setup, it is available to access the query data in the inference stage.
We also perform experiments in such a setup and report the results in Table 2.
As seen, our approach can still achieve the optimal accuracy on all the four datasets, which justiﬁes the effectiveness of our MUSIC.
Regarding the comparisons between (only) using negative and positive pseudo-labels, it has similar observations as those in Table 1. [Table] [Figure]
[Table] [Table] [Table] Distractive semi-supervised few-shot setup In real applications, it might not be realistic to collect a clean unlabeled set without mixing any data of other classes.
To further validate the robustness of
MUSIC, we conduct experiments with the distractive setup, i.e. , the unlabeled set contains distractive classes which are excluded in the support set.
In that case, positive pseudo-labels are more prone to error, while negative pseudo-labels have a much lower risk of error.
Table 3 presents the comparison results and shows that our approach can perform as the best solution in all distractive semi-supervised few-shot classiﬁcation tasks.
Variety-unlabeled semi-supervised few-shot setup In order to analyze the performance in the case of different unlabeled samples
, we perform our MUSIC under the variety-unlabeled semisupervised setup and compare with state-of-the-arts, e.g. , ICI [36], LST [17] and PLCM [9].
As shown in Figure 2, our approach signiﬁcantly outperforms over these methods in different K -shot tasks of SSFSL.
It further validates the effectiveness and generalization ability of our MUSIC.
4.3 Ablation Studies and Discussions
We hereby analyze and discuss our MUSIC approach by answering the following questions based on ablation studies on two datasets, i.e. , miniImageNet and CUB .
Will negative pseudo-labels be easier to predict under SSFSL than positive ones?
As assumed previously, in such an extremely label-constrained scenario, e.g. , 1-shot learning, it might be hard to learn an accurate classiﬁer for correctly predicting positive pseudo-labels.
In this sub-section, we conduct ablation studies by alternatively performing negative and positive pseudo-labeling to verify this assumption.
In Table 4, different settings denote different orders of negative and positive pseudo-labeling in SSFSL.
For example, “neg → pos → · · · ” represents that we ﬁrstly obtain negative pseudo-labels by our MUSIC (without using the ﬁnal positive labels) and update the model, and then we obtain positive pseudo-labels 3 and update model, and so on.
Regarding the iteration time, it is relevant to the number of K in the K -way classiﬁcation. In concretely, for 5 -way classiﬁcation, our MUSIC returns the most conﬁdent negative pseudo-label in the current iteration and excludes it for the next iteration. Thus, after four times of “neg → pos”, all negative pseudo-labelings are ﬁnished and the results can be reported. Similarly, “pos → neg → · · · ” means that we get the positive pseudo-labels ﬁrst, followed by the negative ones.
As the results shown in Table 4, we can see that obtaining negative pseudo-labels ﬁrst obviously achieves better results than positive ﬁrst, which shows that labeling negative pseudo-labels ﬁrst can lay a better foundation for model training, and further answers the question in this sub-section as “ YES ”.
Is the minimum-entropy loss effective?
In our MUSIC, to further improve the probability conﬁdence and then promote pseudo-labeling, we equip the minimum-entropy loss (MinEnt).
We here test its effectiveness and report the results in Table 5.
It can be found that training with MinEnt ( i.e. , the proposed MUSIC) brings 0.2 ∼ 0.3% improvements over training without MinEnt in SSFSL.
[Table]
Is the reject option δ effective?
We hereby verify the effectiveness and necessity of the reject option δ in MUSIC.
The δ in our approach acts as a safeguard to ensure that the obtained negative pseudo-labels are as conﬁdent as possible.
We present the results in
Table 6, and can observe that MUSIC with δ achieves signiﬁcantly better few-shot classiﬁcation accuracy than MUSIC without δ .
Additionally, even without δ , our approach can still perform well, i.e. , the results being comparable or even superior to the results of state-of-the-arts.
[Figure]
What is the effect of iteration manner in our MUSIC?
As aforementioned, our approach works as a successive exclusion manner until all negative pseudo-labels are predicted, and eventually obtaining positive pseudo-labels.
As pseudo-labeling conducting, it is interesting to investigate how the performance changes as the iteration progresses.
We report the corresponding results in Figure 3.
As shown, on each task of these two datasets, our approach all shows a relatively stable growth trend, i.e. , 0.5 ∼ 2% improvements over the previous iteration.
What is the performance of pseudo-labeling in MUSIC?
In this sub-section, we explicitly investigate the error rates of both negative and positive pseudo-labels predicted by our approach.
We take 5-way-5-shot classiﬁcation on miniImageNet and CUB as examples, and ﬁrst present the pseudo-labeling error rates of negative labels in Table 7.
Since the task is 5-way prediction, there are totally four iterations of negative pseudo-labeling in MUSIC reported in that table.
Except for error rates, we also detailedly report the number of wrong labeled samples in each iteration, as well as the total number of labeled samples.
Note that, in the third and forth iterations of negative pseudo-labeling, the total number of labeled samples are less than the number of unlabeled data ( i.e. , 250), which is due to the reject option in MUSIC.
That is to say, those samples cannot be pseudo-labeled with a strongly high conﬁdence.
Meanwhile, we also see that, as the pseudo-labeling progresses, the error rates slowly increase, but the ﬁnal error rate of negative labeling is still no higher than 6.7%.
It demonstrates the effectiveness of our approach from a straightforward view.
On the other side, Table 8 compares the positive pseudo-labeling error rates, and also reports the proportion of labeled samples in the total number of unlabeled samples.
Regarding ICI [36] and iLPC [15], although they designed tailored strategies to ensure the correctness of pseudo-labels, e.g. , instance credibility inference [36] and label cleaning [15], these methods still have high pseudolabeling error rates (over 25%).
Compared with them, our approach has signiﬁcantly low error rates, i.e. , about 10%. Meanwhile, we also note that our MUSIC only predicts about 80% of the unlabeled data, which can be regarded to be relatively conservative. However, it reveals that our
[Table] [Table]
[Figure]
approach still has a large room for performance improvement.
Moreover, Table 8 also shows that, even if our approach removes the reject option strategy, its error rates are still lower than those of state-of-the-arts.
Additionally, we visualize the positive pseudo-labels with high conﬁdence by t -SNE [32] in Figure 4.
Compared with these methods [Table] we can obviously ﬁnd that the positive samples with high conﬁdence predicted by our MUSIC are both more centralized and distinct.
This also explains the satisfactory performance of our approach when using the positive pseudo-labels and using the positive alone (cf.
Table 1 and Table 2) from the qualitative perspective.
[Table]
Are the pseudo-labels of MUSIC a balanced distribution?
In this sub-section, we are still interested in investigating what kind of data distribution the pseudo-labeled samples are to further analyze how our approach works well.
As shown in Table 9, we present the averaged number of both negative and positive pseudo-labeled samples in all 600 episodes of 5-way-5-shot classiﬁcation tasks on miniImageNet .
It is apparent to see that the pseudo-labeled samples present a very clearly balanced distribution, which aids in the modeling of classiﬁers across different classes in SSFSL.
Conclusion
In this paper, we dealt with semi-supervised few-shot classiﬁcation by proposing a simple but effective approach, termed as MUSIC.
Our MUSIC worked in a successive exclusion manner to predict negative pseudo-labels with much conﬁdence as possible in the extremely label-constrained tasks.
After that, models can be updated by leveraging negative learning based on the obtained negative pseudo-labels, and continued negative pseudo-labeling until all negative labels were returned.
Finally, combined with the incidental positive pseudo-labels, we augmented the small support set of labeled data for evaluation in SSFSL.
In experiments, comprehensive empirical studies validated the effectiveness of MUSIC and revealed its working mechanism.
In the future, we would like to investigate the theoretical analyses about our MUSIC in terms of its convergence and estimation error bound, as well as how it performing on traditional semi-supervised learning tasks.