MELR: M ETA -L EARNING VIA M ODELING E PISODEL EVEL R ELATIONSHIPS FOR F EW -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Most recent few-shot learning (FSL) approaches are based on episodic training whereby each episode samples few training instances (shots) per class to imitate the test condition.
However, this strict adhering to test condition has a negative side effect, that is, the trained model is susceptible to the poor sampling of few shots.
In this work, for the ﬁrst time, this problem is addressed by exploiting interepisode relationships.
Speciﬁcally, a novel meta-learning via modeling episodelevel relationships (MELR) framework is proposed.
By sampling two episodes containing the same set of classes for meta-training, MELR is designed to ensure that the meta-learned model is robust against the presence of poorly-sampled shots in the meta-test stage.
This is achieved through two key components: (1) a Cross-Episode Attention Module (CEAM) to improve the ability of alleviating the effects of poorly-sampled shots, and (2) a Cross-Episode Consistency Regularization (CECR) to enforce that the two classiﬁers learned from the two episodes are consistent even when there are unrepresentative instances.
Extensive experiments for non-transductive standard FSL on two benchmarks show that our MELR achieves 1.0%–5.0% improvements over the baseline (i.e., ProtoNet) used for FSL in our model and outperforms the latest competitors under the same settings.
I NTRODUCTION
Deep convolutional neural networks (CNNs) have achieved tremendous successes in a wide range of computer vision tasks including object recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Russakovsky et al., 2015;
He et al., 2016a), semantic segmentation (Long et al., 2015;
Chen et al., 2018), and object detection (Ren et al., 2015; Redmon et al., 2016).
For most visual recognition tasks, at least hundreds of labeled training images are required from each class for training a CNN model.
However, collecting a large number of labeled training samples is costly and may even be impossible in real-life application scenarios (Antonie et al., 2001; Yang et al., 2012).
To reduce the reliance of deep neural networks on large amount of annotated training data, few-shot learning (FSL) has been studied (Vinyals et al., 2016; Finn et al., 2017;
Snell et al., 2017; Sung et al., 2018), which aims to recognize a set of novel classes with only a few labeled samples by knowledge transfer from a set of base classes with abundant samples.
Recently, FSL has been dominated by meta-learning based approaches (Finn et al., 2017; Snell et al., 2017; Sung et al., 2018; Lee et al., 2019;
Ye et al., 2020), which exploit the ample samples from base classes via episodic training.
During meta-training, to imitate an N -way K -shot novel class recognition task, an N -way K -shot episode/meta-task is sampled in each iteration from the base classes, consisting of a support set and a query set.
By setting up the meta-training episodes exactly the same way as the meta-test ones (i.e., N -way K -shot in the support set), the objective is to ensure that the meta-learned model can generalize to novel tasks.
However, this also leads to an unwanted side-effect, that is, the model will be susceptible to the poor sampling of the few shots.
Outlying training instances are prevalence in vision benchmarks which can be caused by various factors such as occlusions or unusual pose/lighting conditions.
When trained with ample samples, modern CNN-based recognition models are typically robust against abnormal instances as long as they are not dominant.
However, when as few as one shot per class is used to build a classiﬁer for
FSL, the poorly-sampled few shots could be catastrophic, e.g., when the cat class is represented in the support set by a single image of a half-occluded cat viewed from behind, it would be extremely hard to build a classiﬁer to recognize cats in the query set that are mostly full-body visible and frontal.
Existing episodic-training based FSL models do not offer any solution to this problem.
The main reason is that different episodes are sampled randomly and independently.
When the cat class is sampled in two episodes, these models are not aware that they are the same class, and thus cannot enforce the classiﬁers independently learned to be consistent to each other, regardless whether there exist poorly-sampled shots in one of the two episodes.
In this paper, a novel meta-learning via modeling episode-level relationships (MELR) framework is proposed to address the poor sampling problem of the support set instances in FSL.
In contrast to the existing episodic training strategy, MELR conducts meta learning over two episodes deliberately sampled to contain the same set of base classes but different instances.
In this way, cross-episode model consistency can be enforced so that the meta-learned model is robust against poorly-sampled shots in the meta-test stage.
Concretely, MELR consists of two key components: Cross-Episode
Attention Module (CEAM) and Cross-Episode Consistency Regularization (CECR).
CEAM is composed of a cross-episode transformer which allows the support set instances to be examined through attention so that unrepresentative support samples can be identiﬁed and their negative effects alleviated (especially for computing class prototypes/centers).
CECR, on the other hand, exploits the fact that since the two episodes contain the same set of classes, the obtained classiﬁers (class prototypes) should produce consistent predictions regardless whether there are any poorly-sampled instances in the support set and/or which episode a query instance comes from.
This consistency is enforced via cross-episode knowledge distillation.
Our main contributions are three-fold: (1)
For the ﬁrst time, the poor sampling problem of the few shots is formally tackled by modeling the episode-level relationships in meta-learning based FSL.
(2) We propose a novel MELR model with two cross-episode components (i.e., CEAM and CECR) to explicitly enforce that the classiﬁers of the same classes learned from different episodes need to be consistent regardless whether there exist poorly-sampled shots.
(3) Extensive experiments for non-transductive standard FSL on two benchmarks show that our MELR achieves signiﬁcant improvements over the baseline ProtoNet (Snell et al., 2017) and even outperforms the latest competitors under the same settings.
We will release the code and models soon.
R ELATED W ORK
Few-Shot Learning.
Few-shot learning (FSL) has become topical recently.
Existing methods can be generally divided into four groups: (1) Metric-based methods either learn a suitable embedding space for their chosen/proposed distance metrics (e.g., cosine similarity (Vinyals et al., 2016), Euclidean distance (Snell et al., 2017), and a novel measure SEN (Nguyen et al., 2020)) or directly learn a suitable distance metric (e.g., CNN-based relation module (Sung et al., 2018;
Wu et al., 2019), ridge regression (Bertinetto et al., 2019), and graph neural networks (Satorras & Estrach, 2018; Kim et al., 2019; Yang et al., 2020)).
Moreover, several approaches (Yoon et al., 2019; Li et al., 2019a; Qiao et al., 2019; Ye et al., 2020; Simon et al., 2020) learn task-speciﬁc metrics which are adaptive to each episode instead of learning a shared task-agnostic metric space.
(2) Model-based methods (Finn et al., 2017; Nichol et al., 2018; Rusu et al., 2019) learn good model initializations on base classes and then quickly adapt (i.e., ﬁnetune) them on novel classes with few shots and a limited number of gradient update steps.
(3) Optimization-based methods (Ravi & Larochelle, 2017;
Munkhdalai & Yu, 2017; Li et al., 2017) aim to learn to optimize, that is, to meta-learn optimization algorithms suitable for quick ﬁnetuning from base to novel classes.
(4) Hallucination-based methods (Hariharan & Girshick, 2017; Wang et al., 2018; Schwartz et al., 2018; Li et al., 2020) learn generators on base classes and then hallucinate new novel class data to augment the few shots.
Additionally, there are also other methods that learn to predict network parameters given few novel class samples (Qiao et al., 2018; Gidaris & Komodakis, 2019; Guo & Cheung, 2020).
Although the metric-based ProtoNet (Snell et al., 2017) is used as our baseline in this paper, our proposed MELR framework can be easily integrated with other episodic-training based methods.
Modeling Episode-Level Relationships.
In the FSL area, relatively less effort has been made to explicitly model the relationships across episodes.
For modeling such episode-level relationships, there are two recent examples: (1) LGM-Net (Li et al., 2019b) proposes an inter-task normalization strategy, which applies batch normalization to all support samples across a batch of episodes in each training iteration.
(2) Among a batch of episodes, Meta-Transfer Learning (Sun et al., 2019) records the class with the lowest accuracy in each episode and then re-samples ‘hard’ meta-tasks from the set of recorded classes.
In this work, instead of utilizing the relationships implicitly, we propose to model episode-level relationships (MELR) explicitly by focusing on episodes with the same set of classes.
Furthermore, our MELR is speciﬁcally designed to cope with the poor sampling of the few shots – an objective very different from those in (Li et al., 2019b; Sun et al., 2019).
Attention Mechanism.
Attention mechanism was ﬁrst proposed by (Bahdanau et al., 2015) for machine translation and has now achieved great success in natural language processing (Vaswani et al., 2017) and computer vision (Xu et al., 2015).
An attention module typically takes a triplet (queries, keys, values) as input and learns interactions between queries and key-value pairs according to certain task objectives.
It is referred to as self-attention or cross-attention depending on whether keys and queries are the same.
Several recent works (Hou et al., 2019; Guo & Cheung, 2020; Ye et al., 2020) have utilized attention mechanism for meta-learning based FSL.
CAN (Hou et al., 2019) employs cross-attention between support and query samples to learn better feature representations.
AWGIM (Guo & Cheung, 2020) adopts both self- and cross-attention for generating classiﬁcation weights.
FEAT (Ye et al., 2020) only uses self-attention on the class prototypes of the support set.
The biggest difference between these methods and our MELR lies in whether attention is modeled within each episode or across episodes.
Only MELR allows modeling cross-episode instance attention explicitly so that the meta-learned model can be insensitive to badly-sampled support set instances.
In addition, in our MELR, query set instances are also updated using cross-attention whilst existing models such as FEAT only apply attention to prototypes obtained using support set instances.
They thus cannot directly handle instance-level anomalies.
M ETHODOLOGY 3. P ROBLEM D EFINITION
Let D b
= { ( x i , y i ) | y i
∈ C b , i = 1 , 2 , · · · , N b } denote an abundant meta-training set from base classes C b , where x i is the i -th image, y i denotes the class label of x i , and N b is the number of images in D b .
Similarly, let D n = { ( x i , y i ) | y i ∈ C n , i = 1 , 2 , · · · , N n } denote a few-shot sample set from a set of novel classes C n (e.g., K -shot means that each novel class has K labeled images and N n = K |C n | ), where C b ∩ C n = ∅ .
We are also given a test set T from C n , where D n ∩ T = ∅ .
By exploiting D b and D n for training, the objective of few-shot learning (FSL) is to predict the class labels of test images in T .
3. M ETA -L EARNING BASED FSL
Most FSL methods are based on meta-learning (Vinyals et al., 2016; Finn et al., 2017;
Snell et al., 2017; Sung et al., 2018; Lee et al., 2019;
Ye et al., 2020), which adopt episodic training on the base class sample set D b and test their models over few-shot classiﬁcation tasks sampled from the novel classes C n .
Concretely, an N -way K -shot Q -query episode e =
( S e , Q e ) is generated as follows:
(1) We ﬁrst randomly sample a subset C e from base classes C b during meta-training (or from novel classes C n during meta-test) and re-index it as C e = { 1 , 2 , · · · , N } .
For each class in C e , K support and Q query images are then randomly sampled to form the support set S e = { ( x i , y i )
| y i ∈
C e , i = 1 , 2 , · · · , N × K } and the query set Q e = { ( x i , y i )
| y i ∈
C e , i = 1 , 2 , · · · , N × Q } ( S e ∩ Q e = ∅
), respectively.
A meta-learning based FSL approach typically designs a few-shot classiﬁcation loss over the query set Q e for each meta-training episode e :
[Equation]
where ψ denotes a feature extractor with an output dimension d , f ( · ; S e ) : R d → R N can be any scoring function constructed from the support set S e , and L ( · , · ) is the classiﬁcation loss (e.g., widely used cross-entropy).
By minimizing the above loss function via back propagation to update the part of the model to be meta-learned (e.g., ψ in ProtoNet), the model is trained over many meta-training episodes and then evaluated on the meta-test episodes.
[Figure]
3. M ODELING E PISODE -L EVEL R ELATIONSHIPS (MELR)
In the FSL area, relatively less effort has been made to explicitly model the relationships across episodes, and many FSL methods deﬁne their loss functions within each episode independently.
In contrast, with our Modeling Episode-Level Relationships (MELR), two N -way episodes are sampled in each training iteration from exactly the same set of N base classes.
Cross-Episode Attention
Module (CEAM) and Cross-Episode Consistency Regularization (CECR) are then devised to exploit this type of episode-level relationship explicitly (see Figure 1).
Cross-Episode Attention Module (CEAM).
We are given two N -way K -shot Q -query episodes e (1) =
( S (1) e , Q (1) e ) and e (2) =
( S (2) e , Q (2) e ) sampled from the same subset C e of C b , where C e is re-indexed as C e = { 1 , 2 , · · · , N } , and e (1) ∩ e (2) = ∅ .
For both episodes, to minimize the negative impact of badly-sampled few shots for a given query instance, we propose CEAM for cross-episode attention modeling, which is detailed below.
Concretely, let S (1) =
[ ψ ( x i ) T ; x i ∈ S (1) e ] ∈ R NK × d (or S (2) ) and Q (1) =
[ ψ ( x i ) T ; x i ∈ Q (1) e ] ∈
R NQ × d (or Q (2) ) denote the feature matrices of support and query samples in e (1) (or e (2) ), respectively, and let F (1) =
[ S (1) ; Q (1) ]
∈ R N ( K + Q ) ×
d (or F (2) =
[ S (2) ; Q (2) ] ) be the feature matrix of all samples in e (1) (or e (2) ).
For episode e (1) , CEAM takes the triplet ( F (1) , S (2) , S (2) ) as input, which corresponds to the input (queries, keys, values) in a typical attention module:
[Equation]
where the inputs are ﬁrst linearly mapped into a latent space with the same dimension of the feature space (using projection matrices W Q , W K , W V ∈ R d × d ):
[Equation]
Similarly, for episode e (2) , we have (analogous to Eq.
[Equation]
where the learnable parameters of fully connected layers (i.e., W Q , W K and W V ) are shared across
Eq.
(2) and Eq.
We can then obtain the transformed support and query embedding matrices in e (1) (or e (2) ) from ˆF (1) =
[ ˆS (1) ; ˆQ (1) ] (or ˆF (2) =
[ ˆS (2) ; ˆQ (2) ] ).
Cross-Episode Consistency Regularization (CECR).
In our MELR model, CEAM utilizes instance-level attention to alleviate the negative effects of the poor support set instance sampling so that each query set instance can be assigned to the right class with minimal loss.
Our CECR is designed to further reduce the model sensitivity to badly-sampled shots in different episodes by forcing the two classiﬁers learned over the two episodes to produce consistent predictions.
There are various options on how to enforce such consistency.
CECR adopts a knowledge distillation based strategy as empirically it is the most effective one (see Section 4.3).
Let f ( · ; ˆS (1) ) :
R d → R N and f ( · ; ˆS (2) ) :
R d → R N be the scoring functions of the two classiﬁers constructed from ˆS (1) and ˆS (2) , respectively.
To determine which classiﬁer/scoring function is stronger, we compute the few-shot classiﬁcation accuracies of the two classiﬁers on the merged query samples from both episodes.
Concretely, let ˆ Q (1) e
[Equation]
1 , 2 , · · · , NQ } (or ˆ Q (2) e ) denote the set of transformed embedding vectors of query samples in e (1) (or e (2) ), where ˆq (1) i (or ˆq (2) i ) denotes the i -th row of the transformed embedding matrix ˆQ (1) (or ˆQ (2) ).
With ˆ Q (1 , 2) e = ˆ Q (1) e ∪ ˆ Q (2) e = { ( ˆq (1 , 2) i , y (1 , 2) i ) , i = 1 , 2 , · · · , 2 NQ } ), we are able to compute the few-shot classiﬁcation accuracies of the two classiﬁers w.r.t.
the ground-truth labels and the corresponding predicted ones (i.e., arg max j σ j ( f ( ˆq (1 , 2) i ; ˆ S (1) )) and arg max j σ j ( f ( ˆq (1 , 2) i ; ˆ S (2) ))
( j = 1 , 2 , · · · , N ) for ( ˆq (1 , 2) i , y (1 , 2) i ) ∈
ˆ Q (1 , 2) e , where σ j ( v ) (cid:44) exp( v j ) (cid:80) Nj (cid:48)
=1 exp( v j (cid:48) )
( v ∈ R N ) is the softmax function).
The classiﬁer with higher accuracy is thus considered to be the stronger one and subsequently used as the teacher classiﬁer for the student to behave consistently with it.
Without loss of generality, we assume that f ( · ; ˆS (1) ) is stronger than f ( · ; ˆS (2) ) .
We choose the knowledge distillation loss (Hinton et al., 2015) for CECR, which is stated as:
[Equation]
where T is the temperature parameter as used in (Hinton et al., 2015).
More speciﬁcally, when the softmax function σ j ( v ; T ) (cid:44)
[Equation]
) ( v ∈ R N , j = 1 , 2 , · · · , N ) is used, we deﬁne
L (cid:48)
( f ( ˆq (1 , 2) i ; ˆS (1) ) , f ( ˆq (1 , 2) i ; ˆS (2) ); T ) in Eq.
(7) with the cross-entropy loss:
L (cid:48)
( f ( ˆq (1 , 2)
i
[Equation]
Note that we cut off the gradients over f ( · ; ˆS (1) ) when back-propagating since the output of the teacher scoring function is treated as the soft target for the student.
3. MELR- BASED FSL A LGORITHM
As we have mentioned above, in each training iteration, we randomly sample two N -way K -shot Qquery episodes e (1) =
( S (1) e , Q (1) e ) and e (2) =
( S (2) e , Q (2) e ) , which must have exactly the same set of classes but with different instances.
We ﬁrst transform the feature embeddings with Cross-Episode
Attention Module (CEAM) and then compute the few-shot classiﬁcation loss adopting ProtoNet (Snell et al., 2017) for both episodes ( e ∈ { e
[Equation]
) , e
[Equation] [Equation]
[Equation]
Next we determine the stronger/teacher episode to compute the Cross-Episode Consistency Regularization (CECR) loss between the two episodes.
The total loss for MELR is ﬁnally given by:
[Equation]
(10) where e ( t ) ∈ { e (1) , e (2) } denotes the teacher episode, and e ( s ) ∈ { e (1) , e (2) } is the student.
1: for all
MaxIteration do 2:
Randomly sample e (1) and e (2) from D b , satisfying that C e = C
) e , e
∩
Compute ˆF (1) for e (1) using CEAM with Eq.
(2), and obtain ˆF (2) with Eq.
(6) similarly.
Compute L fsc ( e (1) ) and L fsc ( e (2) ) with Eq.
(9), respectively; 5:
Construct ˆ Q (1 , 2) e = ˆ Q (1) e ∪ ˆ Q (2) e based on the two episodes; 6:
Determine the teacher episode e ( t ) and the student e ( s ) by computing the few-shot classiﬁcation accuracies of the two classiﬁers within e (1) and e (2) , respectively; 7:
Compute the CECR loss L cecr ( e ( t ) e ( s ) ; T ) with Eq
Compute the total loss L total with Eq.
Compute the gradients ∇ Θ L total ; 10:
Update Θ using stochastic gradient descent; 11: end for 12: return The model.
By combining CEAM and CECR for episodic training, our MELR-based FSL algorithm is summarized in Algorithm 1.
Once learned, with the optimal model found by our algorithm, we randomly sample multiple N -way K -shot meta-test episodes from C n for evaluation.
In other words, the episode-level relationship is only exploited during meta-training.
E XPERIMENTS 4. D ATASETS AND S ETTINGS
Datasets.
Two widely-used benchmarks are selected: (1) mini ImageNet (Vinyals et al., 2016):
It contains 100 classes from ILSVRC-12 (Russakovsky et al., 2015).
Each class has 600 images.
We split it into 64 training classes, 16 validation classes and 20 test classes, as in (Ravi & Larochelle, 2017).
(2) tiered ImageNet (Ren et al., 2018):
It is a larger subset of ILSVRC-12, containingclasses and 779,165 images in total.
We split it into 351 training classes, 97 validation classes and 160 test classes, as in (Ren et al., 2018).
All images of the two datasets are resized to 84 × 84 .
Evaluation Protocols.
The 5-way 5-shot/1-shot settings are used.
Each test episode e ( test ) =
( S ( test ) e , Q ( test ) e ) has 5 classes randomly sampled from the test split, with 5 or 1 shots and 15 queries per class.
We thus have N = 5 , K = 5 or 1 , Q = 15 as in previous works.
Although we meta-train our MELR with two episodes in each training iteration, we still evaluate it over test episodes one by one, strictly following the standard setting.
Moreover, since no cross-episode relationships can be used when testing, we take ( F ( test ) , S ( test ) , S ( test ) ) as the input of CEAM.
Note that the meta-test process is non-transductive since the embedding of each query sample in e ( test ) is independently updated using the keys and values coming from the support set.
We report average 5-way classiﬁcation accuracy (%, top-1) over 2,000 test episodes as well as the 95% conﬁdence interval.
Implementation Details.
Our MELR algorithm adopts Conv4-64 (Vinyals et al., 2016), Conv4and ResNet-12 (He et al., 2016b) as the feature extractors ψ for fair comparison with published results.
The output feature dimensions of Conv4-64, Conv4-512 and ResNet-12 are 64, 512, and 640, respectively.
To accelerate the entire training process, we pre-train all three backbones on the training split of each dataset as in many previous works (Zhang et al., 2020;
Ye et al., 2020;
Simon et al., 2020).
For ResNet-12, the stochastic gradient descent (SGD) optimizer is employed with the initial learning rate of 1e-4, the weight decay of 5e-4, and the Nesterov momentum of 0.9.
For Conv464 and Conv4-512, the Adam optimizer (Kingma & Ba, 2015) is adopted with the initial learning rate of 1e-4.
The hyper-parameters λ and T are respectively selected from { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } and { 16 , 32 , 64 , 128 } according to the validation performances of our MELR algorithm (see Appendix A.5 for more details).
The code and models will be released soon.
[Table] [Table]
M AIN R ESULTS
We compare our MELR with the representative/state-of-the-art methods for standard FSL on the two benchmark datasets in Table 1.
Note that we re-implement our baseline (i
[Table]
e., ProtoNet, denoted with † ) by sampling two episodes in each training iteration since it is still considered as a strong
FSL approach especially when the backbone is deep.
We can observe from Table 1 that: (1) Methods trained with ResNet-12 generally perform better than those employing shallower backbones.
Also, methods trained with Conv4-512 generally perform better than those employing Conv4even though the two backbones are of the same depth.
This is expected because deeper and wider backbones have better representation learning abilities.
(2) Our MELR achieves the new state-ofthe-art performance on both benchmarks under all settings.
Particularly, the improvements over the baseline (i.e., ProtoNet † ) range from 1.0% to 5.0%, which clearly validates the effectiveness and the strong generalization ability of our MELR.
On both benchmarks, the improvements obtained by our MELR over ProtoNet † under the 1-shot setting are signiﬁcantly larger than those under the 5-shot setting.
This demonstrates the superior performance of our MELR for FSL with less shots.
Again this is expected: FSL with less shots is more likely to suffer from the poor sampling of the few support instances; such a challenging problem is exactly what our MELR is designed for.
4. F URTHER E VALUATION
Ablation Study.
To demonstrate the contributions of each cross-episode learning objective in our
MELR, we conduct experiments on mini ImageNet by adding these learning objectives to the baseline (one at a time) under the 5-way 1-shot and 5-shot settings.
Note that ProtoNet without † is trained with one episode in each training iteration while ProtoNet † is trained with two episodes per iteration.
The ablation study results in Figure 2(a) show that: (1) Increasing mini-batch size helps
[Figure]
little for ProtoNet, indicating that our MELR beneﬁts from two cross-episode objectives rather than doubling the mini-batch size.
(2) CEAM or CECR alone clearly improves the performance of the baseline model and CEAM appears to be more beneﬁcial to FSL than CECR.
The combination of the two cross-episode learning objectives in our full model (i.e., MELR) achieves further improvements, suggesting that these two learning objectives are complementary to each other.
Moreover, in
Appendix A.2, we conduct more ablative experiments when the attention module is applied within each episode, validating the necessity of cross-episode attention.
Comparison to CEAM Alternatives.
As we have described, our CEAM takes the support samples as ‘keys’ and ‘values’, and all samples in one episode as ‘queries’ for attention module (denoted as Support → All).
We can also input prototypes (mean representations of support samples from the same class) as ‘keys’ and ‘values’ or input only query samples as ‘queries’ for attention.
This results in other three alternatives of CEAM:
Prototype → Query, Support → Query, and Prototype → All.
Note that under the 5-way 1-shot setting, Prototype → Query is equal to Support → Query and
Prototype → All is the same as Support → All.
Additionally, we compare to All → All: inputting all samples from the other episode as ‘keys’ and ‘values’ for CEAM when training but still testing as
Support → All (not violating the non-transductive setting).
From the comparative results of different choices in Figure 2(b), we can see that Support → All is the best for CEAM.
Moreover, All → All works worse than both Prototype → All and Support → All.
One possible explanation is that All → All exploits all query set instances during meta-training but only has access to the support set during meta-test to conform to the inductive learning setting.
This mis-match reduces its effectiveness.
Comparison to CECR Alternatives.
Our consistency regularization loss L cecr in Eq.
(7) is deﬁned with the knowledge distillation (KD) loss, which can be easily replaced by the negative cosine similarity (NegCos), symmetric Kullback–Leibler divergence (symKL), or the L2 distance (see Appendix A.1 for more details).
The results obtained by ProtoNet † +CECR using different consistency losses are shown in Figure 2(c).
It can be seen that ProtoNet † +KD performs slightly better than other implementations.
We thus choose KD as our CECR loss.
Visualizations of Data Distributions and Attention Maps.
MELR is designed to alleviate the negative effects of poorly-sampled few shots.
To validate this, we further provide some visualization results in Figure 3.
(1) We sample one episode in the test split of mini ImageNet under the 5-way 5-shot setting and obtain the embeddings of all images using the trained models of ProtoNet † ,
ProtoNet † +CEAM, and our MELR (i.e., ProtoNet † +CEAM+CECR), respectively.
We then apply tSNE to project these embeddings into a 2-dimensional space in Figure 3(a) – 3(c).
Across three subﬁgures, samples with the same color belong to the same class and diamonds are class prototypes/centers.
We can observe that adding CEAM makes the distribution of different classes more separable (see Figure 3(b) vs. 3(a)), validating the effectiveness of our CEAM.
Moreover, the embeddings obtained by our MELR are clearly much more evenly distributed with the prototypes right in the center, indicating less outlying instances (see Figure 3(c) vs. 3(b)).
This shows that when CECR is combined with CEAM, those badly-sampled shots in Figure 3(a) are now pulled back to the center of the class distributions.
(2) We also visualize the attention maps over the meta-test episode using our trained MELR.
Since we take all samples in the episode as ‘queries’, and support samples as ‘keys’ and ‘values’ for CEAM when meta-testing, each of 100 samples has a 25-dimensional weight vector under the 5-way 5-shot setting.
For each weight vector, we average the weights of the same
[Figure]
[Figure]
class and obtain a 5-dimensional vector.
For 75 query samples, we average the vectors of samples with the same class, resulting in a 5 × 5 instance attention map (see Figure 3(d)).
Similarly, we obtain the attention map for 25 support samples (see Figure 3(e)).
It can be seen that the two attention maps are very much alike, indicating that support and query sample embeddings are transformed by our CEAM in a similar way, which thus brings performance improvements for FSL.
Visualizations of the Generalization Ability of MELR.
We further provide the visualization of the generalization ability of our MELR during meta-test in Figure 4.
Concretely, we randomly sample 1,000 episode pairs from the test split of mini ImageNet under the 5-way 1-shot setting, where the two episodes in each pair have identical sets of classes.
We then compute the average 5-way classiﬁcation accuracy over all 2,000 episodes (from the 1,000 episode pairs) and the average
L cecr in Eq.
(7) over all 1,000 episode pairs at each training epoch.
We present the visualization results w.r.t.
accuracy and CECR loss in Figure 4.
As expected, the accuracy of our MELR is consistently higher than that of our baseline ProtoNet † .
Moreover, as compared with ProtoNet † , the CECR loss of our MELR is also lower across the whole training process, indicating that MELR has better performance consistency between two episodes.
This provides direct evidence that our
CEAM and CECR can boost the generalization ability of the learned model on novel classes.
C ONCLUSION
We have investigated the challenging problem of how to counter the negative effects of badlysampled few shots for FSL.
For the ﬁrst time, we propose to exploit the underlying relationships between training episodes with identical sets of classes explicitly for meta-learning.
This is achieved by two key components: CEAM is designed for neutralizing unrepresentative support set instances, and
CECR is to enforce the prediction consistency of few-shot classiﬁers obtained in the two episodes.
Extensive experiments for non-transductive standard FSL on two benchmarks show that our MELR achieves 1.0%–5.0% improvements over the baseline (i.e., ProtoNet) used for FSL in our model and outperforms the latest competitors under the same settings.
A A PPENDIX A. D ETAILS ABOUT CECR A LTERNATIVES
For the cross-episode consistency regularization (CECR) loss L cecr in Eq.
(7), we compare the knowledge distillation (KD) loss to the negative cosine similarity (NegCos), the L2 distance, and the symmetric Kullback-Leibler (KL) divergence (SymKL) in the main paper.
Here we give the details about three CECR alternatives.
Concretely, let σ (1) ( ˆq (1 , 2) i ) (or σ (2) ( ˆq (1 , 2) i ) ) denote the normalized vector of f ( ˆq (1 , 2) i ; ˆS (1) ) (or f ( ˆq (1 , 2) i ; ˆS (2) ) ) using softmax, we then have
[Equation] [Equation]
ˆ
[Equation] [Equation]
[Equation]
(13) where < · , · > is the inner product of two vectors, T is the temperature parameter, and KL ( u , v ) =
(cid:80) Nj =1 σ j ( u ) log σ j ( u ) σ j ( v ) ( u , v ∈ R N are two unnormalized scoring vectors, σ denotes the softmax function, and σ j ( u ) denotes the j -th element of σ ( u ) ).
A. M ORE A BLATIVE R ESULTS
In Table 2, we show more ablative results when the attention module is applied within each episode independently (named as Intra-Episode Attention Module (IEAM)).
Concretely, IEAM is used for
ProtoNet † +IEAM ( † means that ProtoNet is trained with two episodes in each training iteration for fair comparison) as follows (i = 1, 2):
[Equation] [Equation]
We also add our Cross-Episode Consistency Regularization (CECR) for ProtoNet † +IEAM+CECR to see the performance when IEAM instead of our CEAM is adopted.
We can see from Table 2 that adding IEAM to the baseline ProtoNet † also improves its performance, but IEAM is not as beneﬁcial as our CEAM (see ProtoNet † +IEAM vs. ProtoNet † +CEAM).
When our CECR is applied on top of ProtoNet † +IEAM (i.e., ProtoNet † +IEAM+CECR), the improvement is rather minor under the 5-way 1-shot setting and the result even gets worse under the 5-shot setting.
However, our MELR can still beneﬁt from CECR (see MELR vs. ProtoNet † +CEAM), indicating that CECR is not suitable for IEAM and our CEAM is necessary.
[Table]
A. M ORE V ISUALIZATION R ESULTS
Similar to Section 4.3, we provide more visualization results in Figure 5.
(1) We sample ﬁve episodes (corresponding to ﬁve rows in Figure 5) in the test split of mini ImageNet under the 5-way 5-shot setting and visualize the data distributions in the ﬁrst three columns using the trained models of
ProtoNet † , ProtoNet † +CEAM, and our MELR (i.e., ProtoNet † +CEAM+CECR), respectively.
We can observe that adding CEAM makes the distributions of different classes more separable in the ﬁrst four rows (see the second column vs. the ﬁrst column), validating the effectiveness of our
CEAM.
Moreover, the embeddings obtained by our MELR are clearly much more evenly distributed with the prototypes generally right in the center, indicating less poorly-sampled instances (see the third column vs. the second column).
Speciﬁcally, ProtoNet † +CEAM brings an obvious outlying instance in the last row, but adding CECR stabilizes the training of CEAM.
In a word, when CECR is combined with CEAM, the badly-sampled shots can be pulled back to the center of the class distributions.
(2) We also visualize the attention maps over each meta-test episode using our trained
MELR model.
For each of the ﬁve episodes, we obtain two 5 × 5 attention maps for query and
[Table]
Method
[Table]
MELR (
[Table]
[Figure]
support sets in the last two subﬁgures of each row, respectively.
It can be seen that the two attention maps in each row are very much alike, indicating that support and query sample embeddings are transformed by our CEAM in a similar way, which thus brings performance improvements for FSL.
A. R ESULTS FOR F INE -G RAINED FSL
To evaluate our MELR under the ﬁne-grained setting where the poorly-sampled shots may have greater negative impact since the classes are much more close, we conduct experiments on CUB200-2011 Birds (CUB) (Wah et al., 2011) with Conv4-64 as the feature extractor.
CUB has 200 ﬁnegrained classes of birds and 11,788 images in total.
We follow (Ye et al., 2020) and split the dataset into 100, 50, and 50 classes for training, validation, and test, respectively.
For direct comparison, we also use the pre-trained backbone model released by (Ye et al., 2020), which is pre-trained on the training set.
The comparative results in Table 3 show that: (1) Our MELR achieves the best results and improves over the second-best FEAT by 1.4% – 2.1%, validating the effectiveness of MELR under the ﬁne-grained setting.
(2) Our ProtoNet † +CEAM alone outperforms all the competitors, and adding CECR into ProtoNet † +CEAM (i.e., our MELR) further brings noticeable improvements (0.5% – 1.3%), indicating that both CEAM and CECR are crucial for ﬁne-grained FSL.
A. A NALYSIS OF H YPER - PARAMETER S ENSITIVITY
As we have mentioned in Section 4.1, the hyper-parameters λ and T are respectively selected from { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } and { 16 , 32 , 64 , 128 } according to the validation performance of our MELR algorithm.
Concretely, on mini ImageNet (with Conv4-64 as the feature extractor), we choose λ = 0 .
1 and T =
128 under the 5-way 1-shot setting, and choose λ = 0 .
05 and T = 64 under the 5-way 5-shot setting.
In Figure 6, we further present our hyper-parameter analysis on mini ImageNet.
The results show that our algorithm is quite insensitive to these parameters.
[Table] [Table]
[Equation]
A. R ESULTS BY V ARYING THE N UMBER OF E PISODES
We also conduct experiments by varying the number of episodes N e in each training iteration.
For the implementation of CEAM, we have two slightly different choices.
Concretely, for each episode e ( i ) ( i = 1 , · · · , N e ), the output of CEAM can be deﬁned as:
[Equation]
where [ S ( j ) ] N e j =1 ,j (cid:54) = i ∈ R NK ( N e − 1) ×
d is the concatenation of S ( j ) ∈ R NK × d ( j = 1 , · · · , i − 1 , i + 1 , · · · , N e ).
As for CECR, we determine the episode with the best accuracy as the teacher and distill knowledge to the rest N e − 1 student episodes.
The results on mini ImageNet using Conv4-64 in Table 4 show that the performance drops slightly as the number of episodes in each training iteration increases for both implementations.
One possible explanation is that too much training data make the model ﬁt better on the training set but fail to improve its generalization ability on novel classes.
A. C OMPARISON R EGARDING THE N UMBER OF P ARAMETERS
We select several representative/latest FSL models from Table 1 of the main paper and list their numbers of parameters in Table 5.
We can observe that: (1) With an extra CEAM in addition to the backbone (Conv4-64 or ResNet-12), our MELR has about 13% – 15% relatively more parameters than the baseline ProtoNet.
Note that CECR (in our MELR) leads to no extra parameters.
Considering the (statistically) signiﬁcant improvements achieved by our MELR over ProtoNet, we think that our MELR is cost-effective because it requires not much additional parameters.
The number of MELR’s parameters is almost the same as that of FEAT’s and is much less than that of PARN’s, but our MELR achieves better results than FEAT and PARN, indicating that our MELR is the most cost-effective among these three methods.
[Equation]
[Figure]
[Equation]
𝐱
[Figure]
[Table]
A. S CHEMATIC I LLUSTRATION OF CEAM
To demonstrate how our proposed CEAM can alleviate the negative effect of the poorly sampled shots, we present a schematic illustration of CEAM with a pair of episodes as its input in Figure 7.
For easy understanding (but without loss of generality), a toy visual example is considered: only one outlying instance x exists in the support set S (1) of the ﬁrst episode, but the support set S (2) of the second episode is properly sampled.
Since the two episodes are sampled from the same set of classes, the data distributions of ˜ S (1) =
S (1) \ { x } and S (2) are similar.
On one hand, when S (1) (including the outlier x ) is used as keys and values to update S (2) , the distribution of S (2) will not be inﬂuenced too much by the outlier x since all the shots in S (2) are far away from x and the weights on x will be very small.
That is, our CEAM is insensitive to few outliers in the keys and values.
On the other hand, when S (1) is transformed based on S (2) , the distribution of ˜ S (1) will be changed little (the data distributions of ˜ S (1) and S (2) are similar).
Particularly, for the outlier x , its updated embedding ˆx will be pulled back to ˜ S (1) (i.e., its negative effect is mitigated) since ˆx =
CEAM ( x , S (2) , S (2) )
≈ x + WS (2) ≈ x + ˜W˜S (1) , where x is the original embedding of x ,
S (2) and ˜S (1) are respectively the feature matrices of S (2) and ˜ S (1) , W and ˜W are two normalized weight matrices.
Note that this cannot be done by attending on x with S (1) .
Additionally, we provide the results obtained by an extra CEAM alternative in Table 6: only the embeddings of support samples are transformed (denoted as ‘Support → Support’) instead of transforming all samples as our choice.
We can see from Table 6 and also Figure 2(b) that our choice (i.e., Support → All) achieves the best results among all CEAM alternatives.
One possible explanation for why we resort to updating all samples is that transforming support and query samples into the
A. S IGNIFICANCE A NALYSIS OF CECR
Since CECR (in our MELR) requires no extra learnable parameters and only computes a loss for consistency constraint, it brings very limited computational cost.
Empirically, the meta-training time of MELR is almost the same as that of ProtoNet † +CEAM, indicating that the performance improvement over ProtoNet † +CEAM is obtained by CECR at an extremely low cost.
To study when CECR has a signiﬁcant impact on the ﬁnal FSL performance, we select 10 meta-test episodes from the 2,000 ones used in the evaluation stage and visualize them in Figure 8.
Concretely, for each of the 10 selected meta-test episodes, we visualize three data distributions (from left to right) obtained by ProtoNet † , ProtoNet † +CEAM, and our MELR, respectively.
That is, each meta-test episode is denoted by a group of three subﬁgures.
In each subﬁgure, we compute the test accuracy over query samples and present it as the title.
We can see that: (1) When the few-shot classiﬁcation task is hard (i.e., ProtoNet † obtains relatively low accuracy), CEAM leads to signiﬁcant improvements (about 3% – 9%).
In the same hard situation, CECR further achieves signiﬁcant improvements (about 5% – 8%) on top of CEAM and shows its great effect on the ﬁnal FSL performance.
This indicates that CECR and CEAM are complementary to each other in hard situations, and thus both are crucial for solving the poor sampling problem in meta-learning based FSL.
A. R ESULTS OF T RANSDUCTIVE FSL
The main difference between standard and transductive FSL is whether query samples are tested one at a time or all simultaneously.
As we have mentioned in Section 4.1, we evaluate our MELR model strictly following the non-transductive setting for standard FSL since the embedding of each query sample in the test episode is independently transformed using the keys and values coming from the
[Table]
support set.
However
[Table]
in this section
[Table]
we further conduct experiments under the transductive FSL setting to study how well our MELR can make use of the unlabeled query samples.
Concretely, for each meta-test episode e ( test ) , we input all samples (both support and query ones) as keys and values into the trained CEAM:
[Equation]
(17) such that the relationships of all unlabeled query samples can be taken into consideration.
With the transformed embeddings, we make predictions for all query samples based on Semi-ProtoNet (Ren et al., 2018), which utilizes the unlabeled query samples to help construct better class prototypes and then makes predictions similar to ProtoNet.
To match the meta-test process, we also make changes to meta-training accordingly.
Speciﬁcally, for one episode (out of the two) in each training iteration, we use all samples from the other episode as keys and values for CEAM to update all of its own embeddings.
This is followed by obtaining the prototypes based on Semi-ProtoNet as well as computing the FSL loss and CECR loss.
The results of transductive FSL on mini ImageNet are shown in Table 7.
It can be seen that: (1)
By utilizing the unlabeled query samples under transductive FSL, our MELR achieves further improvements, as compared to MELR under standard FSL.
Particularly, the performance improvement under 1-shot (i.e., 6.3%) is more signiﬁcant than that under 5-shot (i.e., 2.6%), indicating that exploiting unlabeled query samples brings more beneﬁts to FSL with less labeled support samples.
(2) Our MELR achieves the best results among all the transductive FSL methods.
Speciﬁcally,
MELR outperforms FEAT by a large margin (2.0% – 4.6%).
Since FEAT also makes predictions based on Semi-ProtoNet, this clearly validates the effectiveness of our MELR under the transductive setting.
(3) Our ProtoNet † +CEAM alone outperforms all the competitors, and adding CECR into
ProtoNet † +CEAM (i.e., our MELR is obtained) further brings noticeable improvements (0.6% – 1.4%), indicating that both CEAM and CECR play important roles under transductive FSL.