A Closer Look at Prototype Classifier for Few-shot Image Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
The prototypical network is a prototype classifier based on meta-learning andis widely used for few-shot learning because it classifies unseen examples byconstructing class-specific prototypes without adjusting hyper-parameters duringmeta-testing.
Interestingly, recent research has attracted a lot of attention, showingthat training a new linear classifier, which does not use a meta-learning algorithm,performs comparably with the prototypical network.
However, the training ofa new linear classifier requires the retraining of the classifier every time a newclass appears.
In this paper, we analyze how a prototype classifier works equallywell without training a new linear classifier or meta-learning.
We experimentallyfind that directly using the feature vectors, which is extracted by using standardpre-trained models to construct a prototype classifier in meta-testing, does notperform as well as the prototypical network and training new linear classifiers onthe feature vectors of pre-trained models.
Thus, we derive a novel generalizationbound for a prototypical classifier and show that the transformation of a featurevector can improve the performance of prototype classifiers.
We experimentallyinvestigate several normalization methods for minimizing the derived bound andfind that the same performance can be obtained by using the L2 normalization andminimizing the ratio of the within-class variance to the between-class variancewithout training a new classifier or meta-learning.
Introduction
Few-shot learning is used to adapt quickly to new classes with low annotation cost.
Meta-learning isa standard training procedure to tackle the few-shot learning problem and the prototypical network[1], a.k.a ProtoNet is a widely used meta-learning algorithm for few-shot learning.
In ProtoNet, weuse a prototype classifier based on meta-learning to predict the classes of unobserved objects byconstructing class-specific prototypes without adjusting the hyper-parameters during meta-testing.
ProtoNet has two advantages.
(1) Since the nearest neighbor method is applied on query data andclass prototypes during the meta-test phase, no
The classifiers can quickly adapt to new environments because they do not have to be re-trainedfor the support set in the meta-test phase when new classes appear.
Moreover, the generalizationbound of ProtoNet in relation to the number of shots in a support set has been studied [2].
The boundsuggests that the performance of ProtoNet depends on the ratio of the within-class variance to thebetween-class variance of the features extracted using the meta-trained model.
There have been studies on training a new linear classifier on the features extracted using a pre-trainedmodel without meta-learning, which can perform comparably with the meta-learned models [3, 4].
We call this approach the linear-evaluation-based approach.
In these studies, the models are trained with the standard classification problem, i.e., models are trained with cross-entropy loss after linear projection from the embedding space to the class-probability space.
The linear-evaluation-basedapproach has three advantages over meta-learning.
(1) Training converges faster than meta-learning.(2) Implementation is simpler.
(3) Meta-learning decreases in performance if the number of shotsdoes not match between meta-training and meta-testing [2]; however, the linear-evaluation-basedapproach does not need to take this into account.
However, it requires retraining a linear classifierevery time a new class appears.
To avoid meta-learning during the training phase and the linear evaluation during the testing phase, wefocus on using a prototype classifier in the testing phase and training models in a standard classificationmanner.
As we discuss in section 4, we found that when we directly constructed prototypes from thefeature vectors extracted using pre-trained models and applied the nearest neighbor method as in thetesting phase in ProtoNet , this does not perform as well as the linear-evaluation-based approach.
Wehypothesize that the reason is the difference between the loss function in ProtoNet and pre-trainedmodels.
As described in section 3, if we consider a prototype as a pseudo sample average of thefeatures in each class, the loss function of ProtoNet can be considered as having a regularizing effectthat makes it closer to the sample average of a class.
Since standard classification training computescross-entropy loss with additional linear projection to make the features linearly separable, the lossfunction does not have such an effect and can cause large within-class variance.
Figure 1 shows ascatter plot of the features extracted using a neural network with two dimension output trained on
CIFAR-10 with ProtoNet(1(a)) and cross-entropy loss with a linear projection layer(1(b)).
This figureindicates that the features extracted using a model trained in a standard classification manner getdistributed away from the origin and cause large within-class variance along the direction of the classmean vectors, while those of ProtoNet are more centered to its class means.
This phenomenon is alsoobserved in face recognition literature [5–8].
We now focus on a theoretical analysis for a prototype classifier.
A recent study [2] analyzed anupper bound of risk by using a prototype classifier.
The bound depends on the number of shots ofa support set, between-class variance, and within-class variance.
However, it has two drawbacks.
The analysis requires the class-conditioned distribution of features to be Gaussian and to have thesame covariance matrix among classes.
Moreover, since the bound does not depend on the norm ofthe feature vectors, it is not clear from the bound what feature-transformation method can lead toperformance improvement.
Thus, we need to derive a novel bound for a prototype classifier.
Our contributions are threefold.
We relax the existing assumption; specifically the bound
does not require that
the features
be distributed
in Gaussian distribution
, and
each
covariance matrix
does
not have to
be thesame among classes.
We
show
that our generalization bound consists of three terms: (1) the variance of the normof feature vectors,
) the difference in the distribution shape constructed from each classembedding,
and (
) the ratio of the within class variance to the between class variance.
From our theoretical analysis and empirical investigation, we found that reducing two termsis a critical factor to improve the performance of prototype classifiers: the variance ofthe norm of feature vectors and the ratio of the within-class variance
to
the between classvariance.
While it is empirically known that the L 2 -normalization of a feature vector and minimize the ratio ofthe within-class variance to the between-class variance can improve the performance of a prototypeclassifier, our work is the first to theoretically show that these transformation can improve theperformance.
The bound of Cao et al.
[2] cannot explain the performance improvement of L 2normalization because L 2 -normalization does not reduce the ratio of the within-class variance to thebetween-class variance as shown in Section 4.
A prototype classifier can be applied to any trained feature extractor.
It is simpler than linearevaluation methods such as logistic regression and support vector machine (SVM) for three reasons.(1)
A prototype classifier does not require training in the test phase.
The training requires timeand memory consumption.
(2) Since the nearest neighbor method is applied on query data andclass prototypes, the computation in the test phase does not depend on the number of classes whilelinear-evaluation methods do.
(3) A prototype classifier requires fewer hyperparameters than linearevaluation methods.
Note that our goal is not to replace linear-evaluation methods with a prototype
[Figure]
classifier, but to show that a prototype classifier can be used as a counterpart of the linear-evaluationmethods and can be a practically useful first step in tackling few-shot learning problems.
Related Work
We summarize related work by describing a prototype classifier with meta-learning, linear-evaluationbased approach without meta-learning, a prototype classifier without meta-learning and theoreticalanalysis related to the few-shot learning problem.
Prototype classifier with meta-learning On the basis of the hypothesis that features well distinguished in the training phase are also useful for classifying new classes, constructing one or multipleprototypes for classifying unseen examples is a widely used approach [1, 9–15].
Certain algorithmscompute similarities between multiple prototypes and unseen examples by using their own modules,such as an attention mechanism [9, 14], a relation network [12], reweighting mechanisms by takinginto account between-class and within-class interaction [11], and latent clusters [13].
Another line ofresearch is transforming the space of extracted features to a better distinguishable space [16–19] ortaking the variance of features into account [20] .
Update-based meta-learning In contrast to the prototype classifier with meta-learning
, updatebased meta-learning adjusts the model parameters to adapt to new classes.
Model-agnostic metalearning (MAML) and its variants [21–23] search for good initialization parameters that adapt to newclasses with a few labeled data and a few update steps.
These approaches require additional learningof hyper-parameters and training time; thus, they prevent quick adaptation to new classes.
Linear-evaluation-based approach without meta-learning Interestingly, recent studies haveshown that training a new linear classifier with features extracted using a model trained with crossentropy loss on base-dataset performs comparably with meta-learning based methods [3, 24].
A moreeffective method for training a new classifier in few-shot settings has been proposed [25, 26], such ascalibrating distribution generated by a support set [25] , self-supervised learning [26], and distillingknowledge to obtain better embeddings [4].
However, similar to update-based meta-learning, thesestudies require additional hyper-parameters and training for application to new classes; thus if wetackle these points, we would have an alternative method that is easier and more convenient to use.
A prototype classifier without meta-learing Recent studies have shown that using a prototypeclassifier without meta-learning or a linear classifier can lead to comparable performance [24, 27].
Wang et al.
[24] has empirically shown that Centering and normalizing the feature vectors can improvethe performance of a prototype classifier.
Another study empirically shows that applying NCA loss inthe training phase can improve the performance of a prototype classifier [27].
However, their studyrequires a model to be trained with NCA loss while our studies do not require a model to be trained with any specific loss.
These studies do not show any theoretical insights on the performance of aprototype classifier and they do not compare with linear-evaluation-based methods.
Theoretical analysis of few-shot learning Even though much improvement has been empiricallymade on few-shot learning, theoretical analysis is scarce.
In the context of meta-learning, Du et al.[28] provided a risk bound on the meta-testing phase that is related to the number of meta-trainingdata and meta-testing data.
Lucas et al.
[29] derived information-theoretic lower-bounds on min-maxrates of convergence for algorithms that are trained on data from multiple sources and tested on noveldata.
Cao et al.
[2] derived a new bound on a prototype classifier and theoretically demonstrated thatthe mismatch regarding the number of shots in a support set between meta-training and meta-testingdegrades the performance of prototypical networks, which has been only experimentally observed.
However, their bound depends on two assumptions: the class-conditional distributions of features are
Gaussian and have the same covariance matrix among classes.
In contrast, we derive a novel boundthat does not depend on any specific distribution.
Theoretical Analysis of Prototype Classifier in Terms of Variance of Normof Feature Vectors
In this section, we first formulate our problem settings and explain the recent theoretical analysis ona prototype classifier.
Next we provide our novel bound for a prototype classifier with the boundrelated to the variance of the norm of the feature vectors.
Finally, we list several methods that canimprove the performance of a prototype classifier based on our bound.
3.1 Problem Setting
Let Y be a space of a class, τ a probability distribution over Y , X a space of input data, D aprobability distribution over X , D y a probability distribution over X given a class y .
We define
D ⊗ nk and D ⊗ n y by D ⊗ n y =
Π ni =1 D y and D ⊗ nk ≡ Π ni
=1 D ⊗ k i , respectively.
We sample N classesfrom τ to form the N -way classification problem.
Denote by K the number of annotated data ineach class and x ∈ X , y ∈ Y as input data and its class respectively.
We define a set of supportdata of class c sampled from τ as S c
= { x i | ( x i , y i ) ∈
X × Y ∩ y
i
=
c } Ki =1 and a set of supportdata in the N -way K -shot classification problem as S
=
(cid:83) Nc =1 S c .
Suppose a feature extractorcomputes a function ϕ : X → R D , where D is the number of the embedding dimensions.
ϕ ( S c )is defined by ϕ ( S c ) = 1 K (cid:80) x ∈ S c
ϕ
( x ) .
Let Φ be a space of the extractor function ϕ .
Denote by
M : Φ × X × ( X × Y ) NK → R N a prototype classifier function that computes the probability ofinput x belonging to class c as follows.
[Equation]
where ∥ v ∥ 2 = (cid:80) Dd =1 (cid:0) v ( d ) (cid:1) 2 , and v ( d ) is the d -th dimension of vector v .
The prediction of an inputx , denoted by ˆ y ∈ Y , is computed by taking argmax for M ( ϕ, x , S ) , i.e., ˆ y =
argmax M ( ϕ, x , S ) .
We denote by E z ∼ q ( z )
[ g ( z )] an operation to take the expectation of g ( z ) over z distributed as q ( z ) ,and we simply denote E z ∼ q ( z ) [ g ( z )] as E z [ g ( z )] when z is obviously distributed on q ( z ) .
We define
Var z ∼ q ( z ) [ g ( z )] as an operation to take the variance of g ( z ) over z distributed as q ( z ) .
With Idenoting the indicator function, we define the expected risk R M of a prototype classifier as
[Equation]
We show a case of multi-class classification in Appendix A.5 due to lack of space. We obtain thebound on multi-class case by Frechet’s inequality.
Let c 1 and c 2 denote any pair of classes sampled from τ .
We consider that a query data point xbelongs to class c 1 and support sets S consist of class c 1 ’s support set and c 2 ’s support set.
Then,equation 2 is written as follows.
[Equation]
where E S,c 1 , x = E S ∼D ⊗ 2 k E c 1 ∼ τ E x ∼D c 1 .
3.2 What Feature-Transformation Method Is Expected to Be Effective?
The current theoretical analysis for a prototype classifier [2] has the following two drawbacks (see
Appendix A.2 for the details).
The first is that the modeling assumption requires a class-conditioneddistribution of the features to follow a Gaussian distribution with the same covariance matrix amongclasses.
For example, when we use the ReLU activation function in the last layer, it is not normallydistributed and the class-conditioned distribution does not have the same covariance matrix as shownin Figure 1.
The second drawback is that it is not clear what feature-transformation method can reducethe upper bound.
A feature-transformation method that maximizes the between-class variance andminimizes the within-class variance such as linear discriminant analysis (LDA) [30] and Embedding
Space Transformation (EST) [2] can be expected to improve performance; however, it is not clearhow the second term of the denominator changes.
From Figure 1, the distribution of each class feature stretches in the direction of its class-mean featurevector.
This property is also observed in metric learning literatures [5, 6].
A model trained in thecross-entropy loss after linear projection from the embedding space to the class-probability spacecomputes a probability of input x belonging to class c with a weight matrix W ∈ R D × K given by
[Equation]
Comparing equation 4 with equation 1, we found that equation 4 does not have a regularization termsimilar to the one appearing in equation 1 that forces the features to be close to its class-mean featurevector.
This implies that the features extracted using a model trained with the cross-entropy lossusing equation 4 are less close to its class-mean feature vector than the ProtoNet loss on equation 1.
Through this observation and the property mentioned above, we hypothesize that normalizing thenorm of the feature vectors can push the features to each class-mean feature vector and can boost theperformance of a prototype classifier trained with cross-entropy loss on equation 4.
3.3 Relating Variance of Norm to Upper Bound of Expected Risk
To understand what effect the variance of the norm of the feature vectors has on the performance of aprototype classifier, we analyze how variance contributes to the expected risk when an embeddingfunction ϕ is fixed.
The following theorem provides a generalization bound for the expected risk ofa prototype classifier in terms of the variance of the norm of the feature vectors computed using afeature extractor.
Theorem 1.
Let M be an operation of a prototype classifier for binary classification defined byequation 1.
For µ c =
E x ∼D
[Equation] [Equation]
] , Σ
[Equation]
=
[Equation]
∼D c [( ϕ ( x
[Equation]
− µ
[Equation]
)( ϕ ( x ) − µ c ) ⊤ ] , µ = E c ∼ τ [ µ c ] , Σ = 195 E c ∼ τ [( µ c − µ )( µ c − µ ) ⊤ ] , and Σ τ =
E c ∼ τ [Σ c ] , if ϕ ( x ) has the variance of its norm, then themisclassification risk of the prototype classifier for binary classification R M satisfies
[Equation]
5) where E V[ h L
[Equation]
[Equation] [Equation]
Remark.
The term E V[ h L 2 ( ϕ ( x ))] is the variance of the norm of the feature vectors.
The term
V Tr (Σ c 1 ) is the variance of the summation with the diagonal element of the covariance matrix fromeach class embedding; it can be interpreted as the difference in the distribution shape constructed from each class embedding.
The term V wit (Σ τ , Σ , µ ) is related to the within-class variance.
The lastterm of equation 8 is approximately linear with Tr ( Σ c 1 ) Tr(Σ) because ( µ c 2 − µ c 1 ) ⊤ ( µ c 2 − µ c 1 ) Tr(Σ) is supposedto be constant.
Thus V wit (Σ τ , Σ , µ ) Tr(Σ) is a secondary expression for the ratio of the within-class varianceto the between-class variance.
The term E dist 2L2 ( µ c 1 , µ c 2 ) is the expectation of the Euclideandistance between the class-mean vectors .
This bound has the following properties.1.
Its derivation does not require the features to be distributed in Gaussian distribution and theclass-conditioned covariance matrix does not have to be the same among classes2.
The bound can decrease when any of four statistics decreases with fixed between-classvariance ( Σ ): (i) the variance of the norm of the feature vectors, as discussed in Section3.2, (ii) the difference in the distribution shape constructed from each class embedding, (iii)the ratio of the within-class variance ( Σ τ ) to the between-class variance ( Σ ), and (iv)
the
Euclidean distance between the class-mean vectors.
As a result, our bound loosens the modeling assumption of Theorem2 in Appendix A.2 and has itsproperty.
We show the proof in Appendix A.4.
3.4 Feature-Transformation Methods
We hypothesize from Theorem 1 that in addition to a feature-transformation method related toequation 6, lowering the ratio of between-class variance to within-class variance can improve theperformance of a prototype classifier.
As shown in Figure 2 left in experimantal analysis, the valueof equation 7 is relatively small compared to equation 6 and equation 8.
Regarding equation 9,the ratio of the Euclidean distance between the class mean vectors and the between-class varianceis supposed to be constant.
Thus, we focus on transformation methods related to equation 6 andequation 8.
We analyze four feature-transformation methods: L 2 -normalization
( L 2 -norm), EST,LDA,
EST+
L 2 -norm and LDA+ L 2 -norm.
We give the details on each method in Section A.1 due to lack ofspace.
Experimental Evaluation
In this section, we experimentally analyzed the effectiveness of the feature-transformation methodsmentioned in Section3.4 under two scenarios: (1) standard object recognition and (2) cross-domainadaptation.
The center loss [5] and affinity loss [31] have been proposed to efficiently pull thefeatures of the same class to their centers in the training phase; however,our goal is not to achieve
SOTA but, through comparison with existing studies, show that we can achieve the same levelof performance without fine-tuning and thus we focus on a widely used pre-trained model in theexperiments following the line of studies Chen et al.
[3] and Tian et al.
4.1 Datasets and Evaluation Protocol
For standard object recognition, we use the mini ImageNet dataset, tiered ImageNet dataset, the
CIFARFS dataset, and the FC100 dataset.
miniImageNet.
The mini ImageNet dataset [9] is astandard bench-mark for few-shot learning algorithms in recent studies.
It contains 100 classesrandomly sampled from ImageNet.
Each class contains 600 images.
We follow a widely usedsplitting protocol [32] to split the dataset into 64 / 16 / 20 for training/validation/testing respectively.tieredImageNet
The tiered ImageNet dataset [33] is another subset of ImageNet but has 608 classes.
These classes are grouped into 34 higher level categories in accordance with the ImageNet hierarchyand these categories are split into 20 training ( 351 classes), 6 validation ( 97 classes), 8 testingcategories ( 160 classes).
This splitting protocol ensures that the training set is distinctive enoughfrom the testing set and makes the problem more realistic than miniImageNet since we generallycannot assume that test classes will be similar to those seen in training.
CIFAR-FS
The CIFAR-FS dataset [34] is a recently proposed fewshot image classification benchmark, consisting of all 100 classes from CIFAR- 100 [35].
The classes are randomly split
[Table]
[Figure]
[Table]
FC
The FC100 dataset [36] is another dataset derived from CIFAR- 100 [35], containing 100classes which are grouped into 20 categories.
These categories are split into 12 categories for training( 60 classes), from 4 categories for validation ( 20 classes), 4 categories for testing ( 20 classes).
Thissplitting protocol is similar to the protocol used in tieredImageNet so that the training set is distinctiveenough from the testing set and makes the problem more realistic than CIFAR-FS.
For cross-domain scenario, we use the mini ImageNet dataset during pre-training stage and we use the
CUB-200-2011 dataset [37], a.k.a CUB during testing stage.
CUB
The CUB dataset contains 200 classes and 11,788 images in total.
Following the evaluation protocol of Chen et al.
[3], we split the dataset into 64 / 16 / 20 for training/validation/testingrespectively.
4.2 Implementation Details
We compared the feature-transformation methods against ProtoNet [1], Baseline, and Baseline++ [3].
For Baseline and Baseline++, we trained the linear projection layer on a support set.
The differencebetween Baseline and Baseline++ is that the norm of the linear projection layer and features in
Baseline++ are normalized to be constant.
We call Baseline and Baseline++ as “linear evaluation”methods.
We also compared with the feature-transformation method proposed in a previous study[24].
In that study they transformed the features so that the mean of all features was the origin before
L 2 -normalization and then used a prototype classifier without training a new linear classifier.
Wecall this operation centering+ L 2 -norm.
We re-implemented these methods following the training procedure in a previous study [3].
In the pre-training stage, where the cross-entropy loss was usedand meta-learning was not used, we trained 400 epochs with a batch size of 16 .
In the meta-trainingstage for ProtoNet, we trained 60 , 000 episodes for 1 -shot and 40 , 000 episodes for 5 -shot tasks.
Weused the validation set to select the training episodes with the best accuracy.
We evaluated the modelevery 5 epochs, not every epoch as experimented in Chen et al.
[3] , and we performed early stoppingwhen the validation score does not improve for 50 epochs.
In each episode, we sampled N classesto form N -way classification (in meta-training N =20 and meta-testing N =5 following the originalstudy for ProtoNet [1]).
For each class, we selected K labeled instances as our support set and 16instances for the query set for a K -shot task.
In the linear evaluation or meta-testing stage for all methods, we averaged the results over 600 trials.
In each trial, we randomly sampled 5 classes from novel classes, and in each class, we also selected
K instances for the support set and 16 for the query set.
For Baseline and Baseline++, we used theentire support set to train a new classifier for 100 iterations with a batch size of 4 .
With ProtoNet, weused the models trained in the same shots as meta-testing stage since a mismatch in the number ofshots between meta-training and meta-testing degrades performance [2].
All methods were trainedfrom scratch, and the Adam optimizer with an initial learning rate of 10 − 3 was used.
We appliedstandard data augmentation including random crop, horizontal-flip, and color jitter in both the trainingstages.
We used a ResNet12 network and ResNet18 network following the previous study [3, 17].
For LDA we set λ = 0 . 0001 for equation 11 in Appendix A.1 and for EST we set the dimensions ofthe projected space to 60 following the settings of the original study [2].
Following the procedure of
EST, we calculate the transformation matrix of LDA based on the features of the training dataset.
3 Results
We present the experimental results of standard object recognition in Table 1 on the basis of backboneswith ResNet12 for a comprehensive comparison.
We show the result of backbones with ResNet18 in
Table 3.
We present the discussion on cross-dataset scenario and the result of the scenario in A.7.
Comparison of feature-transformation methods with ProtoNet, linear evaluation methods, andcentering+ L 2 -norm From Table 1, we can observe that the prototype classifier with L 2 -norm,
EST+
L 2 -norm, LDA+ L 2 -norm performs comparably with ProtoNet and the linear-evaluation-basedapproach in all settings.
Comparing the feature-transformation methods described in Section 3.4with centering+ L 2 -norm, centering+ L 2 -norm can slightly improve the performance of the prototypeclassifier in several 1 -shot settings .
However, in 5 -shot settings, the boost decreases and evenperforms worse than linear-evaluation methods, e.g. mini ImageNet and tiered ImageNet.
Comparison among feature-transformation methods In the 1 -shot setting, although EST fallsshort of ProtoNet and the linear-evaluation-based approach, it also improves the performance of aprototype classifier.
The performance gain of both L 2 -norm, EST, LDA, EST+
L 2 -norm and LDA
+
L 2 -norm decrease when the number of shots increases.
This phenomenon can be explained through
Theorem 1.
The term relating to the variance of the norm and the ratio of the within-class variance tothe between-class variance depends on K .
Since the term diminishes as K increases, the performancegain of the feature-transformation methods decreases.
Comparison of feature-transformation methods with current studies Table 2 shows the performance of MetaOptNet [36], TapNet [17] and feature-transformation methods that performs comparable with ProtoNet or linear-evaluation-based methods in Table 1.
We can observe that a prototypeclassifier can achieve comparable performance with current studies in most of the settings with thefeature transformations.
Moreover, we can further boost the performance of a prototype classifierby combining L 2 -norm and the methods to reduce the ratio of the within-class variance to thebetween-class variance, e.g. CIFAR-FS and FC100.
Discussion on feature-transformation methods LDA and EST in the 5 -shot setting do not improvethe performance so much compared with L 2 -norm variants while in the 1 -shot settings, LDA and ESTimprove the performance.
This is because, in the 5 -shot settings, the values computed from equation 8get smaller with larger K , and the effect of equation 8 on the risk of a prototype classifier in the1 -shot settings is larger.
Especially, EST and LDA outperform L 2 -norm in FC100.
We found from
Figure 2 that the features of FC100 show the largest ratio of the within-class to between-class variance among all datasets.
Thus the method of reducing the ratio works better with FC100 features than withany other dataset’s features.
As discussed in Section 4.3, the combination of feature-transformationmethods that reduce the variance of the norm and the ratio of the within-class variance to the betweenclass variance can further boost the performance of a prototype classifier.
This indicates that reducingthe term equation 6 and equation 8 is an important factor for the performance of a prototype classifier.
4.4 The comparison of the ratio of the within-class variance to the between-class variancebefore and after applying L 2 norm, EST and LDA
We show the ratio of the within-class variance to the between-class variance before and after applying
L 2 -norm, EST, LDA, EST + L 2 -norm, LDA + L 2 -norm in Figure 2 right.
We calculated the ratioof the between-class variance to each class’s variance of each dataset.
This figure shows that L 2normalization slightly changes the ratio while EST and LDA can reduce the ratio.
The combinationof EST and L 2 -norm or LDA and L 2 -norm can reduce the ratio while decreasing the variance ofthe norm of feature vectors to 0.
Therefore, EST + L 2 -norm and LDA + L 2 -norm performs betterthan L 2 -norm.
Moreover, L 2 -norm does not affect the bound of Cao et al.
[2] because their bounddepends on the ratio of the between-class variance to each class’s variance as shown in 2.
Thus thebound of Cao et al.
[2] cannot explain the performance improvement of a prototype classifier with
L 2 -norm while our bound can.
Conclusion
We theoretically and experimentally analyzed how the variance of the norm of feature vectors affectsthe performance of a prototype classifier.
We derived a generalization bound that does not requirethe features to be distributed in Gaussian distribution and the class-conditioned covariance matrixdoes not have to be the same among classes.
We found that using EST+ L 2 -norm makes the classifiercomparable with ProtoNet and the linear-evaluation-based approach.
Our experiments show that whenthe number of shots in a support set increases, the performance gain from a feature-transformationmethod decreases, which is consistent with the results of theoretical analysis.
One limitation of this paper is that, in the first place, the prototype classifier works well under theassumption that the data distribution for each class is unimodal and isotropic modeling.
However,we consider that this assumption may be sound in practice because the multimodality of the datadistribution in a class is typically caused by the way the data is annotated.
Since the number of asupport set is small in few-shot settings (for example, in 5 -way 5 -shot, the number of labeled data is25 ), we can easily check the labels and re-label them so that they don’t become multi-modal.
Broader impact.
We believe that our work shows that a prototypical classifier is expected tobe a practically useful first step in tackling few-shot learning problems because of its simplicity.
The progress in few-shot learning can impact importnant problems such as medical images andre-identification problems.
We also recognize our work might constitute a threat that authoritarianentities deploy few-shot learning algorithms for surveillance.
Checklist
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See contribution part in Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seee
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theorem
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We only show
the implementation details.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.2(c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section A.
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[No] We omit these information because
total amount of compute is not so large and not our intrests.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[No] We can confirm the license in thecited paper.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Section 4.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We used widely used benchmark dataset.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Detail of Feature-Transforming Methods
L 2 normalization ( L 2 -norm) From equation 6, normalizing the norm of the feature vectors canimprove the performance of a prototype classifier.
We denote a function normalizing the norm asψ L 2 given by
[Equation]
LDA From equation 8, decreasing the ratio of the within-class variance to the between-classvariance can improve the performance of a prototype classifier.
LDA [30] is widely used to search fora projection space that maximizes the between-class variance and minimizes the within-class variance.
It computes the eigenvectors of a matrix ˆΣ τ − 1 ˆΣ , where ˆΣ is the covariance matrix of prototypes andˆΣ τ is the class-conditioned covariance matrix.
Since the number of data is small in few-shot settings,ˆΣ τ − 1 cannot be estimated stably and we add a regularizer term to ˆΣ τ and define it as ˆΣ τ reg .
[Equation]
11) where λ ∈ R 1 , I ∈
R D
× D is identity matrix.
EST Since computation of LDA is unstable, we also analyze the effect of EST [2].
EST computeseigenvectors of a matrix ˆΣ − ρ ˆΣ τ : the difference between the covariance matrix of the class meanvectors and the class mean covariance matrix with weight parameter ρ .
Similar to LDA, EST alsosearches for the projection space that maximizes the Σ and minimizes the Σ τ .
EST+
L 2 -norm We hypothesize that the combination of the transforming methods can improvethe performance of a prototype classifier independently from each other.
Specifically, we focus onreducing equation 6 and equation 8 by combining EST and L 2 -norm .
We first apply EST to reduceequation 8 and after that we apply L 2 -norm to reduce equation 6.
At the end of the operation wewant the variance of the norm to be 0 , thus we apply EST and after that we apply L 2 -norm.
LDA
[Equation]
L 2 -norm We focus on reducing equation 8 and equation 7 by combining LDA and L 2 -norm .
Following the similar procedure of EST+ L 2 -norm , we first apply LDA and after that we apply
L 2 -norm.
A.2 Existing Upper Bound on Expected Risk for Prototype Classifier
To analyze the behavior of a prototype classifier, we start from the current study [2].
The followingtheorem is the upper bound of the expected risk of prototypical networks with the next conditions.
• The probability distribution of an extracted feature ϕ ( x ) given its class y =
c is Gaussian i.eD y = N ( µ c , Σ c ) , where µ c =
E x ∼D c [ ϕ ( x )] and Σ c = E x ∼D c
[( ϕ ( x ) − µ c )( ϕ ( x ) − µ c ) ⊤ ] .•
All class-conditioned distributions have the same covariance matrix, i.e., ∀ ( c, c ′ ) , Σ c = Σ c ′ .
Theorem 2 (Cao et al.
Let M be an operation of a prototype classifier on binary classification defined by equation 1.
Then for µ =
E c ∼ τ [ µ c ] and Σ = E c ∼ τ [( µ c − µ )( µ c − µ ) ⊤ ] , themisclassification risk of the prototype classifier on binary classification R M satisfies
[Equation]
(12) where E dist 2L2 ( µ c 1 , µ c 2 ) =
E c 1 ,c 2 (cid:104) (( µ c 1 − µ c 2 ) ⊤ ( µ c 1 − µ c 2 ))
2 (cid:105) .
We show the detail of the derivation in Appendix A.3.
A.3 Reviewing Derivation Details of Theorem 2 (Cao et al. [2])
We briefly review the derivation of Theorem 2.
In prototype classifier, from equation 1 and equation 3,
R M is written with sigmoid function σ as follows:
[Equation]
where α ≜ ∥ ϕ ( x ) − ϕ
( S c 2 ) ∥ − ∥
ϕ
( x ) − ϕ ( S c 1 ) ∥ .
we bound equation 13 with expectation andvariance of α by following proposition.
Proposition 1.
From the one-sided Chebyshev’s inequality
[Equation]
it immediately follows that:
[Equation]
equation 13 can be further write down as follows by Law of Total Expectation
[Equation]
Therefore,
[Equation]
We write down the expection and variance of α with following Lemmas 3 and 4.
Lemma 3.
Under the same notation and assumptions as Theorem 2, then,
[Equation]
Lemma 4.
Under the same notation and assumptions as Theorem 2, then,
[Equation]
The proofs of the above lemmas are in the current study [2].
With Proposition 1 and Lemma 3, Lemma 4, we obtain Theorem 2.
A.4 Derivation Details of Theorem
We will describe the detailed derivation of Theorem 1 in this section.
Our derivation is different from
Cao et al.
[2]’s study in the following points.1.
We re-derived Lemma 3 because the term of the difference between the trace of the classcovariance matrices is erased in the lemma.
This term cannot omit in our derivation sincewe do not assume the class covariance matrix to be the same among classes.2.
We re-derived the bound on the variance of squared Euclidean distance of two vectors, e.g
Lemma 4.
The derivation of Cao et al.
[2] uses the property of quadratic forms of normallydistributed random variables and the fact that the sum of normally distributed random variables is also distributed in Gaussian distribution.
The calculation of the variance of squared
L 2 -norm without depending on the property of some distributions is not straightforward[38].
We divide the variance of squared Euclidean distance of two vectors into the varianceof the norm of the feature vectors and the variance of the inner-product of vectors.
Then weapply Cauchy–Schwarz inequality to the inner-product.
We start the proof from equation 15.
We first prove the following Lemma 5 related to the expectationstatistics of α in equation 15.
Lemma 5.
Under the same notations and assumptions as Theorem 1, then,
[Equation]
Proof.
First, from the definition of α , we split E S ∼D ⊗ 2 k E x ∼D c 1 [ α ] in to two parts and examine themseperately.
[Equation]
In regular conditions, for random vector X , the expectation of the norm is
[Equation]
18) and the variance of the vector is
[Equation] [Equation]
Hence,
[Equation]
where the first term inside the trace can be expanded as:
[Equation]
The second term in equation 21 is simply as follows.
[Equation]
[Equation]
From equation 22 and equation 23 we obtain
[Equation]
[Equation]
Similarly for ii,
[Equation]
From ( i ) and ( ii ) , and equation
[Equation]
Since
[Equation]
Next we prove the following Lemma 6 related to the conditioned variance of α
[Equation]
Lemma 6.
Under the same notation and assumptions as Theorem 1,
[Equation] [Equation]
where
[Equation]
Proof.
We start with the inequality between the variance of 2 random variables.
We define Cov( A, B )as covariance of 2 random variables A, B .
[Equation]
For Var x ∼ D c 1 ,S [ α ] ,
Var x ∼ D c 1 ,S [ α ] =
Var
[Equation]
) (cid:17)(cid:105) ( ∵ equation 30) = 2Var (cid:20)(cid:13)(cid:13)(cid:13) ϕ
( S c 1 ) (cid:13)(cid:13)(cid:13) 2 (cid:21) + 2Var (cid:20)(cid:13)(cid:13)(cid:13) ϕ
( S c 2 ) (cid:13)(cid:13)(cid:13) 2 (cid:21) + 4Var (cid:104) ϕ
( x ) ⊤ ( ϕ ( S c 2 ) − ϕ
( S c 1 )) (cid:105) .
From 3 rd line to 4 th line we decompose the variance of (cid:13)(cid:13)(cid:13) ϕ
( S c 1 ) (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) ϕ
( S c 2 ) (cid:13)(cid:13)(cid:13) 2 use the independenceof
( S
1 ) and
( S c 2 ) with
Next we focus on Var (cid:104) ϕ
[Equation]
⊤
[Equation]
ϕ
[Equation]
2 ) − ϕ
[Equation] [Equation]
we
For E (cid:20)(cid:13)(cid:13)(cid:13) ϕ
[Equation]
S c 2 ) − ϕ
[Equation]
S c 1 ) (cid:13)(cid:13)(cid:13) 2 (cid:21) , with equation
E
[Equation]
ϕ
[Equation]
− ϕ
[Equation]
Thus E c
[Equation]
c 2 ∼ τ
[Equation]
is calculated as follows.
[Equation]
+ E c 1 ,c 2 ∼ τ (cid:104)
Tr (Σ c 1 ) (cid:0) µ c 2 − µ c 1 (cid:1) ⊤ (cid:0) µ c 2 − µ c
(cid:1)(cid:105) =
K (cid:16) E c 1 ,c 2 ∼ τ (cid:2) Tr(Σ c 1 ) 2 (cid:3) + E c 1 ,c
2 ∼ τ [Tr(Σ c 1 )] 2 (cid:17) +
K (Tr (Σ τ )) (cid:0)
Tr (Σ) + µ ⊤ µ (cid:1) + E (cid:2) µ ⊤ c 1 µ c 1 ( µ c 2 − µ c 1 ) ⊤ ( µ c 2 − µ c 1 ) (cid:3) + E c 1 ,c
2 ∼ τ (cid:104)
Tr (Σ c 1 ) (cid:0) µ c 2 − µ c 1 (cid:1) ⊤ (cid:0) µ c 2 − µ c
(cid:1)(cid:105) =
K Var c ∼ τ [Tr (Σ c )]
+
K Tr (Σ τ )
+ 2 K (Tr (
[Equation]
)) (cid:0) Tr (Σ) + µ ⊤ µ (cid:1)
[Equation]
E
[Equation] [Equation]
(cid:104)
µ ⊤ c 1 µ
[Equation]
1 (cid:0) µ c 2 − µ c 1 (cid:1) ⊤ (cid:0) µ c 2 − µ c
(cid:1)(cid:105)
+ E c 1 ,c 2 ∼ τ (cid:104)
Tr (Σ c 1 ) (cid:0) µ c 2 − µ c 1 (cid:1) ⊤ (cid:0) µ c 2 − µ c
(cid:1)(cid:105) =
K Var c ∼ τ [Tr(Σ c )]
+
K Tr(Σ τ ) (cid:0) Tr(Σ τ ) + Tr(Σ) + µ ⊤ µ (cid:1) + E c 1 ,c 2 (cid:104) µ ⊤ c 1 µ c 1 (cid:0) µ c 2 − µ
[Equation]
Now we take into account the term − ( µ ⊤ c 1 ( µ c 2 − µ c 1 )) 2 in equation 32,
[Equation]
Thus E c 1 ,c 2 (cid:104) Var[ ϕ ( x ) ⊤ ( ϕ ( S c 2 )
− ϕ ( S c 1 ))]
(cid:105) is calculated as follows.
[Equation]
Regarding (cid:13)(cid:13)(cid:13) ϕ
( S c ) (cid:13)(cid:13)(cid:13) 2 , since the function computing square norm is convex, next equation holds with
D -dimensional Jensen’s inequlality [39]:
[Equation]
Combining equation 31, equation 36, and equation 37 we obtain
[Equation]
[Equation]
To complete the proof of Theorem 1, we calculate E c 1 ,c
2 E x ∼D c 1 ,S ∼D ⊗ 2 K
[ α ] 2 .
E
[Equation]
c 2 E x ∼D c 1 ,S ∼D ⊗ 2 K [ α ]= E c 1 ,c(cid:34)(cid:18)
[Equation]
−
[Equation] [Equation]
= E c 1 ,c
[Equation]
=
[Equation]
From 2 nd line to 3 rd line, we use the symmetry of the last term with respect to c 1 and c 2 and erasethe term.
Combining equation 15, Lemma 5, Lemma 6, and equation 39, we obtain the bound.
A.5 Theorem 1 with N -way Classification
The upper bound on the risk of N -way prototype classifier is as follows.
Theorem 7.
Let operation of binary class prototype classifier M as defined in equation 1.
Thenfor
[Equation] [Equation]
[Equation]
] , if ϕ ( x ) has its fourth moment, missclassification risk of binary class prototype classifier R M satisfy
[Equation]
where
[Equation]
Proof.
Let x, y be the input and its class of a query data.
We define α c by α c = (cid:13)(cid:13)(cid:13) ϕ
( x ) − ϕ ( S c ) (cid:13)(cid:13)(cid:13) 2 −(cid:13)(cid:13)(cid:13) ϕ
( x ) − ϕ ( S y )
(cid:13)(cid:13)(cid:13) 2 .
Then a prototype classifier miss-classify a class of input x , ˆ y , if ∃ c ∈ [1 , N ] ,
c ̸
=y, α c < 0 .
Hence: R M ( ϕ ) =
Pr( (cid:83)
Nc =1 c ̸ =
y α c < 0)
By Frechet’s inequality, next equation holds:
[Equation]
Noting that Theorem 1 can be applied to each term in the summation and then we obtain Theorem
A.6 Detailed performance results
We show in Table 3 the detailed performance results of standard object recognition in this sectionwith 95% confidence margin.
The table shows the similar result with Table 1.
We can observe thatthe prototype classifier with L 2 -norm, EST+
L 2 -norm, LDA+ L 2 -norm performs comparably with
ProtoNet and the linear-evaluation-based approach in most of the settings.
T a b l e 3 :
C l a s s i fi ca ti on acc
u r ac i e
s w it h
R e s
N e t on m i n i
I m a g e N e t , ti e
r e d I m a g e
N e t ,
C
I F
A R
FS , a nd
F C 10o f P r o t o
N e t , li n ea r-
e v a l u a ti on - b
a s e d m e t hod s [ 3 ] , ce n t e r
i ng w it h
L 2no r m [ 24 ] , a nd ou r s .
T h e
B a s e li n e w it hou t li n ea r-
e v a l u a ti on m e t hod s w it
h
acc u r ac y g
r ea t e r t h a n t
h e l o w e r 95 % c on fi d e n ce m a r g i n o f t h e acc u r ac y
o f P r o t o
N e t a n d
B a s e li n e a r e i n bo l d .
R e g a r d i n g t
o
B a s e li n e + + , B a s e li n e + + w it hou t li n ea r-
e v a l u a ti on m e t hod s w it
h
acc u r ac y g
r ea t e r t h a n t
h e l o w e r 95 % c on fi d e n ce m a r g i n o f t h e acc u r ac y
o f P r o t o
N e t a nd
B a s e l i n e ++ a r e i n bo l d .
m i n i
I m a g e
N e t ti e r e d
I m a g e N e t
C
I F
A
R - FS
F C 1 - s h
o t 5 - s h o t 1 - s h o t 5 - s h o t 1 - s h
o t 5 - s h o t 1 - s h
o t 5 - s h o t m e t ho
d s w it h m e t a - l e a r n i ng o r li n e a r - ev a l ua ti on
P r o t o
N e t [ 1 ] 56 .±
.75 .±
.61 .±
.81 .±
.65 .±
.79 .±
.35 .±
.50 .± 0 .
B a s e li n e [ 3 ] 55 .±
.76 .±
.63 .±
.83 .±
.65 .±
.79 .±
.40 .±
[Table] [Table]
[Table] [Table]
A p r o t o t y p
e c l a ss i fi
e r w it h f e a t u r e - t r an s f o r m a ti on m e t hod s
B a s e li n e - w / o - li n ea r , ce n t e r
i
ng
+ L-
no r m [ 24 ] 57 .±
.75 .±
.65 .±
.81 .±
.65 .±
.78 .±
.41 .±
.56 .± 0 .
B a s e li n e - w / o - li n
[Table]
r
[Table]
B a s e li n e - w / o - li n ea r ,
L 2 - no
[Table]
m
[Table]
B a s e li n e - w / o - li n
[Table]
r ,
E S T
[Table]
B a s e li n e - w / o - li n ea r ,
E S T +
L 2 -
[Table]
B a s e li n e - w / o - li n
[Table]
r ,
L DA
[Table]
B a s e li n e - w / o - li n ea r ,
L DA + L 2no
[Table]
m
[Table] [Table]
m e t ho d s w it h m e t a - l e a r n i ng o r li n e a r - ev a l ua ti on
B a s e li n e ++ [ 3 ]
[Table]
A p r o t o t y p
e c l a ss i fi
e r w it h f e a t u r e - t r an s f o r m a ti on m e t hod s
B a s e li n e ++ - w / oli n ea r , ce n t e r
i
ng
+ L-
no r m [
[Table] [Table]
[Table] [Table]
[Table]
B a s e li n e ++ - w / oli n ea r
[Table]
B a s e li n e ++ - w / oli n ea r , L
[Table]
A.7 Performance results of cross dataset
We show in Table 4 the performance results of cross-dataset scenario in this section with 95%confidence margin.
From the table, we can observe that the prototype classifier with L 2 -norm,
EST+
L 2 -norm and LDA+
L 2 -norm performs comparably with linear-evaluation-based methods.
Therefore, our analysis results still hold on the cross-domain scenario.
We also found that ProtoNet performs worse than linear-evaluation-based methods and a prototypeclassifier with feature-transformation methods.
In this scenario, we should focus on improving theperformance of a prototype classifier with features extracted from a model trained without metalearning, e.g, Baseline and Baseline++.
Since there is no guarantee that the features of a modeltrained with cross-entropy loss will be distributed in Gaussian distribution, Cao et al.
[2]’s boundcannot explain the performance improvement of the prototype classifier while our analysis can.
[Table]
A.8 Visualization of the feature distribution
We show in Figure 3 the distribution of features on test-set of FC100 before and after applying thedata transformation.
From figure 3, we can observe that the feature-transformation-methods slightlychange the distributions even the transformations improve the performance of a prototype classifier.
The projection into a lower dimensional space for visualization does not accurately represent therelationship of a higher dimension.
Figure