A
C LOSER
L OOK
AT
P ROTOTYPE
C LASSIFIER
FOR
F EW - SHOT I MAGE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The prototypical network is a prototype classifier based on meta-learning and is widely used for few-shot learning because it classifies unseen examples by constructing class-specific prototypes without adjusting hyper-parameters during meta-testing.
Interestingly, recent research has attracted a lot of attention, showing that a linear classifier with fine-tuning, which does not use a meta-learning algorithm, performs comparably with the prototypical network.
However, fine-tuning requires additional hyper-parameters when adapting a model to a new environment.
In addition, although the purpose of few-shot learning is to enable the model to quickly adapt to a new environment, fine-tuning needs to be applied every time a new class appears, making fast adaptation difficult.
In this paper, we analyze how a prototype classifier works equally well without fine-tuning and meta-learning.
We experimentally find that directly using the feature vector extracted using standard pre-trained models to construct a prototype classifier in meta-testing does not perform as well as the prototypical network and linear classifiers with fine-tuning and feature vectors of pre-trained models.
Thus, we derive a novel generalization bound for the prototypical network and show that focusing on the variance of the norm of a feature vector can improve performance.
We experimentally investigate several normalization methods for minimizing the variance of the norm and find that the same performance can be obtained by using the L2 normalization and embedding space transformation without fine-tuning or meta-learning.
I NTRODUCTION
Few-shot learning is used to adapt quickly to new classes with low annotation cost.
Meta-learning is a standard training procedure to tackle the few-shot learning problem and Prototypical Network (Snell et al., 2017)
a.k.a ProtoNet is a widely used meta-learning algorithm for few-shot learning.
In ProtoNet, we use a prototype classifier based on meta-learning to predict the classes of unobserved objects by constructing class-specific prototypes without adjusting the hyper-parameters dur- ing meta -testing.
ProtoNet has the following advantages.
(1) Since the nearest neighbor method is applied on query data and class prototypes during the meta-test phase, no hyper-parameters are required in meta-test phase.
(2) Since the number of data for few-shot is small, the inference time is almost negligible.
The classifiers can quickly adapt to new environments because they do not have to be re-trained for the support set when new classes appear.
The generalization bound of ProtoNet in relation to the number of shots in a support set has been studied (Cao et al., 2020).
The bound suggests that the performance of ProtoNet depends on the ratio of the between-class variance to the within-class variance of features of a support set extracted using the meta-trained model.
There have been studies on fine-tuning a linear classifier on the features extracted using a pretrained model without meta-learning, which can perform comparably with the meta-learned models (Chen et al., 2019; Tian et al., 2020).
In these studies, the models are trained with the standard classification problem, i.e., models are trained with cross-entropy loss after linear projection from the embedding space to the class-probability space.
The linear classifier with fine-tuning has the following advantages over meta-learning.
(1) Training converges faster than meta-learning.
Implementation is simpler.
(3) Meta-learning decreases in performance if the number of shots does
[Figure]
not match between meta-training and meta-testing (Cao et al., 2020); however, the linear classifier with fine-tuning do not need to take this into account.
However, the fine-tuning-based approach has two drawbacks, i.e., we have to tune additional hyper-parameters for fine-tuning and to apply fine-tuning every time a new class appears, which make fast adaptation difficult in few-shot learning.
To overcome the disadvantages while inherits the advantages of a linear classifier with fine tuning, we analyze how to avoid meta-learning during the training phase and fine-tuning during the testing phase.
Since a prototype classifier can be applied to any trained feature extractor and does not require any hyperparameter or fine-tuning in the testing phase, we focus on using a prototype classifier on the testing phase and training models in a standard classification manner.
As we discuss in section 4, we found that when we directly constructed prototypes from the feature vectors extracted using pretrained models and applied the nearest neighbor method as in the testing phase in ProtoNet , this does not perform as well as the linear classifier with fine-tuning.
We hypothesize that the reason is the difference between the loss function in ProtoNet and pretrained models.
As described in section 3, if we consider a prototype as a pseudo sample average of the features in each class, the loss function of ProtoNet can be considered having a regularizing effect that makes it closer to the sample average of the class.
Since standard classification training computes cross-entropy loss with additional linear projection to make the features linearly separable, the loss function does not have such an effect and can cause large within-class variance.
Figure 1 shows a scatter plot of the features extracted using a neural network with two dimension output trained on cifar-10 with ProtoNet(1a) and cross-entropy loss with a linear projection layer(1b).
This figure implies that the features extracted using a model trained in a standard classification manner distributes away from the origin and causes large within-class variance along the direction of the norm of class mean vectors, while that of ProtoNet is more centered to its class means.
This phenomenon is also observed in face recognition literatures (Wen et al., 2016; Liu et al., 2017;
Wang et al., 2018; Deng et al., 2019).
We now focus on the theoretical analysis on a prototype classifier.
A current study (Cao et al., 2020) analyzed an upper bound of risk by using a prototype classifier.
The bound depends on the number of shots of a support set, between-class variance, and within-class variance.
However, the bound requires the class-conditioned distribution of features to be Gaussian and to have the same covariance matrix among classes.
In addition, since the bound does not depend on the norm of the feature vectors, it is not clear from the bound what feature-transformation method can lead to performance improvement.
Thus, we derive a novel bound for a prototype classifier.
Our contributions are threefold.
We relax the assumption; specifically, the bound does not require that the features distribute on any specific distribution, and each covariance matrix does not have to be the same among classes.
We clarify the effect of the variance of the norm of the feature vectors on the performance of a prototype classifier.
We investigate the effectiveness of reducing the variance of the norm empirically.
R ELATED W ORK
We summarize related work by describing a prototype classifier with meta-learning, fine-tuningbased approach without meta-learning, and theoretical analysis related to the few-shot learning problem.
A prototype classifier with meta-learning On the basis of the hypothesis that features well distinguished in the training phase are also useful for classifying new classes, constructing one or multiple prototypes for classifying unseen examples is a widely used approach (Vinyals et al., 2016;
Snell et al., 2017; Pahde et al., 2021;
Ji et al., 2021; Sung et al., 2018; Allen et al., 2019; Doersch et al., 2020; Qi et al., 2018).
Certain algorithms compute similarities between multiple prototypes and unseen examples by using their own modules, such as attention mechanism (Vinyals et al., 2016;
Doersch et al., 2020), relation network (Sung et al., 2018), reweighting mechanisms by taking into account between-class and within-class interaction (Ji et al., 2021), and latent clusters (Allen et al., 2019).
Prototypes are also constructed in a multi-modal way (Pahde et al., 2021).
Another line of research is transforming the space of extracted features to a better distinguishable space (Simon et al., 2020; Yoon et al., 2019; Das et al., 2020; Das & Lee, 2020) or taking variance of fetures into account (Bateni et al., 2020) .
Because of its convenience, the approach with a prototype classifier is adapted to other domain such as semantic segmentation (Dong & Xing, 2018), text classification (Sun et al., 2019), and speech recognition (Wang et al., 2019a).
Update-based meta-learning In contrast to the approach with a prototype classifier and metalearning
, in update-based meta-learning approaches, model parameters are adjusted in the test phase so that a model can adapt to new classes.
Model-agnostic meta-Learning (MAML) and its variants (Finn et al., 2017; 2018; Rajeswaran et al., 2017) search for a good initialization parameters that adapt to new classes with a few labeled data and a few update steps of the parameters.
Another approach involves learning an effective update rule for the parameters of a base-learner model through a sequence of training episodes (Bertinetto et al., 2019; Lee et al., 2019).
Both approaches require additional learning of hyper-parameters and training time; thus, they prevent quick adaptation to new classes.
fine-tuning-based approach without meta-learning Interestingly, recent studies have shown that fine-tuning a linear classifier with features extracted using a model trained with cross-entropy loss on base-dataset performs comparably with meta-learning based methods (Chen et al., 2019)
(Wang et al., 2019b).
More effective method for fine-tuning in few-shot settings has been proposed (Yang et al., 2021;
Phoo & Hariharan, 2021), such as calibrating distribution generated by a support set (Yang et al., 2021) , self-supervised learning on query data (Phoo & Hariharan, 2021), and distilling knowledge to obtain better embeddings (Tian et al., 2020).
Liu et al.
(2020) focuses on pretraining phase and found the negative margin in cross-entropy helps improving the performance in few-shot settings.
However, similar to update-based meta-learning, these studies require additional hyperparameters and fine-tuning to applied to new classes; thus they prevent quick adaptation to new classes.
Theoretical analysis of few-shot learning Even though much improvement has been empirically made in few-shot learning, theoretical analysis is scarce.
In the context of meta-learning, Du et al.
(2021) provided a risk bound on the meta-testing phase that is related to the number of meta-training data and meta-testing data.
Lucas et al.
(2021) derived information-theoretic lower-bounds on minimax rates of convergence for algorithms that are trained on data from multiple sources and tested on novel data.
Cao et al.
(2020) derived a new bound on a prototype classifier and theoretically demonstrated that the mismatch regarding the number of shots in a support set between meta-training and meta-testing degrades the performance of prototypical networks, which has been only experimentally observed.
However, their bound depends on several assumptions: the class-conditional distributions of features are Gaussian and have the same covariance matrix among classes.
In contrast, we derive a novel bound that does not depend on any specific distribution.
T HEORETICAL A NALYSIS OF P ROTOTYPE C LASSIFIER IN T ERMS OF V ARIANCE OF N ORM OF F EATURE V ECTORS
In this section, we first formulate our problem settings and point out the drawbacks of the current theoretical analysis on a prototype classifier.
Next we provide our novel bound of a prototype classifier with the bound related to the variance of the norm of the feature vectors.
Finally, we list several methods that can improve the performance of a prototype classifier based on our bound.
3. P ROBLEM S ETTING
Let Y be a space of a class, τ a probability distribution over Y , X a space of input data, D a probability distribution over X , D y a probability distribution over X given a class y .
We define
D ⊗ nk and D ⊗ n y by D ⊗ n y =
Π ni =1 D y and D ⊗ nk ≡ Π ni
=1 D ⊗ k i , respectively.
We sample N classes from τ to form the N -way classification problem.
Denote by K a number of annotated data in each class and x ∈ X , y ∈ Y as input data and its class respectively.
We define a set of support data of class c sampled from τ as S c
= { x i | ( x i , y i ) ∈
X × Y ∩ y
i
=
c } Ki =1 and a set of support data in the N -way K -shot classification problem as S
= (cid:83) N c
=1 S c .
Suppose a feature extractor computes a function ϕ : X → R D , where D is the number of the embedding dimensions.
ϕ ( S c ) is defined by ϕ ( S c ) = 1 K (cid:80) x ∈ S c
ϕ ( x ) .
Let Φ be a space of the extractor function ϕ .
Denote by
M : Φ × X × ( X × Y ) NK → R N a prototype classifier function that computes probabilities of input x belonging to class c as follows.
[Equation]
where ∥ v ∥ 2 = (cid:80) Dd =1 (cid:0) v ( d ) (cid:1) 2 , and v ( d ) is the d -th dimension of vector v .
The prediction of an input x , denoted by ˆ y ∈ Y , is computed by taking argmax for M ( ϕ, x , S ) , i.e., ˆ y = argmax M ( ϕ, x , S ) .
We denote by E z ∼ q ( z )
[ g ( z )] an operation to take the expectation of g ( z ) over z distributed as f ( z ) and we simply denote E z ∼ q ( z ) [ g ( z )] as E z [ g ( z )] when z is obviously distributed on q ( z ) .
We define
Var z ∼ q ( z ) [ g ( z )] as an operation to take variance of g ( z ) over z distributed as q ( z ) .
With I denoting the indicator function, we define expected risk R M of a prototype classifier as
[Equation]
For simplicity, we now discuss the binary classification setting.
We show a case of multi-class classification in Appendix A.5 due to lack of space.
Let c 1 and c 2 denote any pair of classes sampled from τ .
We consider that a query data x belongs to class c 1 and support sets S consist of class c 1 ’s support set and c 2 ’s support set.
Then, equation 2 is written as follows.
[Equation]
3. W HAT F EATURE -T RANSFORMATION M ETHOD E XPECTED TO BE E FFECTIVE ?
The current theoretical analysis for a prototype classifier (Cao et al., 2020) has the following two drawbacks (see Appendix A.1 for the details).
The first is that the modeling assumption requires a class-conditioned distribution of the features to follow a Gaussian distribution with the same covariance matrix among classes.
For example, when we use the ReLU activation function in last layer, it is not normally distributed and the class-conditioned distribution does not have the same covariance matrix as shown in Figure 1.
The second drawback is that it is not clear what feature-transformation method can reduce the upper bound.
A feature-transformation method to maximize the betweenclass variance and minimize the within-class variance such as linear discriminant analysis (LDA) (Fukunaga, 1990) and Embedding Space Transformation (EST) (Cao et al., 2020) can be expected to improve performance; however, it is not clear how the second term of the denominator changes.
From Figure 1 , the distribution of each class feature stretches in the direction of the norm of its class-mean feature vector.
This property is also observed in metric learning literatures (Wen et al.,
Liu et al., 2017).
A model trained in the cross-entropy loss after linear projection from the embedding space to the class-probability space computes a probability of input x belonging to class c given by
[Equation]
where W is a weight matrix that transforms features from the embedding space to the classprobability space.
Comparing equation 4 with equation 1, we found that equation 4 does not have a regularization term similar to the one appearing in equation 1 that forces the features to be close to its class-mean feature vector.
This implies the features extracted using a model trained with the cross-entropy loss on equation 4 are less close to its class-mean feature vector than the ProtoNet loss on equation 1.
Through this observation and the property mentioned above, we hypothesis that normalizing the norm of the feature vectors can push the features to each class-mean feature vector and can boost the performance of a prototype classifier trained with cross-entropy loss on equation 4.
3. R ELATING V ARIANCE OF N ORM TO U PPER B OUND OF E XPECTED R ISK
To understand what effect the variance of the norm of the feature vectors has on the performance of a prototype classifier, we analyze how variance contributes to the expected risk when an embedding function ϕ is fixed.
The following theorem provides a generalization bound for the expected risk of a prototype classifier in terms of the variance of the norm of the feature vectors computed using a feature extractor.
Theorem 1.
Let M be an operation of a prototype classifier on binary classification defined by equation 1.
For µ c = E x ∼D c [ ϕ ( x )] , Σ c = E x ∼D c [( ϕ ( x ) − µ c )( ϕ ( x ) − µ c ) ⊤ ] , µ = E c ∼ τ [ µ c ] , Σ = E c ∼ τ [( µ c − µ )( µ c − µ ) ⊤ ] , and E c ∼ τ
[Σ c ] =
Σ τ , if ϕ ( x ) has the variance of its norm, then the miss-classification risk of the prototype classifier on binary classification R M satisfies
[Equation]
where
[Equation]
Remark.
The term E V[ h L 2 ( ϕ ( S y ))] is the variance of the norm of the feature vectors.
The term
V Tr (Σ c 1 ) is the variance of the summation with diagonal element of the covariance matrix from each class embedding; it can be interpreted as the difference of the distribution shape constructed from each class embedding.
The term V wit (Σ τ , Σ , µ ) is correlated with the within-class variance because V wit (Σ τ , Σ , µ ) Tr(Σ) is a secondary expression for the ratio of between-class variance and withinclass variance.
The term E dist 2L2 ( µ c 1 , µ c 2 ) is the expectation of the Euclidean distance between the class mean vectors .
This bound has the following properties.
Its derivation does not require the features to be distributed on any specific distributions and the class-conditioned covariance matrix does not have to be the same among classes 2.
The bound can decrease when any of the following statistics decreases with fixed betweenclass variance ( Σ ): (i) the variance of the norm of the feature vectors, as what we discussed in Section 3.2, (ii) the difference in the distribution shape constructed from each class embedding, (iii) the within-class variance ( Σ τ ), (iv) the Euclidean distance between the class-mean vectors.
As a result, our bound loosens the modeling assumption of Theorem2 in appendix A.1 and has its property in 2-(iii).
We show the proof in Appendix A.2.
3. F EATURE -T RANSFORMATION M ETHODS
We hypothesize from Theorem 1 that in addition to a feature-transformation method related to equation 6, eliminating the affect from the difference in the distribution shape among classes and lowering the ratio of between-class variance to within-class variance can improve the performance of a prototype classifier.
Regarding equation 9, the ratio of the Euclidean distance between the class mean vectors and the between-class variance is supposed to be constant.
Thus, we focus on the transformation methods relating to equation 6, equation 7 and equation 8.
We analyze the following feature-transformation methods: L 2 -normalization
( L 2 -norm), variance-normalization, LDA, EST, and EST+
L 2 -norm.
We show the detail of each method in Section A.3 due to lack of space.
E XPERIMENTAL E VALUATION
In this section, we experimentally analyzed the effectiveness of the feature-transformation methods mentioned in Section3.4.
The center loss (Wen et al., 2016) and affinity loss (Hayat et al., 2019) have been proposed to efficiently pull the features of the same class to their centers in the training phase; however, we focus on a widely used pre-trained model in this experiments following the line of studies Chen et al.
(2019) and Tian et al.
4. D ATASETS AND E VALUATION P ROTOCOL
miniImageNet.
The mini ImageNet dataset (Vinyals et al., 2016) is a standard bench-mark for fewshot learning algorithms for recent studies.
It contains 100 classes randomly sampled from ImageNet.
Each class contains 600 images.
We follow the widely-used splitting protocol (Ravi &
Larochelle, 2017) to split the dataset into 64 / 16 / 20 for training/validation/testing respectively.
tieredImageNet
The tiered ImageNet dataset (Ren et al., 2018) is another subset of ImageNet but has 608 classes.
These classes are grouped into 34 higher level categories in accordance with the
ImageNet hierarchy and these categories are split into 20 training ( 351 classes), 6 validation (classes), 8 testing categories ( 160 classes).
This splitting protocol ensures that the training set is distinctive enough from the testing set and makes the problem more realistic since we generally cannot assume that test classes will be similar to those seen in training.
CIFAR-FS
The CIFAR-FS dataset (Bertinetto et al., 2019) is a recently proposed fewshot image classification benchmark, consisting of all 100 classes from CIFAR- 100 (Krizhevsky et al.).
The classes are randomly split into 64 , 16 , and 20 for meta-training, meta-validation, and meta-testing respectively.
Each class contains 600 images of size 32 × 32 .
FC
The FC100 dataset (Lee et al., 2019) is another dataset derived from CIFAR- 100 (Krizhevsky et al.), containing 100 classes which are grouped into 20 categories.
These categories are split into 12 categories for training ( 60 classes), from 4 categories for validation ( 20 classes), 4 categories for testing ( 20 classes).
4. I MPLEMENTATION D ETAILS
We compared the feature-transformation methods applied on against ProtoNet (Snell et al., 2017),
Baseline and Baseline++ (Chen et al., 2019).
In Baseline, the linear projection layer is fine-tuned on a support set and in Baseline++, the norm of the linear projection layer and features are normalized to be constant.
We also compared with the feature-transformation method proposed in a previous study (Wang et al., 2019b).
In that study they transformed the features so that the mean of all features to be origin before L 2 -normalization and then use a prototype classifier without fine-tuning a linear classifier.
We call this operation as centering+ L 2 -norm.
We re-implemented these methods following the training procedure in previous study (Chen et al., 2019).
In the pre-training stage, where the cross-entropy loss was used and meta-learning was not used, we trained 400 epochs with a batch size of 16 .
In the meta-training stage for ProtoNet, we trained 60 , 000 episodes for 1 -shot and 40 , 000 episodes for 5 -shot tasks.
We used the validation set to select the training episodes with
[Table]
the best accuracy.
In each episode, we sampled N classes to form N -way classification (In metatraining N =20 and meta-testing N =5 following the original study of ProtoNet (Snell et al., 2017)).
For each class, we selected K labeled instances as our support set and 16 instances for the query set for a K -shot task.
In the fine-tuning or meta-testing stage for all methods, we averaged the results over 600 trials.
In each trial, we randomly sampled 5 classes from novel classes, and in each class, we also selected
K instances for the support set and 16 for the query set.
For Baseline and Baseline++, we used the entire support set to train a new classifier for 100 iterations with a batch size of 4 .
With ProtoNet, we used the models trained in the same shots as meta-testing stage since the mismatch of number of shots between meta-training and meta-testing degrades performance (Cao et al., 2020).
All methods were trained from scratch, and the Adam optimizer with initial learning rate 10 − 3 was used.
We applied standard data augmentation including random crop, horizontal-flip, and color jitter in both the training stages.
We used a ResNet12 network and ResNet18 network following the previous study (Chen et al., 2019; Yoon et al., 2019). Since LDA constructs a within-class covariance matrix from a support set and variancenormalization computes sample variance, we did not investigate the LDA score and variancenormalization score of 1 -shot settings. For LDA we set λ = 0 .
0001 for equation 18 in Appendix
A.3 and for EST we set the dimensions of the projected space to 60 following the settings of the original study (Cao et al., 2020).
4. R ESULTS
We present the experimental results on Table 1 and Table 2 on the basis of backbones with ResNet and ResNet18 for a comprehensive comparison.
Table 1 shows the results on mini ImageNet and tiered ImageNet, and Table 2 shows the results on CIFARFS and FC100.
We show the detailed performance results with 95% confidence margin and other transformation methods in A.6.
Method notations in experimental results We denote B@FT and B+@FT as the fine-tuningbased methods: Baseline and Baseline++.
We also denote B@CL2, B+@CL
as centering+ L 2norm (Wang et al., 2019b) on features trained with Baseline and Baseilne++.
B, B@L2-N, B@V-N,
B@LDA, B@EST, and B@EST+L2-N stands for a prototype classifier with applying no transformation, L 2 -norm, variance-normalization, LDA, EST, and EST+L2-N on the features extracted using a model pre-trained in the same manner as B@FT.
B+, B+@L2-N, B+@V-N, B+@LDA, B+@EST,
[Table]
and B+@EST+L2-N stands for a prototype classifier with applying no transformation, L 2 -norm, and variance-normalization on the features extracted using the same pre-trained model as with B+@FT.
Comparison of the feature transformation methods with ProtoNet, fine-tuning-based methods, and centering+ L 2 -norm From Table 1 and Table 2, we can observe that the prototype classifier with L 2 -norm and EST+ L 2 -norm performs comparably with ProtoNet and the fine-tuning-based approach in all settings and EST+ L 2 -norm performs the best among the data transformation methods. Comparing the feature-transformation methods described in Section 3.4 with centering+ L 2 -norm (Wang et al., 2019b), centering+ L 2 -norm can slightly improve the performance of the prototype classifier in several 1 -shot settings . However, in 5 -shot settings, the boost decreases and even performs worse than L 2 -norm, e.g. mini ImageNet and tiered ImageNet with ResNet12. Comparison of the feature transformation methods with B and B+
In the 1 -shot setting, although EST falls short of ProtoNet and fine-tuning-based approach, it also improves the performance of a prototype classifier.
The performance gain of both L 2 -norm, EST and EST+ L 2 -norm decrease when the number of shots increases.
This phenomenon can be explained through Theorem 1.
The term relating to the variance of the norm and the ratio of the between-class variance to the withinclass variance depends on K .Since
the term diminishes with increasing K , the performance gain of the feature-transformation methods decreases.
Discussion on the feature transformation methods Surprisingly, variance normalization performs worse than applying no feature transformation.
This can be explained by the following two reasons.
The first is that the term in the bound related to the difference among the class distribution shapes is smaller than other terms such as the variance of the norm and the ratio of the between-class variance to the within-class variance.
Figure 2 shows the term associated with the bound computed from each dataset and backbone.
The larger the value is, the more it contributes to larger risk.
We found the term associated with equation 7 is small compared with other values and thus variancenormalization did not worked well in our experiment.
The second reason is that estimating variance with a support set is unstable and make prediction unreliable in few-shot learning.
LDA and EST in 5 -shot setting does not improve performance so much compared with L 2 -norm variants while in the 1 -shot settings, EST helps improving performance.
This is because in the 5 -shot settings, the values computed from equation 8 gets smaller with larger K and the effect of equation 8 on the risk of a prototype classifier in the 1 -shot settings is larger.
Especially, EST outperforms L 2 -norm in FC with the 1 -shot setting.
We found from Figure 2 the features of FC100 shows the largest ratio of the
4. E FFECT ON C OVARIANCE M ATRIX
We further analyzed how Tr(Σ) and the trace of class c ’s covariance matrix Tr(Σ c ) fluctuate with
L 2 -norm. Since L 2 -norm lowers the variance in the direction of the normalized vector, if the covariance matrix stretches in the direction of the class mean vector, L 2 -norm can reduce its trace.
Thus we analyzed the cosine similarities between class-mean feature vectors and the eigenvectors with the largest eigen value of the covariance matrices.
Figure 3 illustrates the distribution of cosine similarities of each class-mean feature vector and the eigenvectors in test class of mini ImageNet.
This figure shows Σ c tends to stretches to the direction of class mean vector and Σ stretches in the orthogonal direction to each class-mean feature vector.
From this observation, we found that L 2norm can reduce Tr(Σ c ) and has less effect on Tr(Σ) .
L 2 -norm can reduce the variance of the norm and the ratio of the within-class variance to the between-class variance
C ONCLUSION
We theoretically and experimentally analyzed how the variance of the norm of feature vectors affects the performance of a prototype classifier and found that using EST+ L 2 -norm makes the classifier comparable with ProtoNet and the fine-tuning-based approach.
We also found that when the number of shots in a support set increases, the performance gain from a feature-transformation method decreases, which is consistent with the results of the theoretical analysis.
We believe that this work is a significant step to understanding the behavior of a prototype classifier, which is crucial for the few-shot learning problem.
A A PPENDIX A. E XISTING U PPER B OUND ON E XPECTED R ISK FOR P ROTOTYPE C LASSIFIER
To analyze the behavior of a prototype classifier, we start from the current study (Cao et al., 2020).
The following theorem is the upper bound of the expected risk of prototypical networks with the next conditions.
• The probability distribution of an extracted feature ϕ ( x ) given its class y =
c is Gaussian i.e D y = N ( µ c , Σ c ) , where µ c =
E x ∼D c [ ϕ ( x )] and Σ c = E x ∼D c
[( ϕ ( x ) − µ c )( ϕ ( x ) − µ c ) ⊤ ] .
• All class-conditioned distributions have the same covariance matrix, i e., ∀ ( c, c ′ ) , Σ c = Σ c ′ .
Theorem 2.
Let M be an operation of a prototype classifier on binary classification defined by equation 1.
Then for µ =
E c ∼ τ [ µ c ] and Σ = E c ∼ τ [( µ c − µ )( µ c − µ ) ⊤ ] , the miss-classification risk of the prototype classifier on binary classification R M satisfies
[Equation]
where E dist 2L2 ( µ c 1 , µ c 2 ) =
E c 1 ,c 2 (cid:104) (( µ c 1 − µ c 2 ) ⊤ ( µ c 1 − µ c 2 )) 2 (cid:105) .
We show the detail of the derivation in Appendix A.
[Equation] A. D ERIVATION D ETAILS OF T HEOREM
we will explain the rough sketch of derivation of Theorem 2.
In prototype classifier, from equationand equation 3, R M is written with sigmoid function σ as follows:
[Equation]
where α ≜ ∥ ϕ ( x ) − ϕ
( S c 2 ) ∥ − ∥
ϕ
( x ) − ϕ ( S c 1 ) ∥ .
we bound equation 11 with expectation and variance of α by following proposition.
Proposition 1.
From the one-sided Chebyshev’s inequality
[Equation]
it immediately follows that:
[Equation]
) equation 11 can be further write down as follows by Law of Total Expectation
[Equation]
Therefore,
[Equation]
We write down the expection and variance of α with following Lemmas 3 and 4.
Lemma 3.
Under the same notation and assumptions as Theorem 2, then,
[Equation]
Lemma 4.
Under the same notation and assumptions as Theorem 2, then,
[Equation]
The proofs of the above lemmas are in the current study (Cao et al., 2020).
With Proposition 1 and Lemma 3, Lemma 4, we obtain Theorem 2.
A. D ETAIL OF F EATURE -T RASNFORMING M ETHODS
L 2 normalization ( L 2 -norm) From equation 6, normalizing the norm of the feature vectors can improve the performance of a prototype classifier.
We denote a function normalizing the norm as ψ L 2 given by
[Equation]
(15) variance-normalization From equation 7, normalizing the Euclidean distance in equation 1 by the variance of each class embedding can improve the performance of a prototype classifier.
This can be interpreted as a variation of the Mahalanobis distance with a diagonal covariance matrix.
We denote a function that measures the distance of two vectors normalized by its variance at each dimension as follows:
[Equation]
where Var[ z ] is sample variance of z .
equation 1 is rewritten with ψ maha ( ϕ ( x 1 ) , ϕ ( x 2 )) as the following equation.
[Equation]
This normalization is similar to Bateni et al. (2020)’s work but our method differs in that we do not need to do meta-learning and we do not need to add other modules to the backbone network.
LDA From equation 8, increasing the ratio of the between-class variance to the within-class variance can improve the performance of a prototype classifier.
LDA (Fukunaga, 1990) is widely used to search for a projection space that maximizes the between-class variance and minimizes the withinclass variance.
It computes the eigenvectors of a matrix ˆΣ τ − 1 ˆΣ , where ˆΣ is the covariance matrix of prototypes and ˆ Σ τ is the class-conditioned covariance matrix.
Since the number of data is small in few-shot settings, ˆΣ τ − 1 cannot be estimated stably and we add a regularizer term to ˆΣ τ and define it as ˆΣ τ reg .
[Equation]
18) where λ ∈ R 1 , I ∈ R D
× D is identity matrix.
EST Since computation of LDA is unstable, we also analyze the effect of EST (Cao et al., 2020).
EST computes eigenvectors of a matrix ˆΣ − ρ ˆΣ τ : the difference between the covariance matrix of the class mean vectors and the class mean covariance matrix with weight parameter ρ .
Similar to
LDA, EST also searches for the projection space that maximizes the Σ and minimizes the Σ τ .
We compute the eigenvectors of the matrix with two small modification.
The first is that we remove hyper-parameter rho and then simply computes the eigenvectors of ˆΣ − ˆΣ τ .
The second is that we constructs the matrix from a support set in the multi-shot setting while we follows the original study in the 1 -shot setting.
EST+
L 2 -norm We hypothesize that the combination of the transforming methods can improve the performance of a prototype classifier independently from each other.
Specifically, we focus on reducing equation 6 and equation 8 by combining EST and L 2 -norm .
We first apply EST to reduce equation 8 and after that we apply L 2 -norm to reduce equation 6.
At the end of the operation we want the variance of the norm to be 0 , thus we apply EST and after that we apply L 2 -norm.
variance-normalization+ L 2 -norm
We focus on reducing equation 6 and equation 7 by combining variance-normalization and L 2 -norm . Following the similar procedure of EST+ L 2 -norm , we first apply variance-normalization and after that we apply L 2 -norm. We show the experimental results of this transformation methods in A.6.
LDA+ L 2 -norm We focus on reducing equation 8 and equation 7 by combining LDA and L 2norm .
Following the similar procedure of EST+ L 2 -norm , we first apply LDA and after that we apply L 2 -norm.
We show the experimental results of this transformation methods in A.6.
A
D ERIVATION D ETAILS OF T HEOREM
We start the proof from equation 13.
We first prove the following Lemma 5 related to the expectation statistics of α in equation 13.
Lemma 5.
Under the same notations and assumptions as Theorem 1, then,
[Equation]
Proof.
First, from the definition of α [Equation] we split E S ∼D ⊗ 2 k E x ∼D c 1 [ α ] in to two parts and examine them seperately. [Equation]
[Equation]
In regular conditions
[Equation]
for random vector X
[Equation]
the expectation of the norm is
[Equation]
[Equation]
and the variance of the vector is
[Equation] [Equation]
Hence,
[Equation]
where the first term inside the trace can be expanded as:
[Equation]
(Last terms cancel out)
[Equation]
The second term in equation 23 is simply as follows.
[Equation]
From equation 24 and equation 25 we obtain
[Equation]
[Equation]
Similarly for ii,
[Equation]
From ( i ) and ( ii )
[Equation]
and equation
[Equation]
Since E c 1 ,c 2 , x ,S [ α ] = E c 1 ,c 2 ∼ τ [ E S ∼D ⊗ 2 k E x ∼D c 1 [ α ]] , [Equation]
[Equation]
Next we prove the following Lemma 6 related to the conditioned variance of α
[Equation]
Lemma 6.
Under the same notation and assumptions as Theorem 1,
[Equation]
where
[Equation]
Proof.
We start with the inequality between the variance of 2 random variables.
We define
Cov( A, B ) as covariance of 2 random variables A, B .
[Equation]
[Equation] For Var ,S [ α [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] From 3 rd line to 4 th line we deompose the variance of (cid:13)(cid:13)(cid:13) ϕ ( S c 1 ) (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) ϕ ( S c 2 ) (cid:13)(cid:13)(cid:13) 2 use the independence of ϕ ( S c 1 ) and ϕ ( S c 2 ) with their class given.
Next we focus on Var (cid:104) ϕ ( x ) ⊤ ( ϕ ( S c 2 ) − ϕ ( S c 1 )) (cid:105) .
Var (cid:104) ϕ ( x ) ⊤ ( ϕ ( S c 2 ) − ϕ ( S c 1 )) (cid:105) =
E (cid:20)(cid:16) ϕ
( x ) ⊤ (cid:16) ϕ
( S c 2 ) − ϕ ( S c 1 ) (cid:17)(cid:17) 2 (cid:21) − (cid:16) E [ ϕ ( x )] ⊤ E (cid:104)(cid:16) ϕ ( S c 2 ) − ϕ
( S c 1 ) (cid:17)(cid:105)(cid:17)≤ E (cid:20)(cid:16)
∥ ϕ ( x ) ∥ 2 (cid:13)(cid:13)(cid:13) ϕ
( S c 2 ) − ϕ
( S c 1 ) (cid:13)(cid:13)(cid:13)(cid:17) 2 (cid:21) − ( µ c 1 ⊤ ( µ c 2 − µ c 1 ))=
E (cid:104)
∥ ϕ ( x ) ∥ 2 (cid:105) E (cid:20)(cid:13) (cid:13)(cid:13) ϕ
( S c 2 ) − ϕ
( S c 1 ))
(cid:13) (cid:13)(cid:13) 2 (cid:21) − (cid:0) µ ⊤ c 1 (cid:0) µ c 2 − µ c 1 (cid:1)(cid:1) 2 .
From the 2 nd line to the 3 rd line we use Cauchy–Schwarz inequality.
For E (cid:20)(cid:13)(cid:13)(cid:13) ϕ ( S c 2 ) − ϕ ( S c 1 ) (cid:13)(cid:13)(cid:13)(cid:21) , with equation
[Equation]
Thus E
[Equation]
(cid:21)(cid:21) is calculated as follows.
[Equation]
Now we take into account the term − ( µ c 1 ⊤ ( µ c 2 − µ c 1 )) 2 in equation 34,
[Equation]
E c 1 ,c 2 (cid:104)
Var[ ϕ ( x ) ⊤ ( ϕ ( S c 2 ) − ϕ
( S c 1 ))] (cid:105) is calculated as follows.
[Equation]
[-K (cid:88) i =
-]
Regarding (cid:13)(cid:13)(cid:13) ϕ
( S c ) (cid:13)(cid:13)(cid:13) 2 , since the function computing square norm is convex
[Equation]
next equation holds with
[-K  (cid:88) x i ∈ S c (cid:107) φ ( x i ) (cid:107) 2 + (cid:88) i (cid:54) = j xi,xj ∈ Sc ( (cid:107) φ ( x i ) (cid:107) 
 + (cid:107) φ ( x j ) (cid:107) 2 )   = K(cid:32)-]D -dimensional Jensen’s inequlality (Perlman, 1974):
[Equation]
[Equation]
Combining equation 33, equation
[Equation]
and equation 40 we obtain
[Equation]
To complete the proof of Theorem
[Equation]
we calculate E
[Equation]
[Equation]
From 2 nd line to 3 rd line, we use the symmetry of the last term with respect to c 1 and c 2 and erase the term.
Combining equation 13, Lemma 5, Lemma 6, and equation 42, we obtain the bound.
A. T HEOREM 1 WITH N - WAY C LASSIFICATION
The upper bound on the risk of N -way prototype classifier is as follows.
Theorem 7.
Let operation of binary class prototype classifier M as defined in equation 1.
Then for ϕ ( S c ) = 1 K Σ x ∈ S c
ϕ
( x ) , µ c = E x ∼D c [ ϕ ( x )] , Σ c = E x ∼D c
[( ϕ ( x ) − µ c )( ϕ ( x ) − µ c ) ⊤ ] , µ =
E c ∼ τ [ µ c ] , Σ
= E c ∼ τ [( µ
[Equation]
µ ) ( µ c − µ ) ⊤ ] , E c ∼ τ [Σ c ] =
Σ τ , if ϕ ( x ) has its fourth moment, miss classification risk of binary class prototype classifier R M satisfy
[Equation]
[Equation]
where
[Equation]
= 4 K E y ∼ τ (cid:104)
Var x c ∼D c (cid:104) ∥ ϕ ( x ) ∥ 2 (cid:105)(cid:105) , V Tr (Σ y ) =
(cid:18) K + K(cid:19) Var c ∼ τ [Tr (Σ c )] ,
V wit (Σ τ , Σ , µ ) = 8 K Tr(Σ τ ) (cid:0) Tr(Σ τ ) + Tr(Σ) + µ ⊤ µ (cid:1) + 4(Tr(Σ) + µ ⊤ µ ) 2 , E dist 2 L2 ( µ y , µ c ) = E y,c (cid:20)(cid:16)(cid:0) µ c y − µ c (cid:1) ⊤ ( µ y − µ c ) (cid:17) 2 (cid:21) .
[Table]
Proof.
Let x [Table] y be the input and its class of a query data.
We define α c by α c = (cid:13)(cid:13)(cid:13) ϕ ( x ) − ϕ ( S c ) (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) ϕ ( x ) − ϕ ( S y ) (cid:13)(cid:13)(cid:13) 2 .
Then a prototype classifier miss-classify a class of input x , ˆ y , if ∃ c ∈ [1 , N ] , c ̸
= y, α c < 0 .
Hence: R M ( ϕ ) =
Pr( (cid:83)
Nc =1 c ̸ =
y α c < 0)
By Frechet’s inequality, next equation holds:
[Equation]
Noting that Theorem 1 can be applied to each term in the summation and then we obtain Theorem
A. D ETAILED PERFORMANCE RESULTS
We show the detailed performance results in this section with 95% confidence margin.
We add the two future normalization methods, variance normalization before L 2 -normalization (V-N+L2N) and LDA before L 2 -normalization (LDA+L2-N) to the tables.
Table 3 and Table 4 show the performance on mini Imagenet and tiered Imagenet; table 5 and table 6 show the performance on
CIFARFS and FC100.
A. T HE COMPARISON OF THE RATIO OF THE BETWEEN - CLASS VARIANCE TO THE WITHIN - CLASS VARIANCE BEFORE AND AFTER APPLYING L 2 - NORM
We show the ratio of the between-class variance to the within-class variance before and after applying L 2 -norm in Figure 4.
We calculated the ratio of the between-class variance to each class’s variance and averaged over the test classes of each dataset.
Although L 2 -normalization is expected to reduce Tr(Σ) rather than Σ c according to 4.4, the figure shows L 2 -norm marginaly reduces the ratio in CIFARFS dataset and the ratio does not changed so much in the other dataset.
[Table]
[Table]
[Table]