Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport
Anonymous Author(s)
AffiliationAddress email
Abstract
Few-shot classification aims to learn a classifier to recognize unseen classes duringtraining, where the learned model can easily become over-fitted based on the biaseddistribution formed by only a few training examples.
A recent solution to thisproblem is calibrating the distribution of these few sample classes by transferringstatistics from the base classes with sufficient examples, where how to decide thetransfer weights from base classes to novel classes is the key.
However, principledapproaches for learning the transfer weights have not been carefully studied.
To thisend, we propose a novel distribution calibration method by learning the adaptiveweight matrix between novel samples and base classes, which is built upon ahierarchical Optimal Transport (H-OT) framework.
By minimizing the high-level OT distance between novel samples and base classes, we can view the learnedtransport plan as the adaptive weight information for transferring the statistics ofbase classes.
The learning of the cost function between a base class and novelclass in the high-level OT leads to the introduction of the low-level OT, whichconsiders the weights of all the data samples in the base class.
Experimentalresults on standard benchmarks demonstrate that our proposed plug and play modeloutperforms competing approaches and the learned transport plan can be used torepresent the similarity between base classes and novel samples.
Introduction
Deep learning models have become the regular ingredients for numerous contemporary computervision tasks such as image classification [1] and achieve state-of-the-art performance.
However,the strong performance of deep neural networks typically relies on abundant labeled instances fortraining [2].
Considering the high cost of collecting and annotating a large amount of data, a majorresearch effort is being dedicated to fields such as transfer learning [3] and domain adaptation [4].
As a trending research subject in the low data regime, few-shot classification aims to learn a modelon the data from the base classes, so that the model can generalize well on the tasks sampled fromthe novel classes.
Several lines of works have been proposed such as those based on meta-learningparadigms [5–11] and those directly predicting the weights of the classifiers for novel classes [12, 13].
Recently, methods based on distribution calibration gains increasing attention.
As a representativeexample, Yang et al.
[14] introduce to calibrate the feature distribution of the few-sample classes bytransferring the statistics from the base classes and utilize the sampled data to train a classifier fornovel classes.
The unique advantage of distribution calibration methods over others is that they buildon top of off-the-shelf pretrained feature extractors and do not finetune/re-train the feature extractor.
The key of distribution calibration methods is to select the corresponding base classes and transfertheir statistics for the labeled samples in a novel task.
Existing approaches in this line usually do sowith heuristic or less adaptive solutions.
Specifically, Yang et al.
[14] use the average features of the samples as the representation of a base class and select the top- w (e.g., w = 2 ) closest base classesbased on the Euclidean distance between the features of a novel sample and a base class.
Althoughwith reasonably good performance, it is questionable whether the Euclidean distance is the propermetric to measure the closeness between a base class and a novel sample since viewing a novel sampleand a base class as points in the same space may not be the best solution.
Moreover, it is less soundto characterize a base class only by the unweighted average over all its samples, when measuringits closeness with the novel sample.
Representing a base class in this way would completely ignorethe fact that each sample of a base class may contribute to the classification boundary differently.
Finally, it may also be less effective to treat each top- w base classes equally as their contributions canalso be different, not to mention the omission for other base classes.
To improve the above aspects, this work develops a more adaptive and principled distributioncalibration method leveraging optimal transport (OT), which is a powerful tool for measuring thedistance traveled in transporting the mass in one distribution to match another given a specific costfunction.
First, we formulate a distribution P over the base classes and a distribution Q over thelabelled samples from the novel classes.
With such formulation, how to transfer the statistics fromthe base classes to the novel samples can be viewed as the OT problem between two distributions,denoted as the high-level OT.
By solving the high-level OT, the learned transport plan can be usedas the similarity or closeness between novel samples and base classes.
Since the high-level OTrequires specifying the cost function between one base class and one novel sample, we furtherintroduce a low-level OT problem to learn this cost automatically, where we formulate a base class asa distribution over its samples.
In this way, the similarity between a novel sample and a base class isno longer representing a base class by the unweighted average over all its samples and then using the Euclidean distance.
In our method, the weights of the samples are considered in a principled way.
Insummary, the statistics of base classes can be better transferred to the novel samples for providing amore effective way to measure the similarity between them.
We can refer to this adaptive distribution calibration method as a novel hierarchical OT framework(H-OT) for few-shot learning, which is applicable to a range of semi-supervised and supervised tasks,such as few-shot classification [9] and domain adaptation [4].
We summarize our contributions asfollows: (1) We develop a new distribution calibration approach for few-shot learning, which can bebuilt on top of an arbitrary pre-trained feature extractor for being implemented over the feature-level,without further costly fine-tuning.
(2) We formulate the task of transferring statistics from base classesto novel classes in distribution calibration as the H-OT problem and tackle the task with a principledsolution.
(3) We apply our method to few-shot classification and also explore the cross-domaingeneralization ability.
Experiments on standardized benchmarks demonstrate that introducing the
H-OT into distribution calibration method can learn adaptive weight matrix, paving a new way totransfer the statistics of base classes to novel samples.
Background2.1 Optimal Transport Theory
Owing a rich theory, OT has been widely used for the comparison of probabilities.
Here we limitour discussion to OT for discrete distributions and refer the reader to Peyré and Cuturi [15] formore details.
Denote p = (cid:80) ni
=1 a i δ x i and q = (cid:80) mj
=1 b j δ y
j
as two n and m dimensional discreteprobability distributions, respectively.
In this case, a ∈ ∆ n and b ∈ ∆ m , where ∆ m denotes theprobability simplex of R m .
The OT distance between p and q is defined as
[Equation]
[Equation] ) where ⟨· , ·⟩ denotes the Frobenius dot-product; C ∈ R n × m ≥is the transport cost function with element
C ij = C
( x i , y j ) ; T ∈ R n × m >denotes the doubly stochastic transport probability matrix such thatΠ( p, q ) :
= { T | (cid:80) ni T ij = b j , (cid:80) mj T ij = a i } , meaning that T has to be one of the joint distributionof p and q .
As directly optimising Equation (1) can be time-consuming for large-scale problems,the entropic regularization, H = − (cid:80) ij T ij ln T ij , is introduced in Cuturi [16], resulting in thewidely-used Sinkhorn algorithm for discrete OT problems with reduced complexity.
2.2 Few-Shot Classification
Following a typical few-shot learning problem, we divide whole dataset into a base dataset D base ,a validation dataset D vali , and a novel dataset D novel such that D base ∩ D novel ∩ D vali = ∅ .
Here, D base owns a large number of labelled examples from B different classes while D novel contains veryfew labelled examples, along with some unlabelled ones, all from N all classes. The most commonway to build a task is called an N -way- K -shot task [8], where N classes are sampled from the N allclasses and only K ( e.g. , 1 or 5) labeled samples are provided for each class. The few availablelabeled data are called support set S =
{ ( x i , y i ) } N × K i = and the model is evaluated on another query set
Q = { ( x i , y i ) } N × K + N × q i = N × K +1 , where each class in the task has q test cases.
The performance of a modelis evaluated as the averaged accuracy on (the query set of) multiple tasks from the novel classes.
2.3 Distribution Calibration for Few-Shot Classification
Distribution calibration [14] uses the statistics of base classes to estimate the statistics of novelsamples in the support set and generate more samples.
Specifically, for base class b ∈ B , its samplesare assumed to generate from a Gaussian distribution, with following mean and covariance matrix:
[Equation]
where x j ∈ R V is the V -dimensional feature of sample j extracted from the pre-trained featureencoder, J b the total number of samples in class b , and { µ b , Σ b } are the statistics of base class b .
The samples of a novel class are also assumed to be generated from a Gaussian distribution withmean µ ′ and covariance Σ ′ .
As the novel class only has one or a few labeled samples, it is hard toaccurately estimate the Gaussian distribution.
Thus, the key idea is to transfer the statistics of the baseclasses to calibrate the novel class’s distribution.
Once the distribution of the novel class is calibrated,we can generate more samples from it, which are useful for training a good classifier.
As a result,how to effectively transfer the statistics from the base classes is critical to the success of distributioncalibration based methods for few-shot learning.
Accordingly, Free-Lunch [14] designs a heuristicapproach that calibrates the Gaussian parameters of a novel distribution with one data sample x :
[Equation]
where: 1) ˜ x is the transformed data by Tukey’s Ladder of Powers transformation (TLPT) [17], i.e. ,˜ x
= x β if β ̸ = 0 and ˜ x = log( x ) if β = 0 , for reducing the skewness of distributions and makedistributions more Gaussian-like. 2) S d = (cid:110) − ∥
µ b − ˜ x
∥ 2 | b ∈ B (cid:111) is a distance set defined by the
Euclidean distance between the transformed feature ˜ x of a novel sample in support set and the meanµ b of the base class b . 3) topw( S d ) is the operation that selects the top- w closest base classes from theset S d . 4)
α determines the degree of dispersion of features sampled from the calibrated distribution.
Although effective, Free-Lunch represents a base class by the unweighted average over all its sampleswhen computing its closeness with a novel sample, which ignores the fact that each sample of a baseclass may contribute to the classification boundary differently.
In addition, the Euclidean distancein the feature space may not well capture the relations between a base class and a novel sample.
Moreover, each selected top- w base class has a equal weight ( i.e. , 1 /w ), which may not reflect thedifferent contributions of the base classes and omit useful information in unselected base classes.
Our Proposed Model
3.1 Overall Method
In this work, we propose a novel adaptive distribution calibration framework, a holistic methodfor few-shot classification.
Compared to the novel classes, which only have a limited number oflabeled samples, the base classes typically have a sufficient amount of data, making their statisticsbeing estimated more accurately.
Due to the correlation between novel classes and base classes,it is reasonable to use the statistics of base classes to revise the distribution of the novel sample.
Therefore, the key is how to transfer the statistics from the base classes to a novel class to achievethe best calibration results, which is the focus of this paper.
Here, we develop the H-OT to learn the Transport Plan matrix between base classes and novel samples, where each element of the transportplan measures the importance of each base class for each novel sample and more relevant classesusually have a larger transport probability.
Computing the high-level OT requires the specificationof the cost function between one base class and one novel class, which leads to the introduction ofa low-level OT problem.
By viewing the learned transport plan as the adaptive weight matrix, weprovide an elegant and principled way to transfer the statistics from base classes to the novel classes.
3.2 Hierarchical OT for Few-Shot Learning
Moving beyond Free-Lunch of Yang et al. [14], which uses the Euclidean distance between a novelsample and a base class to decide their similarity and endow the chosen base classes with the equalimportance, we aim to capture the correlations between the base class and novel samples at multiplelevels and transfer the related statistics from base classes to novel samples.
We learn the similarityby minimising a high-level OT distance between base and novel classes and build the cost functionused in high-level OT by further introducing a low-level OT distance.
To formulate the task as thehigh-level OT problem, we model P as a following discrete uniform distribution over B base classes:
[Equation]
where R b represents the base class b in the V -dimensional feature space, which will be introducedlater.
Taking the N -way- 1 -shot task as the example, where each novel class has one labeled samplex , we represent Q as a discrete uniform distribution over N novel classes from support set:
[Equation]
) where ˜ x n is the transformed feature from x following Yang et al.
[14] and detailed under Equation 3.
The OT distance between P and Q is thus defined as OT ( P, Q ) =
min T ∈ Π( P,Q ) ⟨ T , C ⟩ .
We adopt aregularised OT distance with an entropic constraint [16] and express the optimisation problem as:
[Equation]
(6) where ϵ > 0 , C ∈ R B × N ≥is the transport cost matrix and C bn indicates the cost between baseclass b and novel sample n .
Importantly, the transport probability matrix T ∈ R B × N >should satisfyΠ( P, Q ) :
= (cid:110)(cid:80) Nn T bn = 1 B , (cid:80)
Bb T bn = 1 N (cid:111) with element T bn = T ( R b , ˜ x n ) , which denotes thetransport probability between the b th base class and the n th novel sample and is an upper-boundedpositive metric.
Therefore, T bn provides a natural way to weight the importance of each base classwhich can be used as the class similarity matrix when calibrating the novel distribution.
Hence, thetransport plan is the main thing that we want to learn from the data.
To compute the OT in Equation 6, we need to define the cost function C , which is the main parameterdefining the transport distance between probability distributions and thus plays the paramount role inlearning the optimal transport plan.
In terms of the transport cost matrix, a naive method is to specifythe C with Euclidean distance or cosine similarity between the feature space of the novel sample andmean of the features from the base classes, such as C bn =1 − cos (˜ x n , µ b ) .
However, these manuallychosen cost functions might have the limited ability to measure the transport cost between a baseclass and a novel sample.
Besides, representing the base class only with the average of all features inclass b might ignore the contributions of different samples for this class.
Hence the optimal transportplan for these cost functions might be severely biased.
To this end, we further introduce a low-level OT optimization problem to automatically learn the transport cost function C in (6).
Specifically, wefurther treat each base class b as an empirical distribution R b over the features within this class:
[Equation]
where p bj is the weight of data x bj to base class b and captures the importance of this sample and willbe described in short order.
Specifically, we train a classifier parameterized by ϕ with the samples inthe base classes, which predicts which base class a sample is in.
The predicted probability of samplej belonging to the base class b is denoted by s bj and then [ p b 1 , . .
, p bJ b ] is obtained by normalizing thevector [ s b 1 , . . .
, s bJ b ] with the Softmax function.
We further define the low-level OT distance betweeneach distributions R b and distributions Q with an entropic constraint as
[Equation]
where the transport probability matrix M b ∈
R J b × N >should subject to Π( R b , Q ) :
= 176 (cid:110)(cid:80) J b j
[Equation]
The element D bjn of cost function D b ∈ R J b × N >is the distance between the n -th novel sample and the j -th sample from base class b , where we naturallyuse the distance between their features, i.e., ˜ x n and x bj .
Empirically, we find that the cosine similarity
D bjn = 1 − cos (˜ x n , x bj ) works well in practice, although other choices are possible.
Minimizing the low-level OT loss in Equation (8) can learn an optimal transport probability matrix
M b for b ∈ B , where M bjn tells us the transport weight between the n th novel class and j th samplein b base class.
Back to the cost function C bn for the high-level OT, instead of using the manuallychose cost functions, we further adopt the learned total transport cost between novel sample x n andall samples from base class b in the low-level OT to represent the cost function C bn in high-level OT:
[Equation]
where C bn is fed into (6) for learning the transport plan between base classes and novel samples.
Defined in this way C bn is an adaptive cost between base class b and novel sample n in supportset, taking full advantage of all samples in a base class.
With the proposed low-level OT distance,the distance between a novel sample and a base class is no longer representing a base class by theunweighted average over all its samples and then using the Euclidean distance.
In our method, theweights of the samples are considered in a principled way.
3.3 Calibrating Distribution and Training Classifier
Once we obtain the transport plan matrix by minimizing the OT problem in (6), we can compute thestatistics of novel samples by following Yang et al.
For the n -th transformed feature ˜ x n in thesupport set, we calibrate its mean and covariance as follows:
[Equation]
where µ b and Σ b are the statistics of base class b by (2), respectively; T bn provides an adaptiveway to weight the importance of base class b for novel sample n ; N is used to scale T bn since 197 (cid:80) b ∈ B T bn = 1 /N ; α is a hyper-parameter explained in Equation (3).
For N -way- K -shot task with K > 1 , the N in aforementioned Equations (6) and (8) should bereplaced with N ∗ K , and the distribution calibration in Equation (10) should be undertaken K timesfor each novel class.
Notably, this way can potentially achieve more diverse and accurate distributionestimation for avoiding the bias provided by one specific sample.
Thus, for a class y ∈ N in noveltask, we denote the calibrated distribution with a set of statistics as
[Equation]
Based on set S y for novel class y , we can sample from the calibrated Gaussian distributions togenerate a set of feature vectors with label y :
D y =
{ ( x, y ) | x ∼ N ( µ, Σ) , ∀ ( µ, Σ)
∈ S y } .
Giventhe transformed support set ˜ S with TLPT and the generated features D y , we only need to train theclassifier for a task T ( e.g. , N-way-K-shot) by minimizing the cross-entropy loss:
[Equation]
(12) where Y T is the set of classes for the task T and the classifier is parameterized by θ .
We describe ourproposed framework in Algorithm 1.
Related Work
Recently, much efforts have been devoted to explore the idea of learning to learn or meta-learningto alleviate the few-shot challenge.
One approach to this problem is the optimization-based metalearning algorithms [6, 7], which aim to learn a single set of model parameters that can be adapted toindividual tasks with a small number of gradient update steps.
Besides, some simple but effectivealgorithms based on metric learning are also developed, with the goal of “learning to compare”.
Forexample, MatchingNet [8] and ProtoNet [9] learned to classify samples by comparing the distanceto the representatives of each class.
Another line of algorithms is to compensate for the insufficient samples by learning to augment.
Most methods augment the training set using Generative Adversarial
Networks (GANs) [18] or autoencoder [19] to generate samples [20–22] or features [23, 24].
Recently, Free-Lunch [14] is proposed to calibrate the distribution of the novel samples usingthe statistics from base classes and augment the data from the calibrated distribution.
Althoughdistribution calibration belongs to the scope of data augmentation, it does not need to build anycomplex generative models or fine-tune the backbone.
Therefore, distribution calibration can beconsidered as a new promising line, where both Free-Lunch and our model fall into.
Different from Free-Lunch that adopts a heuristic method to choose base classes for novel samples, we develop anovel hierarchical OT to learn the cost function and similarity (transport plan) between base classesand novel samples, providing a general and adaptive distribution calibration framework for few-shotlearning.
A recent metric-based work of connecting few-shot learning with OT is DeepEMD [25],which decomposes an image into a set of local features and uses the optimal matching cost betweentwo images to represent their similarity.
However, we directly minimize the OT between base classesand novel samples to learned the transport plan, which serves as the adaptive weight matrix indistribution calibration.
Although both using OT for few-shot learning, DeepEMD and ours aretotally different methods.
Experiments
5. Experimental Setup
Datasets We evaluate our proposed method on several standard few-shot classification datasets withdifferent levels of granularity, including miniImageNet [26], tieredImageNet [27], CUB [28], and
CIFAR-FS [29].
Here, miniImageNet, tieredImageNet and CIFAR-FS have a broad range of classesincluding various animals and objects while CUB is a more fine-grained dataset that includes variousspecies of birds.
The miniImageNet , which is randomly chosen from ILSVRC-12 dataset [30],contains 100 diverse classes with 600 samples per class.
The image size is 84 × 84 × 3 .
Followingthe previous work [26], we split the dataset into 64 base classes, 16 validation classes, and 20 novelclasses.
The tieredImageNet is a larger subset of ILSVRC-12 dataset [30].
It contains 608 classessampled from hierarchical category structure, where each class belongs to one of 34 higher-levelcategories from the high-level nodes in the ImageNet.
The average number of images in each class is1281.
We adopt 351, 97, and 160 classes for training, validation, and test, respectively.
The CUB is afine-grained few-shot classification benchmark, which contains 200 different classes of birds with atotal of 11,788 images, and we resize them as 84 × 84 × 3 .
We follow previous works [2] and splitthe dataset into 100 base classes, 50 validation classes, and 50 novel classes.
The CIFAR-FS datasetis a recently proposed few-shot image classification benchmark, consisting of all 100 classes from
CIFAR-100 [31].
The classes are randomly split into 64 , 16 and 20 for meta-training, meta-validation,and meta-testing, respectively.
Each class contains 600 images of size 32 × 32 × 3 .
Implementation Details Since Free-Lunch is the SOTA distribution calibration method, we mainlycompare with Free-Lunch.
For miniImageNet and CUB, we directly use pre-extracted features from
Free-Lunch’s code [14], which are produced by WideResNet trained from scratch on the base classesby optimizing a classification loss and a self-supervised rotation loss jointly following Mangla et al.[1], denoted as WRN28+RTLoss.
Free-lunch did not share the features on tieredImageNet and
CIFAR-FS, thus we have taken the pre-trained WRN28+RTloss backbone of Mangla et al.
[1] toextract the features, where we use this same backbone to implement Free-lunch with its official code [Table] and we have explored and selected an appropriate w for Free-lunch.
To confirm the validity of ourproposed method on the commonly-used backbone, we also take the ResNet12 as the example, whichis trained from scratch by minimizing a standard classification loss (without the rotation loss).
Tomake sure the inputs to TLPT are all non-negative, we extract the features of given samples fromthe penultimate layer (with a ReLU activation function) of the feature extractor.
To demonstrate theeffectiveness of our method, we consider the simple Logistic Regression (LR) classifier for θ and ϕin Algorithm 1, where the LR implementation of scikit-learn [32] with the default settings is adopted.
We use the same hyper-parameter value for all datasets except for β .
Specifically, the number ofgenerated features is 750 , the ϵ in Sinkhorn algorithm is 0 .
01 , the α in (10) is 0 . 21 ; and β is 0 .
5 , 1 , 1and 0 .
8 for miniImageNet, tieredImageNet, CUB, and CIFAR-FAS, respectively, selected by a gridsearch using the validation set.
The maximum iteration number in Sinkhorn algorithm is set as 200 .
The reported results are the averaged classification accuracy over 10,000 tasks.
5.2 Evaluation with
the Standard Setting
[Table]
We now conduct experiments on the most common setting in few-shot classification, 1-shotand 5-shot classification, where the results ofdifferent models on the miniImageNet, tieredImagenet, CUB and CIFAR-FS are shown in Table1 and Table 2, respectively.
To validate the effectiveness of our proposed model, we compare itwith three main groups of the few-shot learningmethod, including optimization-based, metricbased [Table] and distribution calibration-based.
Thereare several observations.
First, we can find thatour H-OT equipped with WRN28+RTLoss andsimple linear classifier outperforms all previouscompeting methods on 5-way-1-shot and 5-way5-shot settings, which illustrates the superiorityof the proposed framework.
Second, our H-OTequipped with ResNet12 is better than other baselines except for the 5way1shot task on miniImageNet,where in this case our H-OT is still competing with DeepEMD [25].
This observation proves theeffectiveness of distribution calibration and transferring the statistics from base classes to novelclasses using OT.
Third, the comparison between our model and the strong baseline of distributioncalibration based methods, i.e., Free-Lunch [14], confirms the validity of incorporating the adaptiveweight matrix when transferring the statistics from the base classes to the novel classes.
Althoughboth methods utilize distribution calibration to perform few-shot classification, our model can moreeffectively transfer the knowledge of the base classes to estimate the distribution of novel samplesmore accurately with the help of hierarchical OT (H-OT).
Fourth, in terms of 5-way 1-shot tasks onall datasets, the performance gain from our model over Free-Lunch on the CUB is larger than thatof other datasets.
However, for 5-way 5-shot tasks on all datasets, the improvement gain from ourmodel over Free-Lunch on the CUB is not as significant as that on other datasets.
The reason behind this might be that CUB is a fine-grained image classification dataset, which makes it more effectivefor statistics transfer compared to other datasets under the 1-shot setting.
Furthermore, with thedevelopment of label examples in the novel classes (5-shot), our method can perform more effectiveknowledge transfer from base classes to novel samples in the coarse-grained image classificationdataset.
The results reveal that the granularity of the dataset is an important factor in the currentfew-shot classification setting.
Our model avoids the costly fine-tuning for backbone and it only takestime at the testing stage.
We defer the details on comparison of computational cost to Appendix Aand summary of test results to Appendix B.
5.3 Evaluation with the Cross-domain Setting
[Figure]
To explore the cross-domain generalization ability of H-OT for the few-shot classification task, we consider a practical evaluationsetting following Chen et al.
[2], where there exists domain shiftbetween base and novel classes (e.g., sampling base classes froma coarse-grained dataset and novel classes from a fine-graineddataset).
We design two cross-domain scenarios: miniImageNet→ CUB, CIFAR-FS → CUB, where CUB is the target domainwhile both miniImageNet and CIFAR-FS are the source domains.
Compared with collecting images from a general class, collectingimages from fine-grained classes might be more difficult, so thesetwo scenarios are very meaningful in reality.
We consider thecomparison with Free-Lunch, which is closely related to ours.
Asshown in Fig.
1, both H-OT and Free-Lunch achieve acceptableperformance in these two scenarios, proving the effectiveness ofthe distribution calibration framework in the cross-domain case.
Besides, H-OT shows its clear advantage over Free-Lunch.
It indicates that H-OT can better utilizethe statistics from the base classes of the source domain with the adaptive weight matrix and transferthem to the novel classes of the target domain.
Moreover, H-OT under the CIFAR-FS → CUB (bluebar in the 3rd column) scenario still outperforms Free-Lunch under CUB (orange bar in the 1stcolumn).
These results demonstrate that H-OT is less affected by domain shift between the baseclasses and novel classes and possesses the desired cross-domain generalization ability
[Table]
[Table]
[Table]
5.4 Ablation Study and Qualitative AnalysisImpact of adaptive cost from low-
level OT Recall that we formulate our general goal as a high-level
OT problem
and we propose a low-level OT for computing the cost function for the high-level one inan adaptive way.
The low-level OT captures the distance between a base class and a novel class.
Herewe consider two commonly-used cost functions to replace the adaptive cost, including Euclideandistance and cosine similarity between the feature space of the novel classes and the mean of thefeatures from the base classes, denoted as High-level OT+Euc and High-level OT+cos.
Besides, toexclude the possibility that the gain of our H-OT comes only from the effect of introducing weightedmean in the low-level OT, we also consider use the weighted mean with (7) to implement High-level
OT, denoted as High-level OT+Euc+weighted and High-level OT+cos+weighted.
As summarizedin Table [-
 ,-]3, our proposed H-OT achieves better performance than these degraded variants, whichconfirms the validity of adaptive cost from low-level OT in few-shot classification.
And introducingthe weighted mean into the cost function of high-level OT does not improve the performance.
Itindicates that the performance gain comes mainly from the low-level OT rather than the effect ofweighted mean.
The reason behind this might be that our proposed low-level OT considers all sampleswithin a base class when deciding the cost between novel samples and this base class, instead ofusing the mean of the features from the base class.
Besides, even with the given cost functions, our proposed method still outperforms the competing baseline (i.e., Free-Lunch), indicating theusefulness of transport plan (i.e., adaptive matrix) learned from high-level OT.
Different backbones Since our framework is agnostic with the choice of backbones, following Yanget al.
[14], we further employ other commonly used backbones, such as networks with four or sixconvolutional layers (Conv4 or Conv6), Resnet10 and Resnet18, which are trained from scratchfollowing [2].
Table 4 illustrates the classification performance (5-way-1-shot) comparison betweenour H-OT and strong baseline (Free-lunch) on miniImageNet with different backbones.
We find that
H-OT always outperforms its baseline given the same backbone.
This result reveals the superiority oflearning an adaptive weight matrix to select statistics of base classes on different backbone.
As H-OTis agnostic to the feature extractor and does not require costly fine-tuning, it can also convenientlywork with more advanced feature extractors to get better performance.
Number of generated features Fig.
2 shows the variation of the classification performance ofdistribution calibration models with the increasing number of generated features, where we considerour H-OT (red) and Free-Lunch (green) in a 5-way-1-shot classification setting for miniImageNet.
We can see that increasing the number of generated samples results in consistent improvement in bothcases.
However, given the same number of generated samples, H-OT outperforms its baseline, whichonly selects the top- w base classes for each novel sample using Euclidean distance [14].
Interestingly,our model achieves around 68.5 accuracy with only 50 generated samples while Free-Lunch takes750 generated samples to achieve the similar result.
Hence, the adaptive weight matrix learned by H-OT can contribute to the distribution calibration strategy for sampling more representative features.
Learned transport plan To explore whether the learned transport plan can capture the correlationsbetween the base classes and one given sample from the novel class (“plain”), we show a list of baseclasses with decreasing ranks in Fig.
The top-5 base classes are highly related to the given novelsample about their semantic meanings.
It is reasonable since the images from wild animal-relatedclasses are usually taken on the plain, such as “elephant” and “bear”.
It indicates that the transportplan can be successfully optimized by our H-OT and provides an elegant way to measure the closenessbetween base classes and novel samples.
Besides, different from Free-Lunch that only selects top- wbase classes ( w =2), our model takes full advantage of all base classes for learning adaptive weight foreach base class.
Additional quantitative results and qualitative are deferred to the Appendix B.
Conclusion
This paper introduces a novel hierarchical OT to improve the existing distribution calibration-basedalgorithm for few-shot learning.
We first develop a low-level OT problem to learn the cost betweennovel samples and a base class, which takes into account different weights of the samples in thebase class.
Moreover, we further design a high-level OT for measuring the distance between novelsamples and all base classes, which uses the cost learned from the low-level one to define the costfunction rather than manually chosen cost functions.
The proposed H-OT can capture the correlationsbetween the novel samples and base classes with a two-level hierarchy.
When transferring thestatistics from base classes to novel samples, we view the learned optimal transport plan betweenthem as the adaptive weight matrix, providing an effective way to weight each base class for a novelsample.
Extensive experiments have been conducted, showing that our proposed framework achievescompeting performance on commonly few-shot learning problems and cross-domain scenarios.