F EW -S HOT
C LASSIFICATION
WITH
T ASK -A DAPTIVE
S EMANTIC F
EATURE
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot classiﬁcation aims to learn a classiﬁer that categorizes objects of unseen classes with limited samples.
One general approach is to mine as much information as possible from limited samples.
This can be achieved by incorporating data aspects from multiple modals.
However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples.
Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples.
Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples.
The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of the support samples.
Then the query samples can obtain the semantic features with this module.
Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets, which signiﬁcantly improves model generalization.
We develop two modality combination implementations: feature concatenation and feature fusion, based on the semantic feature learner.
Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts, proving the effectiveness of our method.
I NTRODUCTION
Deep neural network models have gained huge success in various tasks, some of them even achieved the human-level performances (He et al., 2016; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014).
Nevertheless, such success relies on sufﬁcient annotated training data, which limits the applicability of the models to new concepts with very little supervision.
Few shot learning has emerged as an appealing paradigm to solve this problem (Fei-Fei et al., 2006).
It aims to bridge the sample-efﬁciency gap between machine learning and human learning in various application ﬁelds (Wang et al., 2020), such as image classiﬁcation, object detection and domain adaptation (Dixit et al., 2017; Motiian et al., 2017; Snell et al., 2017).
Few-shot learning (FSL) can classify instances from unseen classes with only limited number of labeled samples.
It has witnessed signiﬁcant advances in few-shot classiﬁcation.
In FSL setting, a series of N-way K-shot few-shot tasks (or episodes) are created, where each task (or episode) contains a support set with K labeled examples per classes as supervision, and a query set with some unlabeled samples to be predicted.
Meta-learning is one popular way of developing few-shot learning strategies, which has been proved effective for few-shot classiﬁcation.
It leverages a series of few-shot tasks generated from base dataset to learn a set of parameters of the desired classiﬁer that can generalize well to novel tasks (Finn et al., 2017; Ren et al., 2018; Rusu et al., 2018).
Another category of approach is Metric-based method.
It focuses on learning a generalizable embedding model to transform all samples into a discriminative metric space, in which, samples of the same classes are clustered together, while distances between samples of the different class are widened.
And a non-parametric, distance-based metric is adopted to make classiﬁcation for the unlabeled samples, which simpliﬁes the classiﬁcation process in the embedding space (Gidaris & Komodakis, 2018;
Li et al., 2019a; 2017; Sung et al., 2018;
Vinyals et al., 2016).
Most of few are implemented based on from greatly depends on the visibility and cleanliness of the target in the images.
It is easy to misclassify when the is small, inconspicuous, or the is in a similar background, or there is a similar Interference around the target, see in Figure
Regarding to this problem
Yue (Yue et al., 2020) propose a causal intervention mechanism to the et al
, 2019) proposes a cross attention network to most et al to help appearance
can
However method incapable
Previous works have demonstrated that leveraging multiple in few have great potential improvement (
However, what is missing from the current multi-modal approaches is that the semantic features is only available in the support set but not in query set.
Consequently, separating visually similar objects of different classes in the query set, which are in close vicinities of each other in the visual feature space, remains a fairly difﬁcult task.
To solve this problem, we propose a task-adaptive semantic feature learning mechanism, to predict semantic features for the query samples which is helpful in distinguishing samples with similar background and interference.
The semantic features are the word embeddings of the text labels, which are predicted by the Glove word embedding method.
The task adaptive semantic feature learner is trained within each task with the support samples as the supervision.
Then, the trained semantic feature learner is used to predict semantic features for the query samples, which can guide the model to pay more attention to the object of the corresponding class.
The semantic feature learner is customized to respective task such that the semantic features are most discriminative for a given task.
The contributions of this work are three folds.
(1) We propose a task-adaptive semantic feature learner to predict semantic features for the query samples, which improves the discrimination of the query features.
The visual and semantic features are learned separately without collapse into a common feature space.
which maintains speciﬁc information of different modalities.
(3) we construct some hard tasks with samples having similar background and interference, to prove the efﬁcacy of TasNet in generating discriminative features from hard samples.
(4) We develop two modality combination implementations, feature concatenation and feature fusion (feature weighted sum), both results show the effectiveness of the semantic feature leaner.
R ELATED W ORK 2. F EW S HOT LEARNING
Few-shot classiﬁcation methods exhibit great diversity.
These methods can be roughly divided into two main groups: metric-based and gradient-based approaches.
Metric-based approaches aim at learning discriminative representations that minimize intra-class distances while maximizing the inter-class distances with a non-parametric metric.
Representative works include Siamese Network (Koch et al., 2015), Matching Network (Vinyals et al., 2016) and Prototypical Network (Snell et al., 2017).
On this basis, Relation network (Sung et al., 2018) proposed a learnable deep distance metric, to construct the relation of samples within a task.
The unlabeled samples are classiﬁed according to the metric scores.
Gradient based methods focus on training models that can generalize well to new tasks with only a few ﬁne-tuning updates.
MTL (Sun et al., 2019) approach leverages transfer learning and beneﬁts from referencing neuron knowledge in pre-trained deep nets.
MAML (Finn et al., 2017) aims to learn a good parameter initialization that enables the model easy to ﬁne-tune.
2. M ULTI - MODALITY FEW SHOT LEARNING
Recently, some researches concerning multi-modality have been proposed, such as Adaptive Modality Mixture Mechanism (AM3) (Xing et al., 2019), Multiple-Semantics (Schwartz et al., 2019), taskrelevant additive margin loss (TRAML) (Li et al., 2020) and
Semantic Feature Augmentation (Chen et al., 2019), to leverage extra information of the samples to improve the classiﬁcation performance.
In (Xing et al., 2019), Xing et al.
observed that some visually-similar images of different classes are distant from each other in the semantic feature space.
Thus, they proposed an adaptive feature fusion mechanism to combine features of the visual and semantic modalities to reﬁne the category prototypes of the support set.
Similarly, E. Schwartz et al.
(Schwartz et al., 2019) proposed a improved version of AM3 by incorporating multiple and richer semantics information (category labels, attributes, and natural language descriptions) to the image visual features, which could result in further performance improvements.
This idea came from mimicking the learning process of a human baby.
TRAML (Li et al., 2020) proposed an adaptive margin principle to improve the generalization ability of metric-based meta-learning approaches.
The adaptive margins are deﬁned as the semantic similarity between different categories.
TriNet (Chen et al., 2019) solves the few shot classiﬁcation problem via feature augmentation in the semantic feature space.
It ﬁrst embedded images into a latent semantic space.
And then, it augmented semantic features with semantic gaussian and semantic neighborhood approaches.
After which, the semantic features are transformed back to the image visual features space, which enriches useful information for the limited samples.
A common limitation of these methods is that while additional modalities are used in support samples, there is still a single modality used in the query samples.
Our method solves this problem by incorporating visual and semantic features in both support and query samples.
M ETHODOLOGY
We incorporate data of multiple modalities as an attempt to extract additional semantic information from samples .
The key challenge, however, is that labels of the query set are obstructed from the classiﬁer.
Instead of abandoning the additional modality all together in classifying samples in the query set, we choose to re-construct the missing modality from available information.
This requires a translator that predicts values of the missing modality.
In the realm of image classiﬁcation, we train a task-speciﬁc image-to-semantic translator, termed the semantic feature learner, from the support set, which is then applied to the query set to generate the semantic features for each image.
Since both the visual and the semantic modalities are now present in the query set, we combine these two features via concatenation or fusion operation to produce ﬁnal feature representations for each sample.
In our method, the semantic feature learner is trained separately from the visual feature learner.
The key idea of isolating the two training processes is that while visual features might be shared across all categories and many tasks, semantic features, as a supplementing feature set, is used to help distinguish instances of visually similar categories within the current task.
Hence the visual feature learner is trained globally and the semantics feature learner is trained locally per each task.
Depending on the task setup, the semantic feature learner might focus on different features in images.
For example, the semantic feature learner might put different emphases in egg images to help separate eggs from pingpong balls (in one speciﬁc task), then from separating eggs from carrots (in another speciﬁc task).
By limiting the training process of the semantic feature learner to each task, our method allows greater ﬂexibilities adapting to each task and increases the efﬁcacy of using the semantic features to further distinguish visually similar objects in a task.
Overall, our method trains a global visual feature learner ( f ϕ ) across all tasks during the metatraining phase, while trains a task-speciﬁc semantic feature learner ( g ψ ) within each task separately.
In support set, we compute the visual-feature class centers (visual prototypes) and combines them with label-derived semantic features into ﬁnal prototype feature vectors p (class centers).
In query set, we generate both the visual features and the semantic features from f ϕ and g ψ , respectively, which are then combined into a ﬁnal feature representation q of the query sample.
Finally, the classiﬁcation of query samples is implemented based on the distance to prototypes associated to each class.
3. P ROBLEM D EFINITION
In Few Shot Learning setting, There are three datasets: D b , D v and D n denoting train (base), validation and test (novel) set respectively, with categories disjoint from each other, i.e. D b ∩ D v = ∅ , which is different from traditional supervised learning.
The goal of FSL is to classify samples from unseen classes with few supervised samples.
In general, the few shot learning approaches are based on episodic training paradigm.
The model is trained across a number of episodes T sampled from
D b .
A N -way K S -shot task (or episode) is denoted as T i = {S ; Q} , containing N categories.
Where
S denotes the support set consisting of K S samples per category, i.e. S =
( x i , y i )
N × K S
i
= ; and Q denotes the query set consisting of K Q samples per category, i.e. Q =
( x i , y i )
N × K Q
i = .
Finally, the performance of the model is evaluated on multiple tasks sampled from the novel dataset D n .
3. A RCHITECTURE OF THE PROPOSED MODEL
Figure 2 illustrates the overall framework of the proposed method.
It consists of a pre-training stage, a meta-training stage and a meta-testing stage.
Firstly, we pre-train a feature extractor E θ with C ( ·|D b ) as the classiﬁer from scratch by minimizing a standard cross entropy loss with the base dataset D b as the training data.
After pre-training stage, the parameters of the feature extractor are frozen, which is indicated with light blue color.
In the meta-training stage, images are ﬁrst fed into the feature extractor
E θ to produce feature vectors.
Then, the feature vectors are used as inputs for the visual feature learner f ϕ and the semantic feature learner g ψ .
Meanwhile, the support images and the corresponding class text labels ( I S , W S ) are feed into F ( ·|S ) module to train the task-speciﬁc semantic feature leaner.
The trained semantic feature learner g ψ is represented in light yellow color, denoting the parameters are frozen.
Both the ﬁnal prototypes and the ﬁnal feature representations of the query samples are generated by F C/F module.
The key difference between the support feature representations and the query feature representations is that: in the support set, semantic features are generated from the class text labels, while in the query set, semantic features are predicted with g ψ from image feature vectors.
Finally, the visual feature learner f ϕ is optimized via minimizing the cross-entropy loss between the ground truth labels and the predicted probability vectors.
The meta-testing stage is similar with the meta-training stage, except that the parameters of the visual feature learner f ϕ can be no longer updated, which is represented in light green color.
3. T ASK -A DAPTIVE S EMANTIC F EATURE L EARNING
The feature vectors of the support images and the ground truth semantic feature embeddings (it is predicted by Glove word embedding of the class text labels) are paired up as supervision to train the semantic feature learner g ψ .
The loss function is formulated as the dissimilarity between the predicted semantic feature embeddings and the ground-truth semantic feature embeddings.
See in
Eq. 1.
[Equation]
where (cid:102)
w c
i denotes the predicted semantic feature embedding of the i -th sample belonging to class c , and w c denotes the ground truth semantic feature embedding of class c .
We train the semantic feature learner within each task by minimizing L dissim to produce task-speciﬁc parameters.
3. F EATURE GENERATION 3.4. P ROTOTYPES GENERATION OF THE SUPPORT SAMPLES
Firstly, we extract feature vectors for both support and query samples, as formulated in Eq. 2.
[Equation]
where x i denotes an image sample, and a i is the feature vector extracted from this image.
Then we transform the feature vectors to the visual feature space via the visual feature learner f ϕ : v = f ϕ ( a ) .
And we compute the visual prototypes with Eq.
[Equation]
where S c denotes a subset of S , which contains samples of class c .
Finally, we combine the visual prototypes with the ground truth semantic feature embeddings via concatenation or fusion (weighted sum) operation to obtain the ﬁnal prototypes of the support set.
It is formulated as Eq.
4 and Eq. 5, respectively.
[Equation]
where C ( · , · ) denotes the concatenation operation of the two vectors, w c is the ground truth semantic feature embedding of class c .
3.4. F EATURE GENERATION OF QUERY SAMPLES
The feature representations of the query samples are produced in a similar way as support samples, except that the semantic features of the query samples are learned from images.
The ﬁnal query feature representations are generated via fusion approach formulated as Eq.
6 or concatenation approach formulated as Eq.
[Equation]
7) where a i is the feature vector of i -th query image sample.
do
20 end
Input: W = (cid:8) ( w c ; w (cid:48) c
i ) | c
= 1 , . .
, N, i = 1 , . . .
K S (cid:9) .
( w c is the ground truth semantic features and w (cid:48)
c
i = g ψ ( a c i ) is the predicted semantic features.)
Output: semantic feature learner g ψ 1 while not convergence do
Evaluate L dissim ( ψ ) by Eq. 1; gradient descent ∇L dissim ;
C LASSIFICATION
LOSS OF THE VISUAL FEATURE LEARNER
The negative Euclidean distance between the feature representation q i of i -th query and the prototype p c of the c -th support sample is employed as the predicted scores of the query samples, and the distance is transformed into a probability via a Softmax operation, see Eq. 8.
[Equation]
where ˆ y i denotes the predicted label for i -th sample in the query set.
Then, the loss function is formulated as the negative loglikelihood of the predicted probability scored of each query sample.
[Equation]
We compute the cross-entropy loss between the predicted scores of the query samples and the ground truth labels for all training episodes constructed from the base dataset, see Eq. 9.
Parameters of the visual feature learner is optimized by minimizing L CE .
We summarize the whole process in
Algorithm 1 and Algorithm 2.
[Table]
E XPERIMENTS
We conduct a comparison of our method to state-of-the-art methods in terms of few-shot classiﬁcation accuracy on four benchmarks, including MiniImageNet (Vinyals et al., 2016), CUB-200- (Wah et al., 2011), CIFAR-FS (Bertinetto et al., 2018) and FC100 (Oreshkin et al., 2018).
4. A RCHITECTIRES
For the feature extractor network
[Table]
we use a ResNet-12 consisting of 4 residual blocks with Dropblock as a regularizer and 640-dimensional output features, the details follow the one proposed in TADAM (Oreshkin et al., 2018).
For the visual feature learner and the semantic feature learner, we use a MLP with one layer, outputting 300-dimentional features.
4. R ESULTS AND ANALYSIS
Table 1 and Table 2 summarize the results of the few-shot classiﬁcation tasks on MiniImageNet,
CIFAR-FS, FC100, and CUB-200-2011, respectively.
It is worth noting that our TasNet achieves the state-of-the-art performance for both 1-shot and 5-shot settings on almost all four benchmarks.
4. A BLATION STUDY
To further validate the effectiveness of our method, we conduct a series of ablation studies on miniImageNet and CIFAR-FS datasets.
We ﬁrst show some experimental proofs of the task-adaptive semantic feature learner.
Then we show some other results on the hyperparameters.
4.3. E FFECTIVENESS OF THE SEMANTIC FEATURE LEARNER
The task-adaptive semantic feature learner is designed to alleviate the negative effects of similar background or similar interference, and provide another kind of modality to enrich the information of query samples, which is helpful in producing discriminative feature embeddings.
It can also alleviate the few supervision problem to a certain extent.
We test the model with and without semantic feature learner.
The quantitative and qualitative results are shown in Figure 3(a) and Figure 4.
[Table]
As shown in Figure 3 (a)
[Table]
the classiﬁcation accuracy has achieved a reasonable value in the case of only semantic features.
In the 1-shot case, the contribution of the semantic feature is especially signiﬁcant, which proves the effectiveness of the task-adaptive semantic feature learner.
We can also observe that, the combination of the semantic and visual features resulting in further performance improvement, which demonstrates that the semantic features and the visual features can complement each other to improve the classiﬁcation problem together.
Moreover, performance of the concatenation approach of features is ahead of the fusion approach.
4.3. T-SNE V ISUALIZATION
To qualitatively validate the effectiveness of our method, we further provide some t-SNE visualization results in Figure 4.
We sample one episode in the test split of miniImageNet and CIFAR-FS datasets under the 5-way 1-shot and 5-way 5-shot settings.
And then, we obtain the embeddings of all images using the visual feature learner, the semantic feature learner, and our TasNet, respectively.
Across four subﬁgures, samples with the same color belong to the same class.
Evident in these ﬁgures, the overlap between different classes of embeddings generated by TasNet is much less than that with only visual features, which demonstrates the effectiveness of the task-adaptive semantic feature learner in helping learning feature representations with well separated margin between novel classes.
From the last two columns of Figure 4 (a)(b), we can further observe that compared with the visualization of the fused feature embeddings, the discriminative performance of the concatenated feature embeddings is a little superior than the former, which is in consistent with the quantitative result in Figure 3(a) .
In Figure 3(b), we vary the value of fusion weight λ from 0 to 1 to observe the performance change.
This hyperparameter controls the contribution of the semantic and visual features to the ﬁnal fused features.
As shown in Figure 3(b), we can observe that, the optimal λ for MiniImageNet in 1shot and 5-shot cases are 0.2 and 0.3, respectively, while for the CIFAR-FS dataset, the optimal λ reaches at 0.1 for both 1-shot and 5-shot cases.
It demonstrates that, in order to improve the ﬁnal classiﬁcation performance, we should weigh more on the semantic features, while information from the visual side should be relatively low.
Since too much attention paid to the visual feature will weak the advantage brought by the semantic features.
It is in consistent with the results of the ablation studies in the former sections.
This indicates that the selection of λ plays a key role for best complementing the features information of different modalities.
From the quantitative and qualitative results, we can conclude that the task-adaptive semantic feature learning mechanism is the main reason for the performance boosting.
C ONCLUSION
In this paper, we propose TasNet, a task adaptive semantic feature learning mechanism to incorporate semantic features for the query samples.
Since the feature extractor has been initialized after the pre-training stage, we freeze the parameters of the feature extractor in the meta-training and metatesting stages.
Therefore, only the semantic feature learner and the visual feature learner should be optimized, which is less time consuming.
Our model trains the global classiﬁcation loss and the task-adaptive semantic feature dissimilarity loss separately, which can avoid collapsing different modalities into a common feature space, so as to preserve the structural heterogeneity of different modalities.
Then, we propose two features combination approaches: feature concatenation and feature fusion, to further improve the performance.
Experiments show that our method outperforms the state-of-the-arts on four benchmarks.
In the future work, we expect to explore semantic-guided attention mechanism to pay more attention to the targets in the images, so as to better capture classspeciﬁc information from the images, which can further alleviate the inﬂuence of noise and irrelevant information.
A A PPENDIX A. S ETTINGS
We train the feature extractor E θ with the SGD optimizer using an initial learning rate of 10 − 1 .
Adam (Kingma & Ba, 2014) is used to optimize the parameters of the visual and semantic feature learners with an initial learning rate of 10 − 3 .
During few-shot learning stage, we train f ϕ and g ψ by adopting the episodic training procedure with N-way K-shot training tasks as described in (Snell et al., 2017).
The model is trained for 100 epochs, with each epoch consisting of 600 randomly sampled episodes for both 1-shot and 5-shot cases on MiniImageNet, CUB-200-2011, CIFAR-FS and FC100 datasets and is evaluated by averaging metrics over 600 randomly generated episodes from D n , with each episode having 15 randomly sampled query samples.
Parameters of the model except the backbone layers are trained on 5-way, 15 query samples for both 1-shot and 5-shot cases per episode
A. E XPERIMENTS ON HARD TASKS
To further validate the effectiveness of TasNet in discriminating different targets from similar background and interference, we conduct an additional experiment on hard tasks, which are constructed of samples with similar background and interference.
Firstly, we have constructed a sub dataset (we name it hard dataset) from the test split of miniImagenet dataset manually.
It contains 8 classes, including lion, golden retriever, goose, coral reef, vase, spider web, ant, house ﬁnch.
Among samples from all these categories, some of samples from two or more categories have similar background or similar interference, for example, vase and ant categories, ﬂowers are the common interference in the images from these two classes, and for the bird and spider web categories, trees and grassland are common similar backgrounds in the images from these two classes.
Then we construct 20 tasks from this hard dataset, and compare our TasNet with ProtoNet and AM3 methods on this dataset.
AM3 is a method that combine semantic features for the support samples, while only visual features are extracted from the query samples.
The mean accuracy is show in table 3.
[Table]
In addition to the statistical results, we show the classiﬁcation results of one hard task in Figure 5, which is carried on 5-way 1-shot case.
Five categories of ant, house ﬁnch, spider web, vase, coral reef, are included in this task.
Each class has 1 support samples and 15 query samples, that is the number of test samples is 5 × 15=75.
From Figure 5, we can see that there are 33, 36 and 39 samples correctly classiﬁed with the ProtoNet,
AM3 and our TasNet methods.
A. A NALYSIS OF THE ABLATION STUDY RESULTS
From the quantitative and qualitative results in section 4.3.1 and section 4.3.2, we can conclude that the task-adaptive semantic feature learning mechanism is the main reason for the performance boosting. To make the ablation study results more obvious, we put the accuracy in terms of four kinds of different features in table 4. [Table]
The accuracy computed with the semantic features is shown in the second row of table 4 (in red color).
When only the semantic features are used in the model, it has achieved a signiﬁcant accuracy, which demonstrates that the semantic features for the query samples can certainly work.
A. T RAINING THE SEMANTIC AND VISUAL FEATURE LEARNER JOINTLY
In order to prove that training the visual feature learner across all tasks and training the semantic feature learner within each task is an effective approach, we have made some experiments to jointly train the visual and semantic features learners.
The detailed implementation of the jointly training method is described as follow.
We constructed a cross entropy loss to optimize the visual feature learner, and a word embedding ﬁtting loss, i.e. the cosine dissimilarity loss to optimize the semantic feature learner.
The loss L CE for training visual feature learner is deﬁned as a cross-entropy classiﬁcation loss averaged across all query-support pairs.
It compares the Euclidean distance between the query features and the class mean of the support features to get the predicted score.
[Equation]
where ˆ y i denotes the predicted label for i -th sample in the query set, and y i is the ground truth label of sample q
i .Then, the loss function is formulated as the negative loglikelihood of the predicted probability scored of each query sample.
[Equation]
The word embedding ﬁtting loss L dissim ( ψ ) is deﬁned as dissimilarity between the predicted semantic features and the ground-truth semantic features.
[Equation]
where (cid:102) w c
i denotes the predicted semantic feature of the i -th sample belonging to class c , and w c denotes the ground truth semantic feature of class c.
We train the semantic feature learner within each task to produce task-speciﬁc parameters.
Parameters of semantic feature learner are optimized by minimizing L dissim ( ψ ) .
Finally, we combine the cross-entropy classiﬁcation loss and the word embedding ﬁtting loss via weighted sum, λ =0.5, and train across all tasks to minimize the total loss
L tol .
[Equation]
[Table]
The experiment results show that the jointly optimization method of the visual and semantic feature learners can not achieve good performance
[Table]