R EVISITING F EATURE A CQUISITION B IAS FOR F EWS HOT F INE -G RAINED I MAGE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work on metric-learning based few-shot fine-grained image classification (FSFGIC) has achieved promising success in classification accuracy, where various convolutional neural networks with different similarity measures are utilized to learn a common feature representation for each category for FSFGIC.
In this paper, we identify and analyze for the first time a fundamental problem of existing metric-learning based FSFGIC methods which fail to effectively address the bias in feature information obtained from each input image that causes misclassification.
To solve this problem, we present a robust feature acquisition network (RFANet) that has the ability to effectively address the bias in the feature information obtained from each input image and guide convolution-based embedding models to significantly increase the accuracy.
Our proposed architecture can be easily embedded into any episodic training mechanisms for end-to-end training from scratch.
Extensive experiments on FSFGIC tasks demonstrate the superiority of the proposed method over the state-of-the-arts.
I NTRODUCTION
Learning from very limited training data has attracted increasing attention due to the high cost of data collection and annotation.
In recent years, a variety of few-shot fine-grained image classification (FSFGIC) methods (Vinyals et al., 2016;
He et al., 2016) have been presented for classifying similar images with very limited data.
A basic and straightforward technique is to only leverage feature information from only few data for image classification.
Take the k -nearest neighbors algorithm
(Fix & Hodges, 1989) as an example, the label of a query image can be predicted based on the similarity measure between the query image and a few examples without any learning mechanism.
However, it is nearly impossible for the k -nearest neighbors algorithm to learn a real concept with diverse and complex feature representations from just very limited examples (Li et al., 2019b).
Another widely used technique (Koch et al., 2015; Snell et al., 2017) is to utilize different transfer learning mechanisms to develop transferable feature representations based on additional auxiliary datasets, which can be roughly classified into two groups: meta-learning based methods (Xu et al., 2021;
Tsutsui et al., 2022) and metric-learning based methods (Huang et al., 2020; Liao et al., 2022;
Huang et al., 2022).
Meta-learning based methods aim to train a meta learner with a meta-learning paradigm or a learning-to-learning paradigm for FSFGIC.
Metric-learning based methods aim to learn a transferable model based on similarity metrics between different categories.
In this work, we address a fundamental problem in the metric-learning.
Generally, the metriclearning based FSFGIC methods include two main components: a feature embedding network and a similarity metric learning network.
One of the main issues of the metric-learning methods is how to employ a convolution neural network (CNN) (Vinyals et al., 2016;
He et al., 2016) with different similarity measures (e.g., nearest neighbor metric (Snell et al., 2017), hyperbolic distance (Khrulkov et al., 2020), cosine metric (Li et al., 2019a), or learned parametric options (Sung et al., 2018)) to learn an effective feature representation for each category.
Although the existing metric-learning based FSFGIC methods have achieved some degree of success, the following questions remain open: (1) Have we properly considered how to robustly acquire feature information from each
input
image? (2) Have we properly considered how to effectively address the bias
in the feature informationobtained
from each input image due to image quantization noise 2 caused
by
data augmentation (DA
operations on images (e.g.,
image affine
transformations
random cropping
, or color jittering opera tions and (3) Have we considered how to properly guide the CNN to acquire feature information from each input image?
Take the deep nearest neighbor neural network (DN4) (Li et al., 2019a) as an example as shown in Fig.
1, the five support images belong to five different categories, the support image s 5 and query image q belong to the same category, and DA operations with random cropping, random horizontal flipping, and color jittering are randomly performed on the six images.
DAs are carried out once on the five support images and DAs are carried out three times on the query image for obtaining three new query images (i.e., q 1 , q 2 , and q 3 ).
Under the condition that all parameter settings remain unchanged, the five support images and the three query images after
DA are sent into the trained DN4 model, and their corresponding similarities are shown in Fig.
Although the objects of the same category have the same feature properties, it can be seen from the cosine similarities in Fig.
1 that only query image q 2 and support image s 5 can be correctly classified into one category in these three classification instances.
It can also be seen from Fig.that the similarities between a query image and different support images from different categories are very close.
The reasons are: (1) There is a bias in obtaining image feature information from a given CNN; (2) DA operation increases the quantization noise of images to a certain degree; and (3)
Fine-grained images have small inter-class and large intra-class variations.
DA
[Table]
s [Table] s
[Table]
DA 0.
Scores 0.
4 s
DN 1 q 2 q ( ) 1 1 cos , q s ( ) 1 2 cos , q s ( ) 1 3 cos , q s ( ) 1 4 cos , q s ( ) 1 5 cos , q s ( ) 2 1 cos , q s ( ) 2 2 cos , q s ( ) 2 3 cos , q s ( ) 2 4 cos , q s ( ) 2 5 cos , q s ( ) 3 1 cos , q s ( ) 3 2 cos , q s ( ) 3 3 cos , q s ( ) 3 4 cos , q s ( ) 3 5 cos , q s 3 q q
To address the aforementioned problems, this paper designs a novel and effective architecture for robust feature acquisition.
The contributions of the paper can be summarized as follows: (1) We identify and analyze for the first time that existing metric-learning based FSFGIC algorithms are prone to the bias in the feature information obtained from each input image due to image quantization noise; (2) A novel and robust feature acquisition network (RFANet) is proposed to address this feature representation bias; and (3)
The proposed RFANet has the capability to guide any convolution-based embedding models to obtain robust image feature representation for FSFGIC.
It is worth to note that our designed architecture can be easily embedded into any episodic training mechanisms for end-to-end training from scratch.
Experiments on five fine-gained image classification benchmark datasets, i.e., Stanford Dogs (Khosla et al., 2011), Stanford Cars (Krause et al., 2013), CUB (Wah et al., 2011), NABirds (Van Horn et al., 2015), and Plant Disease (Sharma, 2018), demonstrate that the proposed RFANet significantly outperforms the baseline methods on both 5-way 1-shot and 5way 5-shot FSFGIC tasks.
Furthermore, extensive ablation studies on the proposed method have also been conducted.
The results validate the effectiveness of our proposed RFANet architecture with very encouraging large performance boostings.
R ELATED WORKS
Existing FSFGIC methods can be roughly classified into two main streams: meta-learning based methods and metric-learning based methods.
In the following, representative FSFGIC methods and some representative few-shot learning methods which have been used for performing the finegrained image classification tasks are briefly reviewed.
2. M ETA - LEARNING BASED FSFGIC METHODS
The purpose of meta-learning based methods is to learn a good parameter initialization for FSFGIC.
Wei et al.
(2019) proposed the first FSFGIC model by utilizing two components.
The first component is to obtain subtle image feature information and the second component is to learn the decision boundaries of input datasets.
Zhu et al.
(2020) presented a multi-attention meta-learning (MattML) algorithm which employed multi-attention on the base learner and the task learner for localizing the discriminate parts in images.
Tang et al.
(2020) explored the gap between machine and humans for fine-grained image classification and indicated that humans usually utilize stable feature representations which are robust to spatial deformations for FSFGIC.
Then a pose-normalization feature representation network is designed for FSFGIC.
Xu et al.
(2021) argued that an isotropic
Gaussian distribution can be used for modelling the intra-class variability.
Then, feature information is repeatedly sampled from the learned intra-class variability distribution and added to the class-discriminative feature information for obtaining augmented feature information and performing FSFGIC.
Wang et al.
(2021) introduced a foreground object transformation (FOT) technique for removing image background.
Then a posture transformation generator is presented to generate additional samples for increasing the diversity of data samples.
2. M ETRIC - LEARNING BASED FSFGIC M ETHODS
The purpose of metric-learning based methods is to learn a good task independent embedding for
FSFGIC.
Currently, many discriminative feature learning architectures (Huang et al., 2019; 2020; Li et al., 2019a) have been presented for FSFGIC.
Huang et al.
(2020) presented a low-rank pairwise alignment bilinear network (LRPABN) to obtain feature relationships between different fine-grained categories for FSFGIC.
To learn more discriminative features, Li et al. (2019a) proposed DN4 which designs an image-to-class similarity measure for image classification with very limited samples.
To reduce similarity bias, Li et al.
(2020) proposed a bi-similarity network (BSNet) which applies the sum of the two different similarity measures (i.e., the Euclidean distance and the cosine distance) for FSFGIC.
Huang et al.
(2022) presented a target-oriented alignment network (TOAN) which uses utilizes feature alignment for effectively reducing the intra-class variation in fine-grained images.
Dong et al.
(2020) presented an adaptive task-aware local representations network (ATL-Net) which aims to localize distinguished local patches in the entire task.
Sun et al.
(2020) utilized salient patch localization (Selvaraju et al., 2017) and high-order integration for obtaining discriminative feature representations for FSFGIC.
Liao et al.
(2022) presented an automatic salient region selection architecture for FSFGIC.
Meanwhile, feature distributions with different categories have been explored for FSFGIC.
Li et al.
(2019b) introduced a covariance metric networks (CovaMNet) which employs the feature distribution consistency for performing FSFGIC.
Yang et al.
(2021) argued that feature representations from each category follow a Gaussian distribution.
Then a distribution calibration (DC) strategy is designed to make the distributions of feature representations more Gaussian-like.
It is worth to note that the DC strategy (Yang et al., 2021) can also be implemented in meta-learning based FSFGIC.
Furthermore, feature reconstruction techniques (Zhang et al., 2020; Wertheimer et al., 2021; Li &
Bian, 2022) have been employed for FSFGIC.
Zhang et al.
(2020) used the earth mover’s distance (EMD) technique (Rubner et al., 2000) and proposed DeepEMD which formulates feature reconstruction as an optimal transportation problem for finding optimal matching between a query image and support images.
Wertheimer et al.
(2021) introduced a feature map reconstruction network (FRN) in which a query feature is reconstructed directly from support features via ridge regression in a closed form.
Li & Bian (2022) argued that it is inaccurate in FRN (Wertheimer et al., 2021) that each descriptor contributes equally to the reconstruction error and proposed a foreground-aware
FRN (FAFRN) which uses the weighted squared Euclidean distance as the reconstruction error for
FSFGIC.
Within the scope of our investigation, all the aforementioned meta-learning and metric-learning based FSFGIC methods utilize DA techniques (e.g., random cropping, random horizontal flipping, and color jittering) for enhancing the diversity of data samples, reducing overfitting, and improving classification accuracy.
It is worth to note that DA operations on image inevitably lead to the generation of quantization noise.
The current FSFGIC methods do not address the bias in feature information obtained from each input image which may cause biased feature representations and misclassification.
How to effectively acquire feature information and suppress the interference of quantization noise from images has constrained the development of FSFGIC.
M ETHODOLOGY 3. P ROBLEM S TATEMENT
A typical few-shot image classification setting contains a support set S and a query set Q .
Support set S contains C different image classes and each class in C is composed of K labeled samples.
Query set Q is composed of unlabeled samples.
Set S and set Q have the same data-label space.
The goal of FSFGIC is to train a model which has the ability to successfully classify each query sample q ( q ∈ Q ) into its corresponding class in C .
Thus, the FSFGIC task is called a C -way K -shot task.
3. M OTIVATION OF THE P ROPOSED M ETHOD
This work is largely inspired by the experimental observation shown in Fig.
Although as described in (Krizhevsky et al., 2012) that DA is one of “the two primary ways in which we combat overfitting”, DA operations on each image inevitably lead to the generation of quantization noise, which results in bias in feature information obtained from each input image.
The bias of feature representation has a great impact on the performance of FSFGIC.
The main reasons are as follows: (1) Fine-grained images have small inter-class and large intra-class variations.
The training samples of FSFGIC are very limited.
They present us with a dilemma.
On the one hand, it is necessary to perform DA operations on each image for ensuring the diversity of data images and reducing overfitting.
On the other hand, it is necessary to effectively address bias in the feature information obtained from each input image which is caused by the data augmentation (DA) operations on images.
In general, the attributes of the features obtained by a convolutional embedding module from different DA operations on the same image have better consistency than the features obtained from different images under different DA operations.
The above analysis motivates us to propose a robust feature acquisition (RFA) mechanism for FSFGIC as follows: DA operations (i.e., random cropping, random horizontal flipping, and color jittering) are randomly performed on a support image and a query image.
DA is performed once on the support image and DA is performed n times on the query image for obtaining n augmented query images.
The augmented support and query images are sent into a given convolutional embedding module for obtaining feature descriptors.
Then the regularized feature descriptors of the augmented support image and the regularized feature descriptors of the n augmented query images are employed for similarity calculation.
Finally, the n th root of the product of these n similarities is used as the similarity measure between the query image and the support image.
Under the condition that the DA operation is utilized to achieve diversity on training samples, the proposed RFA mechanism addresses the intrinsic bias problem caused by the image quantization noise.
3. T HE P ROPOSED RFAN ET
The overview of the proposed RFANet framework for FSFGIC is illustrated in Fig.
A novel robust feature acquisition strategy is designed for effectively addressing the interference of quantization noise on feature descriptors under the condition of achieving the diversity of training samples.
Pre- dominantly, the proposed RFANet architecture can be trained in an end-to-end manner from scratch.
In the following, the proposed RFANet framework will be presented in detail.
[Figure]
Query Image q
DA
CNN
Feature Descriptors
Similarity Measure ( ) 1 1 cos ( ), ( ) n n i i =    q
X 1 q 2 q n q
L 1 ( )  q 1 ( )  X 2 ( )  X 3 ( )  X 4 ( ) 
X 5 ( )  X 2 ( )  q ( ) n  q
L ( ) 2 1 cos ( ), ( ) n n i i =    q
X ( ) 3 1 cos ( ), ( ) n n i i =    q
X ( ) 4 1 cos ( ), ( ) n n i i =    q
X ( ) 5 1 cos ( ), ( ) n n i i =    q
X
3.3. R OBUST F EATURE A CQUISITION S TRATEGY
In this work, the most commonly used networks (e.g., Conv64F (Vinyals et al., 2016) and
ResNet256 (He et al., 2016)) in the current FSFGIC methods (Li et al., 2019b;a; Dong et al., 2020;
Huang et al., 2022) are selected as the base neural networks.
For each query sample q and support sample s , DA operations with random cropping, random horizontal flipping, and color jittering are performed once on the support sample and n times on the query sample for obtaining augmented query images (i.e., q 1 , q 2 , · · · , q n ).
The augmented support and query images are sent into a given convolutional embedding module for obtaining feature descriptors which can be represented as Θ( q 1 ) =
[ (cid:101) q 1 , 1 , ..., (cid:101) q 1 ,j , ..., (cid:101) q 1 ,m ] ∈ R d
×
( h × w ) , Θ( q 2 ) =
[ (cid:101) q 2 , 1 , ..., (cid:101) q 2 ,j , ..., (cid:101) q 2 ,m ] ∈ R d
×
( h × w ) , · · · , Θ( q n ) =
[ (cid:101) q n, 1 , ..., (cid:101) q n,j , ..., (cid:101) q n,m ] ∈ R d × ( h × w ) , and Θ( s ) =
[ (cid:101) s 1 , ..., (cid:101) s j , ..., (cid:101) s m ] ∈ R d × ( h × w ) , where (cid:101) s j is the j -th feature descriptor with a length of d , h and w are the height and the width of the feature tensor map, d is the number of filters, R denotes real space, and m ( m = h × w ) is the total number of feature descriptors for training samples.
In this way, the j -th feature descriptor (cid:98) q j ( j = 1 , ..., m ) of query sample q is represented by (cid:101)
q 1 ,j , (cid:101) q 2 ,j , · · · , and (cid:101) q n,j .
Then standardization operation is performed on feature descriptors (cid:101) q 1 ,j , (cid:101) q 2 ,j , · · · , (cid:101) q n,j , and (cid:101) s j ( j = 1 , ..., m ) as follows
[Equation]
where ℧ ( · ) and ℘ ( · ) denote the mean operation and the variance operation respectively.
For each feature descriptor (cid:98) q j and (cid:101) s j ( j = 1 , ..., m ) , their corresponding similarity measure is
[Equation]
where cos ( · ) denotes the cosine similarity.
In this way, k -nearest neighbors ˇ s tj | kt =1 for feature descriptor (cid:98) q j can be found in each class C based on the k -nearest-neighbor technique (Boiman et al.,
2008), where t denotes the t -th k -nearest neighbor.
Then the image-to-class similarity measure (Li et al., 2019a) is calculated for FSFGIC as follows
[Equation]
Finally, the Adam optimization (Kingma & Ba, 2015) with a cross-entropy loss is utilized to train the whole network for learning the parameters and performing FSFGIC tasks.
In this work, k and n are set to 1 and 3 respectively (See Table 4 and Table 5).
We provide the ablation studies of choosing different values of k and n to explore the sensitivity of the proposed method to the hyper-parameters.
E XPERIMENTS 4. D ATASETS
Five fine-gained datasets (i.e., Stanford Dogs (Khosla et al., 2011), Stanford Cars (Krause et al., 2013), CUB (Wah et al., 2011), NABirds (Van Horn et al., 2015), and Plant Disease (Sharma, 2018)) are used for evaluating the performance of the proposed RFANet.
The Stanford Dogs dataset contains 120 dog classes with 20,580 samples.
The Stanford Cars dataset consists of 196 car classes with 16,185 samples.
The CUB dataset contains 200 bird classes with 11,788 samples.
The NABirds dataset consists of 555 bird classes with 48,562 samples.
The Plant Disease dataset containsplant disease classes with 54,306 samples.
Following the latest data split introduced in (Huang et al., 2022), the same data splits are used in our work which are summarized in Table 1.
[Table]
Plant Disease
4. E XPERIMENTAL S ETUP
We conduct 5-way 1-shot and 5-way 5-shot experiments, which are standard FSFGIC settings, on the aforementioned five datasets.
Each input image is resized to a fixed size of 100 × 100 and randomly cropped into 84 × 84 .
Random cropping, random horizontal flipping, and color jittering are utilized for data augmentation.
In terms of the episodic training paradigm (Vinyals et al., 2016), 300,000 episodes are randomly selected and constructed for training models.
In each episode,and 10 query images are randomly selected from each class for the 1-shot and 5-shot settings respectively.
Meanwhile, the Adam optimization technique (Kingma & Ba, 2015) with an initial learning rate of 0.001 is used for optimizing the designed models.
The learning rate is reduced by half for every 100,000 episodes.
During the testing process, 600 episodes are randomly constructed from the testing set for obtaining the top-1 results on mean classification accuracy.
Meanwhile, the 95% confidence intervals are obtained and reported.
It is worth to note that the proposed RFANet architecture is trained in an end-to-end manner from scratch and without any fine-tuning during testing.
4. P ERFORMANCE C OMPARISON
The proposed RFANet architecture has been compared with fourteen state-of-the-art methods (MatchingNet (Vinyals et al., 2016), Prototypical Net (ProtoNet) (Snell et al., 2017),
RelationNet (Sung et al., 2018), CovaMNet (Li et al., 2019b), PCM (Wei et al., 2019), DN4 (Li et al., 2019a), MattML (Zhu et al., 2020), PABN + cpt (Huang et al., 2020), LRPABN cpt (Huang et al., [Table] , SoSN (Zhang & Koniusz, 2019), BSNet (Li et al., 2020), ATL-Net (Dong et al., 2020),
FOT (Wang et al., 2021), and TOAN (Huang et al., 2022)).
Table 2 and Table 3 show the comparison results on the Stanford Dogs, Stanford Cars, CUB, NABirds, and Plant Disease datasets.
It is observed from Table 2 and Table 3 that our proposed method achieves the best performance on both 5-way 1-shot and 5-way 5-shot FSFGIC tasks.
For the 5-way 1-shot and 5-way 5-shot
FSFGIC tasks on the CUB dataset, our proposed RFANet with the backbone of Conv64F achieves 13 .
76% , 3 . 72% , and 5 .
84% improvements and 17 .
19% , and 7 .
95% improvements over MatchingNet, ProtoNet, RelationNet, CovaMNet, PCM, DN4, MattML, PABN + cpt , LRPABN cpt , SoSN, BSNet, ATLNet, FOT , and TOAN respectively.
It is worth to note that the proposed RFANet architecture only considers how to effectively address the bias in feature information obtained from each input image and does not perform further processing on the extracted features such as attention mechanism feature reconstruction.
The algorithm that can perform a fair performance comparison with the proposed RFANet is DN4.
The differences between the proposed RFANet and the DN4 (Li et al., 2019a) are as follows: DAs are performed three times on the query image in this work for obtaining three augmented query images and the regularized feature descriptors of the three augmented query images and the regularized feature descriptors of the augmented support image are used for similarity calculation.
Then the cubic root of the product of these three similarities is used as the similarity measure between the query image and the support image.
It can be seen from Table 2 and Table 3 that the proposed RFANet achieves far better performance than the DN4.
The reason is as described in (Huang et al., 2022): Different fine-grained categories have very similar shapes and visual appearances; the similarities between any two samples are always high which means that the k -nearest features sorted by DN4 in different support classes are also similar which leads to the performance degradation of DN4 in dealing with
FSFGIC.
By addressing the bias problem, the proposed RFANet has the capability to guide any convolution-based embedding model for robustly obtaining feature representations from images and achieves much better performance than DN4 and the other thirteen state-of-the-art methods. [Table] [Table]
4. A BLATION S TUDIES
Influence of the k -nearest neighbors for FSFGIC.
The performance of our designed RFANet architecture on FSFGIC tasks will be affected by the selection of parameter k .
Additional experiments are conducted for investigating the effect of the parameter k in Equation (3) on performance.
Based on the backbone of Conv64F
[Table]
three different k ( k ∈{ 1 , 3 , 5 } ) are selected for performing 5-way 1shot and 5-way 5-shot FSFGIC tasks on the Stanford Dogs and Standford Cars datasets with the fixed parameter n ( n = 3).
The results are shown in Table 4.
It is observed from Table 4 that when k equals 1, the proposed RFANet achieves the best classification performance.
Therefore, k -nearest neighbors with k =1 is recommended for our designed architecture
[Table]
Influence of n DA operations on the query image for FSFGIC.
The performance of our designed
RFANet architecture on FSFGIC tasks will be affected by the selection of parameter n .
Additional experiments on FSFGIC are conducted for investigating the effect of the parameter n in Equation (2) on the performance.
Based on the backbone of Conv64F, four different n ( n ∈{ 2 , 3 , 4 , 5 } ) are selected for performing 5-way 1-shot and 5-way 5-shot FSFGIC tasks on the Stanford Dogs and
Standford Cars datasets with a fixed parameter k
( k = 1).
The results on the 5-way 1-shot and 5-way 5-shot tasks are shown in Table 5.
It is observed from Table 5 that when n equals 3, the proposed
RFANet achieves the allover best classification performance.
Therefore, DA operations on the query image with n =3 is recommended for our designed architecture.
The loss curves and the accuracy curves of the DN4 and the proposed RFANet on the Stanford
Dogs dataset.
Take the 5-way 1-shot FSFGIC task on the Stanford Dogs dataset as an example, the loss curves of training and validation of the DN4 and the proposed RFANet are shown in Fig. [Table] and the accuracy curves of training and validation of the DN4 and the proposed RFANet are shown in Fig.
It can be seen from Fig.
3 that the proposed RFANet has better convergence with faster convergence speed and less loss than DN4.
Meanwhile, it can be seen from Fig.
4 that the proposed
RFANet has a better accuracy than DN4.
Under review as a conference paper at
[Table]
[Figure] [Figure]
Runtime.
In our implementation, during training for a 5-way 1-shot or 5-way 5-shot FSFGIC task, the average running time for one episode is 0.07 and 0.17 second respectively on a single NVIDIA
V100 GPU and a single Intel Xeon Gold 6150 CPU.
During testing, the average running time for one episode is 0.03 second.
Our approach is efficient in computation for practical applications and can be improved with further parallel implementations.
C ONCLUSION
In this paper [Figure] we identify and analyze for the first time a fundamental problem in existing metriclearning based FSFGIC methods which fail to effectively address the bias in features obtained from each input image that causes misclassification.
Then a simple and effective robust feature acquisition network (RFANet) is proposed for FSFGIC.
The proposed RFANet architecture address the image quantization noise problem and has the capability to guide any convolution-based embedding models for robustly obtaining feature information from images.
Our proposed architecture can be easily embedded into any episodic training mechanisms for end-to-end training from scratch.
We have validated the effectiveness of the proposed RFANet on five benchmark datasets which demonstrate the very encouraging great superiority in performance of the proposed method over state-of-thearts.
Furthermore, the proposed RFA mechanism also has a great potential to be applied in object detection, image segmentation, and many other fields.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
the proposed RFANet for a 5-way 1-shot FSFGIC task on the Stanford Dogs dataset.
Runtime.
In our implementation, during training for a 5-way 1-shot or 5-way 5-shot FSFGIC task, the average running time for one episode is 0.07 and 0.17 second respectively on a single NVIDIA
V100 GPU and a single Intel Xeon Gold 6150 CPU.
During testing, the average running time for one episode is 0.03 second.
Our approach is efficient in computation for practical applications and can be improved with further parallel implementations.
C ONCLUSION
In this paper, we identify and analyze for the first time a fundamental problem in existing metriclearning based FSFGIC methods which fail to effectively address the bias in features obtained from each input image that causes misclassification.
Then a simple and effective robust feature acquisition network (RFANet) is proposed for FSFGIC.
The proposed RFANet architecture address the image quantization noise problem and has the capability to guide any convolution-based embedding models for robustly obtaining feature information from images.
Our proposed architecture can be easily embedded into any episodic training mechanisms for end-to-end training from scratch.
We have validated the effectiveness of the proposed RFANet on five benchmark datasets which demonstrate the very encouraging great superiority in performance of the proposed method over state-of-thearts.
Furthermore, the proposed RFA mechanism also has a great potential to be applied in object detection, image segmentation, and many other fields. Runtime.
In our implementation, during training for a 5-way 1-shot or 5-way 5-shot FSFGIC task, the average running time for one episode is 0.07 and 0.17 second respectively on a single NVIDIA
V100 GPU and a single Intel Xeon Gold 6150 CPU. During testing, the average running time for one episode is 0.03 second. Our approach is efficient in computation for practical applications and can be improved with further parallel implementations.  C ONCLUSION
In this paper, we identify and analyze for the first time a fundamental problem in existing metriclearning based FSFGIC methods which fail to effectively address the bias in feature information obtained from each input image that causes misclassification.
Then a simple and effective robust feature acquisition network (RFANet) is proposed for FSFGIC.
The proposed RFANet architecture address the image quantization noise problem and has the capability to guide any convolution-based embedding models for robustly obtaining feature information from images.
Our proposed architecture can be easily embedded into any episodic training mechanisms for end-to-end training from scratch.
We have validated the effectiveness of the proposed RFANet on five benchmark datasets which demonstrate the very encouraging great superiority in performance of the proposed method over state-of-the-arts. Furthermore, the proposed RFA mechanism also has a great potential to be applied in object detection, image segmentation, and many other fields. A PPENDIX M ORE E XPERIMENTAL R ESULTS Additional experiments are conducted for investigating the effect of different DA operation.
Three different operations (i
[Table]
e., random cropping, random horizontal flipping, and color jittering) are selected individually for performing 5-way 1-shot and 5-way 5-shot FSFGIC tasks respectively on the
Stanford Cars dataset.
Their results are shown in Table 6.
It is observed that the proposed RFANet is more effective when applied on random cropping than on flipping and jittering.
[Table]
The proposed RFANet has also been compared with DN4 on the Stanford Cars dataset for 20-way 1-shot and 20-way 5-shot FSFGIC tasks.
127 classes, 20 classes [Table] and 49 classes of Stanford Cars dataset are randomly selected for training, validation, and testing respectively. Their classification accuracies are summarized in Table 7, consistently demonstrating the superior performance of the proposed method when class number increases
[Table]
[Table]
Based on ResNet12 (He et al., 2016) as the backbone, the proposed RFANet is compared with nine methods (i.e., Baseline (Chen et al., 2019), Baseline++ (Chen et al., 2019), MatchingNet (Vinyals et al., 2016), ProtoNet (Snell et al., 2017), RelationNet (Sung et al., 2018), MetaOptNet (Lee et al., 2019), BSNet (Li et al., 2020), DeepEMD (Zhang et al., 2020), and FRN (Wertheimer et al., 2021)) on the CUB dataset.
Following the same data split as used in (Zhang et al., 2020; Wertheimer et al., 2021), 100 classes, 50 classes, and 50 classes of CUB dataset are randomly selected for training, validation, and testing respectively.
In this way, ResNet12 is employed with the same implementation as in (Zhang et al., 2020;
Wertheimer et al., 2021).
The classification accuracy on 5-way 1-shot and 5-way 5-shot FSFGIC tasks for each method is summarized in Table 8.
It is observed that the proposed RFANet achieves the best performance on 5-way 1-shot and 5-way 5-shot FSFGIC tasks.
[Table]