Decoupling Classiﬁer for Boosting Few-shot Object Detection and Instance Segmentation
Abstract
This paper focus on few-shot object detection (FSOD) and instance segmentation (FSIS), which requires a model to quickly adapt to novel classes with a few labeled instances.
The existing methods severely suffer from bias classiﬁcation because of the missing label issue which naturally exists in an instance-level fewshot scenario and is ﬁrst formally proposed by us.
Our analysis suggests that the standard classiﬁcation head of most FSOD or FSIS models needs to be decoupled to mitigate the bias classiﬁcation.
Therefore, we propose an embarrassingly simple but effective method that decouples the standard classiﬁer into two heads.
Then, these two individual heads are capable of independently addressing clear positive samples and noisy negative samples which are caused by the missing label.
In this way, the model can effectively learn novel classes while mitigating the effects of noisy negative samples.
Without bells and whistles, our model without any additional computation cost and parameters consistently outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for FSOD and FSIS tasks.
Introduction
Fully supervised deep convolutional neural network have achieved remarkable progress on various computer vision tasks, such as image classiﬁcation [15], object detection [12, 3, 29], semantic segmentation [22, 2] and instance segmentation [14] in recent years.
However, the superior performance heavily depends on a large amount of annotated images.
In contrast, humans can quickly learn novel concepts from a few training examples.
To this end, a few-shot learning paradigm [8] is presented, and its goal aims to adapt novel classes when only providing a few labeled examples (instances).
Unfortunately, existing few-shot models are still far behind humans, especially for few-shot object detection (FSOD) and few-shot instance segmentation (FSIS).
Various methods have been proposed to tackle the problem of the FSOD and FSIS.
The earlier works [34, 39] mainly follow meta-learning paradigm [9] to acquire task-level knowledge on base classes and generalize better to novel classes.
However, these methods usually suffer from a complicated training process (episodic training) and data organization (support query pair).
The recent transfer-learning (ﬁne-tuning) methods [33, 35, 31, 1, 28] signiﬁcantly outperforms the earlier metalearning ones.
Furthermore, it is more simple and efﬁcient.
These ﬁne-tuning methods mainly use fully supervised object detection or instance segmentation framework, e.g., Faster-RCNN [30] or Mask-RCNN [14].
Therefore, it may be suboptimal for few-shot scenario.
The PASCAL VOC [4] and MS-COCO [21] are widely used to evaluate the performance of object detection or instance segmentation.
Under a fully supervised setting, the model can be well-trained on these two datasets because all interest objects are almost completely labeled.
Under an instancelevel few-shot scenario, however, we ﬁnd that there is a large number of instances that are missing annotations as shown in Fig.
[-1-shot 2-shot 3-shot 5-shot 10-shot
-]1 (detailed discussion in Sec. 3.2).
The reason is that ones consider an instance as a shot for controlling the number of labeled instances when building instance-level
FSOD and FSIS benchmarks.
This is different from image-level few-shot image classiﬁcation [32] because there are generally multiple instances in an image for instance-level few-shot learning.
In fact, missing label (partial label) learning is more difﬁcult and challenging, especially instance-level few-shot scenarios. It requires that learning algorithms deal with training images each associated with multiple instances, among which only partial instances are labeled; while the common supervised learning typically assumes that all interest instances are fully labeled.
In some real-world applications, such as open-vocabulary object detection [40], it is almost impossible to label all instances, and thus there still may exist some instances left to be missing labeled. In addition, it may be more friendly and convenient for users to label partial instances than all ones even in few-shot settings. Most methods [42, 25] have been developed to address missing label (partial label or incomplete label) learning but mainly focus on image-level multi-class or multi-label classiﬁcation. To address the instance-level missing label issue, some recent works have attempted to regard the missing (unlabeled) instances as hard negative samples and re-weight [37] or re-calibrate [41] their losses.
However, these works still only focus on general object detection.
For instance-level few-shot recognition, it may result in biased classiﬁcation and thus limit the generalization ability of novel classes using the model trained on these mislabeled datasets if we don’t take any action. Recently, one work closely related to ours is the state-of-the-art DeFRCN [28] which decouples Faster-RCNN to alleviate the foreground-background confusion between base pre-training and novel ﬁne-tuning in FSOD. It also can be interpreted from a missing label perspective.
Here, we could view ﬁne-tuning few-shot learning paradigm as a domain adaption procedure from base to novel. In this procedure, a few-shot detector may suffer from foreground-background confusion because one background proposal (negative class may be potential novel class) in the base learning stage will become foreground (positive class) in the novel ﬁne-tuning phase. To mitigate the label conﬂict between the two domains, DeFRCN decouples RCNN and RPN by stopping gradient backpropagation of RPN in Faster-RCNN. Different from the missing label of cross-domain in DeFRCN, we focus on the missing label issue only in the novel (or balanced base-novel) ﬁne-tuning stage.
Another recent work, Pseudo-Labelling [26, 18] mines the missing labeled instances for increasing the number of positive training samples and reducing the biased classiﬁcation. Unfortunately, this method may lead to a chicken-and-egg problem–we need a good few-shot detector to generate good pseudo labels, but we need good few-shot annotations to train a good few-shot detector. Unlike this work, our method completely avoids using any pseudo-label information. In this paper, we propose a simple decoupling method to mitigate the biased classiﬁcation issue.
Speciﬁcally, we ﬁrstly decouple the standard classiﬁer into two parallel heads, positive and negative ones.
Then, these two heads independently process clear positive and noisy negative samples with different strategies.
Our contributions are summarized as follows:
• We rethink FSOD and FSIS from the perspective of label completeness and discover that existing ﬁne-tuning learning-based few-shot methods severely suffer from bias classiﬁcation because the missing label issue naturally exists in the instance-level few-shot scenarios.
To be best of our knowledge, this is the ﬁrst to propose missing label issue in FSOD and FSIS.
• To mitigate the bias classiﬁcation, we propose a simple but effective method that decouples the standard classiﬁer into two parallel heads to independently process clear positive samples and noisy negative ones.
Without bells and whistles, the proposed decoupling classiﬁer can be taken as an alternative to the standard classiﬁer in state-of-the-art FSOD or FSIS models.
• Comprehensive experimental results on PASCAL VOC and MS-COCO show that our approach without any additional parameters and computation cost outperforms state-of-the-art both on FSOD and FSIS tasks.
Related Work
FSOD aims to recognize novel objects and localize them with bounding boxes when only providing a few training instances on each novel class.
Existing works can be roughly grouped into two families, meta-learning and transfer-learning, according to training paradigm. The meta-learning
[Figure] methods [34, 17, 39, 5, 38, 19, 20, 43, 16, 13] use episodic training to acquire task-level knowledge on base classes and generalize better to novel classes.
The transfer-learning methods [33, 35, 31, 43, 1, 28] generally utilize two-stage training strategy ﬁrst pre-training on base classes and then ﬁne-tuning on novel classes, which have signiﬁcantly outperformed many earlier meta-learning approaches.
Recent FSCE [31] shows that the degradation of detection performance mainly comes from misclassifying novel instances as confusable classes, and they propose a contrastive proposal encoding loss to ease the issue.
Similar to the FSCE, FADI [1] explicitly associates each novel class with a semantically similar base class to learn a compact intra-class distribution.
DeFRCN [28] decouples Faster-RCNN [30] to alleviate the foreground-background confusion between pre-training and ﬁne-tuning stage.
FSIS needs to not only recognize novel objects and their location but also perform pixel-level semantic segmentation for each detected instance.
Many FSIS approaches typically use the FSOD framework, e.g., Mask-RCNN [14], it is built on the Faster-RCNN by adding a mask segmentation head.
Therefore, most FSIS methods generally follow the FSOD learning paradigm, i.e., metalearning [23, 39, 7, 24] or transfer-learning [10].
Siamese Mask-RCNN [23] and Meta-RCNN [39] commonly compute embeddings of support images and combine these with those features produced by a backbone network.
Their difference is the combination strategy, e.g., subtraction in [23] and concatenation in [39].
These works only focus on the performance of novel classes and ignore that of base classes.
In real-world applications, we expect that one few-shot model not only recognizes novel classes but also remembers base classes.
The recent iMTFA [10] introduces incremental learning into
FSIS and propose incremental FSIS task.
Methodology 3.1 Few-shot Object Detection and Instance Segmentation Setting
Given an image dataset D
=
{ X i , Y i } Ni =1 , where X i denotes the i -th image and Y i is its corresponding annotation.
For object detection, Y i = { b k , c k } Mk =1 , where b k and c k represent bounding box coordinates and category of the k -th instance presented image
X i , respectively.
For instance segmentation, Y i includes pixel-level mask m k annotation beyond category and bounding box ones, i.e., Y i = { b k , m k , c k } Mk =1 .
Under few-shot learning setting, these annotations can be grouped into two sets, base and novel classes, denoted as C base and C novel .
Note, the base and novel classes are non-overlapping (i.e., C base \ C novel = ; ).
FSOD or FSIS aims to detect/segment novel class instances through training a model based on plenty of labeled instances on a set of base classes and a few instances (e.g, 1, 2, 3 and 5) on each novel class.
Note, FSOD or FSIS only focuses on recognizing novel class instances but ignores base class ones.
It is impractical from the perspective of many real-world applications because ones always expect that a few-shot model is capable of not only recognizing novel classes but also remembering base classes.
To this end, generalized FSOD and FSIS (abbreviated as gFSOD and gFSIS) [11, 27, 6, 10] stresses that a good few-shot learning system should adapt to new tasks (novel classes) rapidly while maintaining the performance on previous knowledge (base classes) without forgetting.
As mentioned above, the transfer-learning-based FSOD and FSIS methods mainly consist of two stages, i.e., pre-training on base classes and ﬁne-tuning on novel classes.
The former is trained on plenty of labeled instances in C base while the latter only uses a few labeled instances in C novel .
For generalized few-shot learning, the main difference is novel stage training which uses few labeled instances in both C base and C novel .
Note that the labeled instances are abundant at the base stage, but there are only a few labeled instances for at novel stage no matter FSOD/FSIS or gFSOD/gFSIS.
Therefore, most instances are unlabeled (missing labeled) as only K instances are provided at the ﬁne-tuning stage.
This is, in fact, reasonable from the perspective of data privacy (incremental learning).
3.2 Revisiting Object Detection and Instance Segmentation
We focus on ﬁne-tuning based FSOD and FSIS because it is more simple and effective compared to meta based ones.
As we know, Faster-RCNN and Mask-RCNN are very popular and powerful solutions as two-stage stacking architecture for fully supervised object detection and instance segmentation.
We ﬁrstly describe the two-stage detector and segmenter.
In general, the ﬁrst stage is designed to generate class-agnostic region proposals which can be formulated as:
[Equation]
where the ✓ s 1 is all the parameters of the ﬁrst stage.
Speciﬁcally, an input image
X i is ﬁrstly fed into a backbone network to Extract high-level Features ( F EF ).
Then, a Region Proposal Network (RPN) is adopted to generate candidate regions based on these extracted features ( F RPN ).
Finally, all these region proposals are pooled into a ﬁxed size feature map using a Region of Interest pooling module ( F ROI ) for the following stage.
The structure of the second stage varies depending on the speciﬁc task.
For object detection, the ROI features of sampled region proposals will be parallelly fed into two heads for performing box classiﬁcation and regression, that is
[Equation]
where ✓ cls and ✓ reg are the parameters of classiﬁcation and regression head, respectively.
Instance segmentation method (e.g., Mask-RCNN) follows the second stage structure of object detection and applies it to three heads, i.e., box classiﬁcation, box regression and mask segmentation, that is
[Equation]
where ✓ seg is parameters of the mask segmentation head.
Given an input image X i and its corresponding annotation Y i = { b k , c k } Mk =1 or Y i = { b k , m k , c k } Mk =1 , Eqs. 2 and 3 can be jointly optimized end-to-end by minimizing L ODs2 ( · ) and L ISs2 ( · ) , which follows a multi-task learning paradigm.
Note, we omitted the RPN learning in Eqs.
2 and 3 for simplicity.
Missing Label Issue.
We can obtain a powerful model minimizing Eqs.
2 and 3 when all interesting instances are completely labeled in a large-scale image dataset D .
The completeness of annotation at label space is, in fact, a general precondition for most fully supervised learning algorithms.
However, it is not satisfactory for a few-shot learning scenario.
The reason is that only a few instances are manually labeled in given training images, and a lot of potential instances may be presented but unlabeled.
From the perspective of practical application, it is natural and unavoidable to miss annotations when facing a few-shot setting.
As shown in Fig.
1 (c), we can see that there are at least three instances including two “person” and one “dog” in this image, but only the “dog” instance carrying annotation (box, mask, category), and other two “person” instances are unlabeled (i.e., missing labels).
In order to quantitatively measure the proportion of missing labeled instances, we compute the average missing rate for each shot on PASCAL VOC and MS-COCO benchmark datasets as shown in Fig.
Firstly, it can be seen that there is a high missing rate on each shot.
For example, 74.3% novel class instances potentially present on 3-shot MS-COCO training images, but they are not given any annotations.
Secondly, the missing rate is further increased under a generalized few-shot setting.
For example, the missing rate increases nearly two times of few-shot setting on the PASCAL VOC.
Despite some efforts try mining these missing labeled instances in a semi-supervised manner for boosting few-shot performance, the fully supervised loss with Eqs.
2 and 3 may result in a suboptimal solution when some instances are unlabeled.
To be best of our knowledge, this is the ﬁrst to propose the missing label issue in few-shot object detection and instance segmentation.
Biased Classiﬁcation Issue.
The Eqs.
2 and 3 can be optimized using sampled ROI features and their labels as mentioned above.
The sampling operation performs box matching between “ROI features” and “annotation”, and assigns training labels (positive or negative) to the corresponding ROIs.
Here, the positive (foreground) ROIs are sampled from object proposals that have an IoU overlap with the ground-truth bounding box at a threshold (e.g., 0.5), while negative (background) ROIs are sampled from the remaining proposals.
The classiﬁcation head is trained based on these sampled positive and negative ROIs.
Different from the classiﬁcation head, the box regression, and mask segmentation head learning are only associated with positive ROIs.
The annotations of positive ROIs are accurate, e.g., the annotations of the “dog” instances in Fig.(c).
However, the sampled negative ROIs may be noisy because of the missing label issue under the few-shot setting.
For example, those two “people” instances will be assigned to negative labels (i.e., background) if they are sampled in Fig.
1 (c), according to the above label assignment strategy.
This will make the standard classiﬁcation head confused with positive and negative samples.
On one hand, the model is correctly guided to recognize positive objects because all positive samples are accurate.
On the other hand, the model may be misguided by noisy negative samples and thus recognize positive objects as background.
Therefore, the bias classiﬁcation may happen when meeting the missing label issue, especially in a few-shot scenario.
Furthermore, it potentially limits the generalization ability to adapt to the novel class quickly and efﬁciently.
3.3 Decoupling Classiﬁer to Mitigate the Bias Classiﬁcation
Standard Classiﬁer.
We assume that x 2 R C +1 is the predicted logit of a sampled ROI feature obtained from Eq. 1 and its corresponding class label vector is y 2 R C +1 , where there are C foreground categories and one background class, and y i is 1 if the corresponding proposal belongs to the i -th category, 0 otherwise.
Then, we use a softmax function to transform it into a probability distribution, that is
[Equation]
The cross entropy loss is used as the measurement of the similarity between the ground-truth y and predicted distribution ˆ p , that is
[Equation]
Note that the standard classiﬁer (Eqs. 4 and 5) may confuse with clear positive and noisy negative samples in few-shot scenario.
Decoupling Classiﬁer.
In order to process positive and negative samples differentially, we decouple the standard classiﬁer into two heads, i.e., positive (foreground) and negative (background) heads, which are formulated as
[Equation]
Here, the positive and negative heads are responsible for positive and negative samples, respectively.
Considering that the labels of positive samples (foreground) are accurate, we can use the cross entropy loss (i.e., Eq. 5) for all positive instances.
The labels of those negative examples may be noisy because the missing label issue.
Therefore, it is not reasonable to employ normal cross entropy loss for training the negative head.
Note that these negative examples are generally sampled from those object proposals that have a maximum IoU overlap with the ground truth bounding box at an interval [0 , 0 . 5) , and thus we can infer that they may not belong to the ground truth class, although we don’t know their true category.
We expect that the bias classiﬁcation would be mitigated if the negative head performs learning only between few-shot labeled categories and the background class.
To this end, we ﬁrst collect all category labels of a few-shot labeled image, and denote it as m =[ m 0 , m 1 , · · · , m C  1 , m C ] T , where m i is a binary indicator, and m i is 1 if the image is labeled with the i -th category, 0 otherwise.
Note that m C [-+
 = 1-]=1 indicates that each image at least contains a background class.
Then, we can obtain a constrained logit ¯ x conditioned on the m , that is
[Equation]
Substituting Eq.
7 into the softmax function Eq.
4 yields:
[Equation]
We compute cross entropy loss between ¯ p and the corresponding ground truth y bg , that is
[Equation]
where y bg C[-+
-] =
and y bg i
6 = C =0 .
Optimization with the Decoupling Classiﬁer.
Considering the joint optimization goal (Eq. [-
-]2 and 3) of object detection and instance segmentation, the optimal parameters ⇥ is determined by minimizing
Eq. 2 and 3, where ⇥ = { ✓ s 1 , ✓ cls , ✓ reg } in object detection, and ⇥ = { ✓ s 1 , ✓ cls , ✓ reg , ✓ seg } in instance segmentation.
For simplicity, we only consider the optimization for the classiﬁcation head and omitted the box regression and mask segmentation head in the following analysis.
✓ cls is updated by a gradient descent step, that is
[Equation]
10) where  is the learning rate.
Note that we have decoupled the standard classiﬁer into positive and negative learning in Eq.
We ﬁrstly analyze the ✓ cls optimization for positive head.
According to the chain rule in Eq. 4 and 5, we have
[Equation]
Then, the derivative of L fgCLS with respect to ✓ cls is
[Equation]
Combining the negative head in Eq.
9 and Eqs.
8 and 7, we can similarly obtain derivative of L bgCLS with respect to ✓ cls is
[Equation]
where y bg is the ground truth label vector of a negative sample.
Comparing Eqs.
12 with 13, we can see that the parameters ✓ cls of the classiﬁcation head will be updated with different ways for positive and negative examples.
For the positive head, the gradient is updated in each dimension of the class space.
But for the negative head, the gradient is limited in some special dimension because of introducing m and thus the biased classiﬁcation may be alleviated.
[Figure]
In order to further understand Eqs.
12 and 13, we give a visualization example for decoupling classiﬁer as illustrated in Fig.
Note that we use Gaussian normal distribution for prediction and ground truth distribution for intuition.
Here, we take the one-shot labeled image in Fig.
1 (c) for example, where only the “dog” instance is labeled.
We assume that one “person” instance is sampled as negative sample, and it will be mistaken as background class (ground truth).
Due to the proposed decoupling classiﬁer, the optimization of the “person” instance is constrained between the “dog” and “background” as shown in Fig.
2 (b) and doesn’t affect the predictions of other categories such as the “person” class.
Experiments
In this section, we empirically evaluate the proposed method for FSOD/gFSOD and FSIS/gFSIS tasks and demonstrate its effectiveness by comparison with state-of-the-art methods.
4.1 Experimental Setup
Datasets.
We follow the previous work and evaluate our method on PASCAL VOC [4] and MSCOCO [21] datasets.
For a fair comparison, we use the same data splits given in [33, 28].
PASCAL VOC covers 20 categories, which are randomly split into 15 base classes and 5 novel classes.
There are three such splits in total.
In each class, there are K (1, 2, 3, 5, 10) objects for [Table] few-shot training.
And the PASCAL VOC 2007 testing set is used for evaluation.
The dataset is only used to evaluate the FSOD.
We report the Average Precision (IoU=0.5) for novel classes (AP50).
MS-COCO contains 80 classes.
The 20 categories presented in the PASCAL VOC are used as novel classes and the remaining 60 categories are used as base classes.
We train a few-shot model based on
K (1, 2, 3, 5, 10, 30) instances for each class and evaluate on the MS-COCO validation set.
This dataset has been widely used to evaluate the performance of FSOD and FSIS.
We report Average
Precision (IoU=0.5:0.95), Average Precision (IoU=0.5) on novel classes for FSOD and FSIS settings.
In addition, we also report AP and AP50 for overall classes, base classes, and novel classes under gFSOD and gFSIS settings, respectively.
Experimental Details.
The experiments are conducted with Detectron2 [36] on NVIDIA GPU V on CUDA 11.0.
We use standard Faster-RCNN with ResNet-101 backbone extracted features from the ﬁnal convolutional layer of the 4-th stage for few-shot object detection as the same as DeFRCN [28].
For instance segmentation, we add a mask prediction head at the ROI of the Faster-RCNN.
For model training, we employ a two-stage ﬁne-tuning approach: ﬁrst training the network on the base classes with pre-trained by ImageNet and then ﬁne-tuning on K -shots for every class.
The SGD is used to optimize our network end-to-end with a mini-batch size of 16, momentum 0.9, and weight decay 5 e  5 on 8 GPUs.
The learning rate is set to 0.02 during base training and 0.01 during few-shot ﬁne-tuning.
Following the previous work [33], all experimental results are averaged over 10 seeds.
For the FSOD, we also report the average results over 10 times repeated runs on seed0 as the same as DeFRCN [28].
Strong Baseline.
We compare our method with the state-of-the-art DeFRCN [28] for FSOD.
For
FSIS, we extend DeFRCN similarly to how Mask-RCNN extends Faster-RCNN, i.e., adding a mask prediction head at the ROI of the DeFRCN and keeping others as the same as DeFRCN, and thus we call it Mask-DeFRCN.
Our method only replaces the standard classiﬁer in DeFRCN and MaskDeFRCN with our decoupling classiﬁer.
Therefore, DeFRCN and Mask-DeFRCN can be taken as our strong baseline for FSOD/gFSOD and FSIS/gFSIS tasks.
4.2 Comparison with the State-of-the-Art
Few-shot Instance Segmentation on the MS-COCO.
Our method (simple decoupling classiﬁer) outperforms the state-of-the-art on the MS-COCO in either FSIS or gFSIS setting.
The main results on
MS-COCO are reported in Table 1 and 2 for FSIS and gFSIS, respectively.
Based on the experiment results, we have the following observations: 1)
The strong baseline (i.e., Mask-DeFRCN) outperforms all the state-of-the-art methods for FSIS and gFSIS; 2) Our method consistently outperforms the baseline Mask-DeFRCN for FSIS;
Compared to FSIS, our method has signiﬁcant improvements for gFSIS.
This is not surprised because the missing rate of gFSIS always is higher than that of FSIS as shown in Fig.
1 (b).
This indicates our method is capable of addressing the missing label issue in the few-shot setting; 3)
Our method has a better advantage, especially in low-shot (1-, 2-, 3-shot), and thus it is very suitable for a few-shot scenario.
On one hand, the missing rate of low-shot is generally higher than that of high-shot so that it leaves more improvement space for our method.
On the other hand, common few-shot models may be weak against noisy negative samples when the number of positive training samples is very small under the few-shot conditions.
In contrast, our model is designed to deal with noisy negative samples issue, and thus it is more effective.
In short, the proposed decoupling classiﬁer is a promising approach to cope with the missing labels issue for FSIS/gFSIS. [Table] [Table] [Table]
Few-shot Object Detection on the PASCAL
VOC and MS-COCO.
Our method signiﬁcantly outperforms the state-of-the-art few-shot object detection methods by a large margin both on the PASCAL VOC and MS-COCO datasets under gFSOD setting again.
For the FSOD setting, our method is also better than the stateof-the-art under most cases.
The results on the PASCAL VOC and MS-COCO are reported Tables 3, 5 and 4, respectively.
Some interesting observations are summarized as follows: 1) Our method signiﬁcantly and consistently exceeds the current state-of-the-art DeFRCN under the gFSOD setting both on the PASCAL VOC and
MS-COCO, which is similar to that of the gFSIS; 2) Our method is better than the strong
DeFRCN in all shots on the MS-COCO and in most cases on the PASCAL VOC under the FSOD setting (averaging the results on 10 seeds).
It is worth noting that our method’s performance on the MS-COCO is close (maybe slightly worse, about 0.5%) to that of DeFRCN if we compare the results based on 10 times repeated runs on the seed0 .
We recheck the missing rate of the seed0 , and ﬁnd
[Table]
[Table]
that the corresponding missing rate is signiﬁcantly lowered (even zero) than that of the other 9 seeds.
This also further indicates that our method is robust when the missing rate is small even zero [Table]
4.3 Ablation Study and Analysis
We conduct the ablation study to analyze the component of our method.
Models in this section are based on the gFSIS setting (1-, 5-, 10-shot) using MS-COCO.
Note that the DeFRCN uses a Prototypical Calibration Block (PCB) to reﬁne the classiﬁcation score which is effective for improving the FSOD performance, but this brings additional computation cost.
Therefore, we consider these two factors including DC (decoupling classiﬁer) head and the PCB in the following analysis. [Figure]
Effectiveness.
We only simply replace the standard classiﬁer with the proposed decoupling classiﬁer in DeFRCN, which results in signiﬁcant improvements, especially for the higher missing rate (e.g., low shot on MS-COCO).
What’s more, our decoupling classiﬁer is effective not only on novel classes but also on base classes, while the PCB seems only effective on novel classes.
In addition, our decoupling classiﬁer without the PCB signiﬁcantly outperforms the counterpart with the PCB on base classes and is also comparable on novel classes.
Efﬁciency.
Firstly, our decoupling classiﬁer does not introduce any additional parameters or computation costs.
Secondly, our method obtains better detection and segmentation performance when using the same complexity as Mask-DeFRCN whether the PCB is used or not.
Last but not least, we only need almost half of the parameters and fewer GFLOPs when removing the PCB block, and still achieve signiﬁcant improvements on base classes and comparable performance on novel classes compared to Mask-DeFRCN using the PCB. O u r s M a s kD e F RCN
O u r s M a s kD e F RCN
Why DC works?
We have given some analysis from the perspective of gradient optimization in
Sec. 3.
Here, we try to discuss from the knowledge of the learned model and compared with the baseline.
We may want to know whether the decoupling classiﬁer mitigates the bias classiﬁcation.
To this end, we employ Recall metric to evaluate the classiﬁcation head for all ground-truth foreground objects.
Note that the classiﬁcation head outputs a multi-class probability distribution ˆ p 2 R C +1 .
The predicted class is determined by argmax i ˆ p
i .
We deﬁne that an object is recalled if its prediction is not background, i.e., any foreground category.
Considering that the number of each foreground category varies considerably, we also compare mRecall (mean the Recall s of all classes).
The comparison results are shown in Fig.
We can see that the mRecall and Recall of the decoupling classiﬁer signiﬁcantly outperforms the standard one on each shot no matter FSIS or gFSIS.
This indicates that our decoupling classiﬁer is helpful to mitigate the bias classiﬁcation thus boosting the FSOD and FSIS performance.
Qualitative Evaluation In Fig. 6, we visualize the results of our method and the strong baseline (Mask-DeFRCN) on MS-COCO validation images using the gFSIS setting with K =10. In the top rows, we show success cases with our method but partly failures with the baseline. These failures are mainly caused by the missing detection because the baseline method may tend to incorrectly recognize positive objects as background (i.e., bias classiﬁcation).
In addition, our method may also produce the failure predictions as shown in the bottom row from left to right, including the missing detection of small or occultation objects, coarse boundary segmentation, and the misclassiﬁcation of similar appearance objects.
Conclusion
In this paper, we ﬁrstly ﬁnd that the missing label widely exists in few-shot scenario.
Furthermore, we analyze that the missing label issue may result in biased classiﬁcation and thus limit the generalization ability on novel classes.
Therefore, we propose a simple but effective method that decouples the standard classiﬁer into two parallel heads to independently process positive and negative examples.
Comprehensive experiments on the few-shot object detection and instance segmentation benchmark datasets show that our approach can effectively and efﬁciently boost FSOD/gFSOD and FSIS/gFSIS performance without any additional parameters and computation cost.
We hope this study attract more interest in designing a simple method for FSOD or FSIS in the future.
A limitation of our method is that it may not be suitable when the missing label rate is small.
However, our method is still comparable to its counterpart even if the missing label rate is zero, which indicates its robustness.