A DAPTIVE M ASKED W EIGHT I MPRINTING FOR F EWS HOT S EGMENTATION
A BSTRACT
Deep learning has mainly thrived by training on large-scale datasets.
However, for continual learning in applications such as robotics, it is critical to incrementally update its model in a sample efﬁcient manner.
We propose a novel method that constructs the new class weights from few labelled samples in the support set while updating the previously learned classes.
Inspiring from the work on adaptive correlation ﬁlters an adaptive masked imprinted weights method is proposed
It utilizes a masked average pooling layer on the output embeddings and acts as a positive proxy for that class.
It is then used to adaptively update the 1x convolutional ﬁlters that are responsible for the ﬁnal classiﬁcation
Our proposed method is evaluated on PASCAL
5 i dataset and outperforms the state of the art in the 5-shot semantic segmentation
Unlike previous methods our proposed approach does not require a second branch to estimate parameters or prototypes and it enables the adaptation of previously learned weights
We further propose a novel setup for evaluating incremental object segmentation which we term as incremental PASCAL (iPASCAL) where our adaptation method has shown to outperform the baseline method
I NTRODUCTION
Few-shot learning literature has mainly focused on image classiﬁcation Finn et al.
(2017)Koch et al.
(2015)Lin et al.
(2017)Vinyals et al.
(2016)Snell et al.
Qi et al.
(2017)Qiao et al.
However, unlike image classiﬁcation, semantic segmentation requires to learn a pixel-wise classiﬁcation and can provide multiple classes in the support set.
Thus, segmentation is more challenging in the low-shot regime than classiﬁcation.
One of the methods that are based on learning a parameter predictor was proposed by Shaban et al.
A conditional network method was proposed by Rakelly et al.
based on sparse or dense labels to guide the segmentation network.
Another method that inspires from prototypical networks was proposed by Dong & Xing (2018).
The previous methods require the training of an additional branch to act as a prototype learner or a parameter prediction branch.
If a continuous stream of data is presented to the model that has annotations for both novel and previously learned classes, there is no direct extension to adapt their model.
A concurrent work to ours is proposed by Zhang et al.
(2018) that proposes a similarity guided network similar to Rakelly et al.
but it utilize only one branch.
In this paper we propose an adaptive masked weight imprinting scheme for few-shot semantic segmentation.
Our main inspiration is from classical approaches in learning adaptive correlation ﬁlters Bolme et al.
Henriques et al.
Correlation ﬁlters date to 1980s by Hester &
Casasent (1980) that proposed learning an averaged matched spatial ﬁlter constructed as a weighted linear combination of basis functions.
Bolme et al.
(2010) proposed a fast object tracking method based on adaptive correlation ﬁlters, where the ﬁlters are updated using a running average.
Our method proposes a novel scheme to compute convolutional ﬁlters to match the objects through masked weight imprinting, while adapting the learned ones.
Weight imprinting Qi et al.
(2017) has been proposed for image classiﬁcation and relates metric learning methods to softmax classiﬁcation.
It utilizes the normalized embeddings for the support set as proxies and concatenate it to the original weight matrix in the last classiﬁcation layer.
However, the method does not provide a way to adapt the weights that were previously learned which would not be suitable for a continuous data stream.
It
[Figure]
was designed mainly for classiﬁcation and not semantic segmentation which is more challenging to operate with few labelled samples.
Unlike previous methods, our approach can easily operate with any pretrained network without the need to train a second branch.
The contributions of this paper are: (1) we propose a multi-resolution masked weight imprinting scheme for few-shot segmentation.
(2) we propose a novel adaptive weight imprinting scheme that inspires from adaptive correlation ﬁlters, in order to update the weights of previously learned classes.
(3) Our method outperforms the state of the art on the 5-shot case on
PASCAL- 5 i , and the adaptation method outperforms the baseline method.
(4) We propose iPASCAL which is the incremental version of PASCAL-VOC to evaluate incremental object segmentation.
P ROPOSED M ETHOD
We formulate a problem similar to Shaban et al.
(2017) for the few-shot setting.
However, for the incremental object segmentation setup we propose a novel setup using incremental PASCAL which we term as iPASCAL.
The PASCAL VOC dataset Everingham et al.
(2015) classes are split into L train and L incremental with 10 classes each, where L train ∩ L incremental = ∅ .
The classes belonging to the L train are used to construct the training dataset D train and pre-train the segmentation network.
Unlike the static setting in the few shot case, the incremental mode provides the image-label pairs incrementally with different encountered tasks.
Each task introduces two novel classes to learn.
The tasks are in the form of triplets ( t i , ( X i , Y i )) , where ( X i , Y i ) represent the overall batch of images and labels from task t i .
The batch labels are for the two novel classes belonging to task t i , and the previously learned classes in the encountered tasks t 0 , ..., t i − 1 .
In each task the model encounters each image-label pair from the current batch only once.
The backbone architecture used in our segmentation network is a VGG-16 Simonyan & Zisserman (2014) that is pre-trained on ImageNet Deng et al.
Inspiring from the work in few-shot image classiﬁcation with imprinted weights Qi et al.
(2017) we propose to utilize a masked weight imprinting scheme.
We utilize the embeddings for the few labelled samples from the novel class as proxies.
The weight ﬁlter is used as a proxy that convolves the output feature map and computes the extent to which the different parts in the feature map matches these proxies.
In order to incorporate the pixels that belong mainly to the novel class, masked feature maps with the binary labels provided in the support set are used.
This is followed by average pooling the masked feature maps per channel as in equation 1, we denote this layer as masked average pooling Zhang et al.
[Equation]
Here Y il is a binary mask for i th image with the novel class l , F ri are the corresponding output feature maps for i th image and r th resolution.
X is the set of all possible spatial locations and N is the number of pixels that are labelled as foreground for class l .
The normalized output from the masked average pooling layer ˆ P rl can be further used as proxies representing class l and resolution r .
In the case of a novel class the proxy can be utilized directly as the weight ﬁlter.
An average of all the masked pooling features for the k -shot samples provided in the support set is used.
In case of the previously learned classes, the convolutional layer weights in our model can be updated with the newly imprinted weights for that class in an adaptive scheme.
A running average is used to update the weights following equation 2 with the update rate α .
Figure 1 shows the adaptive masked weight imprinting scheme.
Masked weight imprinting is performed on multiple resolution levels in order to improve the segmentation accuracy.
Our proposed online adaptation method can be followed by back-propagation, instead of performing back-propagation on randomly generated weights.
Starting with random weights could cause overﬁtting when ﬁne-tuning with few labelled samples.
However, with the proposed adaptation method it can be interleaved with back-propagation and ensures that the few labelled samples are properly utilized to change the convolutional ﬁlters.
[Equation]
E XPERIMENTAL R ESULTS
The setup for pretraining the models to be tested on PASCAL- 5 i is detailed.
The base network is trained using RMSProp Hinton with learning rate 10 − 6 , and L2 regularization with a factor of 5x 10 −on the 15 classes outside of the current fold.
In the few-shot evaluation 1000 samples are used similar to OSLSM setup Shaban et al.
The alpha parameter used for adapting the previously learned weights is 0.26 for the 1 shot and 0.20 for the 5 shot.
Table 1 and Table 2 show the results using the evaluation method from Co-FCN Rakelly et al.
(2018) and the method in OSLSM Shaban et al.
(2017) respectively on PASCAL- 5 i .
Our method is compared to the state of the art and the baseline methods for few-shot segmentation.
Our method outperforms OSLSM and Co-FCN in the 5-shot case.
Additionally, our method does not need to train an extra branch for predicting the parameters.
Figure 2 shows the qualitative results on PASCAL- 5 i which shows both the support set image-label pair, and our predicted segmentation for the query image.
We conducted further experiments on iPASCAL, where triplets for the task, the corresponding images and semantic labels are provided.
Semantic labels include the new classes in the current and previous encountered tasks.
Figure 3 shows the comparison between naive ﬁne-tuning from random weights against our proposed adaptive masked weight imprinting without any ﬁne-tuning operations in terms of mIoU. It shows that masked imprinting provides better mIoU in comparison to ﬁne-tuning that will lead to over-ﬁtting.
Fine-tuning was conducted using RMSProp with learning rate 10 − 10 .
Finetuning is performed only to the last layers responsible for pixel-wise classiﬁcation, while the feature extraction weights for VGG16 are ﬁxed.
It is worth noting that the current evaluation setting is a n -way 1-shot, where n increases with 2 additional classes with each encountered task resulting in 10-way 1-shot evaluation in the last task.
[Table] [Table]
C ONCLUSION
In this paper we proposed a novel approach for few-shot semantic segmentation using an adaptive masked imprinting scheme.
Our proposed method outperforms the state of the art few-shot segmentation methods in the 5-shot setting, while it alleviates the need for training a second branch as the previous literature.
We proposed a novel setup iPASCAL to evaluate the effectiveness of our adaptation method.