Mask Matching Transformer for Few-Shot Segmentation
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we aim to tackle the challenging few-shot segmentation task from anew perspective.
Typical methods follow the paradigm to firstly learn prototypicalfeatures from support images and then match query features in pixel-level to obtainsegmentation results.
However, to obtain satisfactory segments, such a paradigmneeds to couple the learning of the matching operations with heavy segmentationmodules, limiting the flexibility of design and increasing the learning complexity.
To alleviate this issue, we propose Mask Matching Transformer (MM-Former),a new paradigm for the few-shot segmentation task.
Specifically, MM-Formerfirst uses a class-agnostic segmenter to decompose the query image into multiplesegment proposals.
Then, a simple matching mechanism is applied to merge therelated segment proposals into the final mask guided by the support images.
Theadvantages of our MM-Former are two-fold.
First, the MM-Former follows theparadigm of decompose first and then blend , allowing our method to benefit fromthe advanced potential objects segmenter to produce high-quality mask proposalsfor query images.
Second, the mission of prototypical features is relaxed to learncoefficients to fuse correct ones within a proposal pool, making the MM-Former bewell generalized to complex scenarios or cases.
We conduct extensive experimentson the popular COCO- 20 i and Pascal-
i benchmarks.
Competitive results welldemonstrate the effectiveness and the generalization ability of our MM-Former.
Code is available in the supplementary materials.
Introduction
Semantic segmentation, one of a fundamental tasks in computer vision, has achieved grand success [3, 5, 36] in recent years with the advantages of deep learning techniques [12] and large-scaleannotated datasets [14, 7].
However, the presence of data samples naturally abides by a long-taileddistribution where the overwhelming majority of categories have very few samples.
Therefore,few-shot segmentation [21, 27] is introduced to segment objects of the tail categories only accordingto a minimal number of labels.
Mainstream few-shot segmentation approaches typically follow the learning-to-learning fashion,where a network is trained with episodic training to segment any objects conditioned on a handfulof labeled samples.
The fundamental idea behind it is how to effectively use the informationprovided by the labeled samples (called support ) to segment the test (referred to as the query )image.
Early works [27, 35, 33, 24, 30] achieve this by first extracting one or few semantic-levelprototypes from features of support images, and then pixels in the query feature map are matched(activated) by the support prototypes to obtain the segmentation results.
We refer this kind ofmethod as “few-to-many” matching approach since the number of support prototypes is typicallymuch less ( e.g., two to three orders of magnitude less) than the number of pixels in the query
[Figure]
feature map.
While acceptable results obtained, this few-to-many matching paradigm turns outto be restricted in segmentation performance due to the information loss in extracting prototypes.
Therefore recent advances [34, 26, 19] proceed to a “many-to-many” matching fashion.
Concretely,pixel-level relationships are modeled between the support and query feature maps either by attentionmachanism [34, 26] or 4D convolutions [19].
Benefiting from these advanced techniques, themany-to-many matching approaches exhibit excellent performance over the few-to-many matchingcounterparts.
Overall , the aforementioned approaches construct modules combining the matchingoperation with segmentation modules and optimizing them jointly.
For the sake of improvingthe segmentation quality, techniques of context modeling module such as atrous spatial poolingpyramid [3], self-attention [38] or multi-scale feature fusion [24] are integrated with the matchingoperations [33] and then are simultaneously learned via the episodic training [30, 34, 24].
However,this joint learning fashion not only vastly increases the learning complexity, but also makes it hard todistinguish the effect of matching modules in few-shot segmentation.
Therefore, in this work, we steer toward a different perspective for few-shot segmentation: decouplingthe learning of segmentation and matching modules as illustrated in Fig.
Rather than matchingwith the pixel-level query features, the support samples are directly matched with a few class-agnosticquery mask proposals, forming a “few-to-few” matching fashion.
By performing matching in themask level, several advantages are provided: 1) Such a few-to-few matching mechanism releasesmatching from the segmentation module and focuses on the matching problem itself.
It reducesthe training complexity, thus a simple few-to-few matching is enough for solving the few-shotsegmentation problem.
While previous works turned out to be overfitting when using high-levelfeatures for matching and predicting the segmentation mask [24, 27, 34], the learning of our matchingand segmentation module would not affect each other and hence avoids this daunting problem.
To achieve this few-to-few matching strategy, we introduce a two-stage framework, named Mask
Matching Transformer (dubbed as MM-Former), that generates mask proposals for the query imagein the first stage and then matches the support samples with the candidates in the second stage.
Recently, MaskFormer [4, 5] formulates semantic segmentation as a mask classification problem,which obtains semantic segmentation results by combining the predictions of binary masks and thecorresponding classification scores, where the masks and the scores are both obtained by using atransformer decoder.
It provides the flexibility for segmenting an image of high quality withoutknowing the categories of the objects in advance.
Inspired by this, we also use the same transformerdecoder as in [4] to predict a set of class-agnostic masks based on the query image only.
To furtherdetermine the target objects indicated by the support annotation, a simple Mask Matching Module isconstructed.
Given the features extracted from both support and query samples, the Mask Matching
Module obtains prototypes from both support and query features through masked global averagepooling [27].
Further, a matching operation is applied to match the supports with all query candidatesand produces a set of coefficients for each query candidate.
The final segmentation result for a givensupport(s)-query pair is acquired by combining the mask candidates according to the coefficients.
Inaddition, to resolve the problem of representation misalignment caused by the differences betweenquery and support images, a Feature Alignment Block is integrated into the Mask-Matching Module.
Concretely, since the features for all images are extracted with a fixed network, they may not bealigned well in the feature space, especially for testing images with novel classes.
A Self Alignment
[Figure]
Potential Objects Segmenter Fixed
[Figure]
Backbone
[Figure]
Transformer Decoder
[Figure]
Align
[Figure]
li gn
Cos
[Figure]
Similarity
MLP
Mask
[Figure]
GAP GAP
N
[Figure] [Figure]
[Figure]
Prototype
[Figure]
Prototypes
[Figure]
[Figure]
block and a Cross Alignment block are introduced to consist of the Feature Alignment Block toalign the query and support samples in the feature space so that the matching operation can be safelyapplied to the extracted features.
We evaluate our MM-Former on two commonly used few-shot segmentation benchmarks: COCO- 20 iand Pascal-
5 i .
Our model stands out from previous works on the challenging COCO- 20 i dataset.
While our MM-Former only performs comparably with previous state-of-the-art methods due tothe limited scale of the Pascal dataset, our MM-Former exhibits a strong transferable ability acrossdifferent datasets ( i.e. , COCO- 20 i → Pascal- 5 i ), owing to our superior mask-matching design.
Ina nutshell , our contributions can be summarized as follows: (1) We put forward a new perspectivefor few-shot segmentation, which decouples the learning of matching and segmentation modules,allowing more flexibility and lower training complexity.
(2) We introduce a simple two-stageframework named MM-Former that efficiently matches the support samples with a set of query maskcandidates to obtain segmentation results.
(3) Extensive evaluations on COCO- 20 i and Pascal- 5 idemonstrate the potential of the method to be a robust baseline in the few-to-few matching paradigm.
Methodology
Problem Setting : Few-shot segmentation aims at training a segmentation model that can segmentnovel objects with very few labeled samples.
Specifically, given two image sets D train and D testwith category set C train and C test respectively, where C train and C test are disjoint in terms of objectcategories ( C train ∩ C test = ∅ ).
The model trained on D train is directly applied to test on D test .
Theepisodic paradigm was adpoted in [24, 37] to train and evaluate few-shot models.
A k -shot episode{{ I s } k , I q } is composed of k support images I s and a query image I q , all { I s } k and I q containobjects from the same category.
We estimate the number of episodes for training and testing set are
N train and N test , the training set and test set can be represented by D train =
{{ I s } k , I q } N train and
D test =
{{ I s } k , I q } N test .
Note that both support masks M s and query masks M q are available fortraining, and only support masks M s are accessible during testing.
Overview : The proposed architecture can be divided into three parts, i.e. , Backbone Network,
Potential Objects Segmenter and Mask Matching Module.
Specifically, the Backbone Network isused to extract features only, whose parameters are fixed during the training.
The Potential Objects
Segmenter (dubbed as POS) is applied to produce multiple mask proposals that may contain potentialobject regions within the given image.
The Mask Matching Module (dubbed as MM module) takessupport cues as guidance to choose the most likely ones from the mask proposals.
The selected masksare finally merged into the target output.
The complete diagram of the architecture is shown in Fig.
Each of the modules will be explained in detail in the following subsections.
2.1 Feature Extraction Module
We adopt a ResNet [12] to extract features for input images.
Unlike previous few shot segmentationmethods [24, 37, 31] using Atrous Convolution to replace stides in convolutions for keeping larger resolutions, we keep the original structure of ResNet following [6].
We use the outputs from thelast three layers in the following modules and named them as F S and F Q for I S and I Q , where
F = (cid:8) F i (cid:9) , i ∈ [3 , 4 , 5] and F is the features of I S or I Q , i is the layer index of backbone network.
We further extract the output of query layer 2 to obtain segmentation mask (named as F 2 Q ).
2.2 Potential Objects Segmenter
The POS aims to segment all the objects in one image.
Follow Mask2Former [4], a standardtransformer decoder [25] is used to compute cross attention between F S and N learnable embeddings.
The transformer decoder consists of 3 consecutive transformer layers, each of which takes thecorresponding F i as an input.
Each layer in the transformer decoder can be formulated as
[Equation]
where E l and E l +1 represent the N learnable embeddings before and after applying the transformerlayer respectively.
TLayer denote a transformer decoder layer.
We simplify the representation oftransformer decoder, whereas we conduct same pipeline proposed by Mask2Former.
The output ofthe transformer decoder is multiplied with F 2 Q to get N mask proposals
M ∈ R N × H/ 4 × W/ 4 .
Notethat Sigmoid is applied to normalize all mask proposals to [0 , 1] .
Besides, our POS abandons theclassifier of Mask2Former since we don’t need to classify the mask proposals.
2.3 Mask Matching (MM) Module
In MM, our goal is using support cues as guidance to match the relevant masks.
The building blocksof MM are a Feature Alignment block and a Learnable Matching block.
We first apply Feature
Alignment block to align F Q and F S from the pixel level.
Then, the Learnable Matching blockmatches appropriate query masks correspondence to the support images.
Feature Alignment Block : We achieve the alignment using two types of building blocks: a SelfAlignment block and a Cross-Alignment block.
The complete architecture is shown in Fig.
Self Alignment 𝐹 𝑄𝑖 𝑐×ℎ×𝑤
Cross Alignment 𝑐×1×ℎ𝑤 LayerNorm 𝑐×ℎ𝑤 𝑐×ℎ𝑤 (Weight-shared)
Self Alignment ...
𝑐×ℎ𝑤
Reshape
Average 𝑐×ℎ𝑤 𝑐×ℎ𝑤 ...
S
Matrix-Multiplication
Position-Wise-Multiplication S Softmax
Transformer Decoder
Transformer Decoder ...
𝐹 𝑆𝑖 ෠𝐹 𝑆 𝑖 ෠𝐹 𝑄 𝑖
We first adopt the Self-Alignment block tonormalize the features in each channel.
Inspired from Polarized Self-Attention [16], wedesign a non-parametric block to normalizerepresentations.
The Self Alignment blockprocesses F S and F Q individually and doesnot involve interactions across images.
Then,the Cross-Alignment block is introduced tomitigate divergence across different images.
F S and F Q are fed into two weight-sharedtransformer decoders in parallel.
We take i thlayer as an example in Fig.
3, which can be formulated as
[Equation]
where F iQ and F iS represent the i th layer feature in F Q and F S .
ˆ F i represent the alignment features.
( ˆ F = (cid:110)
ˆ F i (cid:111)
L i , i ∈ [3 , 4 , 5] ).
MLP denote MultiLayer Perceptron and MHAtten represents multihead attention [25]
[Equation]
where q, k, v mean three matrices, and d k is the dimension of q and k elements.
To distinguish fromthe phrase “Query Image” in few-shot segmentation and “matrix Q” in Transformer, we name “Query
Image” as Q and “matrix Q” as q .
We
Learnable Matching Block : After acquiring ˆ F S and ˆ F Q , we first apply masked global averagepooling (GAP) [37, 27, 24, 31] on each ˆ F i and concat them together to generate prototypes forsupport ground-truth and N query mask proposals.
Named as (cid:8) P gtS (cid:9) , (cid:8) P iQ (cid:9) , P gtS , P nQ ∈ R 3 d andn ∈ [1 , 2 ...N ] .
Here d represents the dimension of ˆ F i , 3 d is achieved by concatenating (cid:110)
ˆ F 3 , ˆ F 4 , ˆ F 5 (cid:111)together.
We use cosine distance to measure the similarity between the prototypes of P gtS and P nQ .
For some cases, the mask corresponding to the prototype with the highest similarity may not becomplete ( e.g. , the support image has only parts of the object).
So, we further use a MLP layer tomerge corresponding masks.
The detailed diagram can be formulated as
[Equation]
where ˆ M is our final result, MLP and cos indicate the fully connected operation and the cosinesimilarity.
We take N similarities ( S ) as the input of MLP, and use the output to perform a weightedaverage of N mask proposals.
Note that we do not select the mask with the highest similarity directly,our ablation studies prove that using this block can help improve the performance.
2.4 Objective
In POS, we adopt segmentation loss functions proposed by Mask2Former (denote as L P ).
We apply
Hungarian algorithm to match mask proposals with groud-truth and only conduct binary cross entropyto supervise on masks with the best matching.
In MM module, we conduct binary cross-entropy loss on ˆ M (denote as L M ) and design a contrastiveloss to constrain the cross-alignment module.
Our goal is to make prototypes for the same classmore similar while different classes less similar by constraining S .
We first normalize S to [0 , 1] bymin-max normalization ˆ S =
S − min( S ) max( S ) − min( S )+ ε .
Then we calculate IoU between N mask proposalsand Query ground-truth.
We assume that mask proposals contain various objects in query image.
It isunrealistic to constrain the corresponding prototypes across different categories since it is hard toacquire the proper similarity among them.
Therefore, we apply a criterion at the location of max (IoU)and denote the point as positive point ˆ S pos .
Meanwhile, only constrain on ˆ S pos may lead all outputsof Cross Alignment block tend to be same.
Thus, we add a constraint on the point in ˆ S correspondingto the lowest IoU, and denote it as negative point ˆ S neg .
Therefore, we assign y pos =[-
-]
and y neg
= 0to ˆ S pos and ˆ S neg during the optimization, respectively.
The cross-alignment loss L co is defined as
[Equation]
Thus, the final loss function can be formulated as L =
L P + λ 1 L M + λ 2 L co , where λ 1 and λ 2 areconstants and are set to 10 and 6 in our experiments.
2.5 Training Strategy
In order to avoid the mutual influence of POS and MM module during training, we propose atwo-stage training strategy of first training POS and then training MM.
In addition, by decoupling
POS and MM, the network can share the same POS under 1-shot and K-shot settings, which greatlyimproves the training efficiency.
K-shot Setting : Based on the two stages training strategy, MM-Former can easily extend to the
K-shot setting by averaging knowledge from K samples, i.e. , P gtS .
Note that after pre-trained the
POS, MM-Former can be applied to 1-shot / K-shot tasks with only a very small amount of training(about 2 epochs).
[Table] [Table]
Experiments
3.1 Dataset and Evaluation Metric
We conduct experiments on two popular few-shot segmentation benchmarks, Pascal- 5 i [10] and
COCO- 20 i [14], to evaluate our method.
Pascal- 5 i with extra mask annotations SBD [11] consistingof 20 classes are separated into 4 splits.
For each split, 15 classes are used for training and 5 classesfor test.
COCO- 20 i consists of annotated images from 80 classes.
We follow the common datasplit settings in [20, 37, 19] to divide 80 classes evenly into 4 splits, 60 classes for training andtest on 20 classes.
1,000 episodes from the testing split are randomly sampled for evaluation.
Toquantitatively evaluate the performance, we follow common practice [24, 27, 35, 19, 37], and adoptmean intersection-over-union (mIoU) as the evaluation metrics for experiments.
3.2 Implementation details
The training process of our MM-Former is divided into two stages.
For the first stage, we followthe training setting of Mask2former [4] to train the POS on each fold of Pascal- 5 i and COCO- 20 i .
Note that we freeze the ImageNet [7] pre-trained backbone to prevent it from learning class-specificfeatures of "seen" classes.
The POS is trained on Pascal- 5 i for 20,000 iterations and 60,000 iterationson COCO- 20 i , respectively.
Learning rate is set to 1 e − 4 , batch size is set to 8.
For the secondstage, we freeze the parameters of the backbone and the POS, and only train the MM module for10,000/20,000 iterations on Pascal- 5 i / COCO- 20 i , respectively.
Learning rate is set to 1 e − 4 , batchsize is set to 4.
For both stages, we use AdamW [17] optimizer with a weight decay of 5 e − 2 .
Thelearning rate is decreased using the poly schedule with factor of 0 .
All images are resized andcropped into 480 × 480 for training.
We also employ random horizontal flipping and random croptechniques for data augmentation.
During the testing, we resize the short side of the given image to480 and keep it aspect ratio.
All the experiments are conducted on a single RTX A6000 GPU.
Thestandard ResNet-50 [12] is adopted as the backbone network.
3.3 Comparison with State-of-the-art Methods
We compare the proposed approach with state-of-the-art methods [24, 31, 13, 28, 19, 18, 1, 15, 37]on Pascal- 5 i and COCO-
i datasets.
The results are shown in Tab. 2 and Tab. 1.
Results on COCO- 20 i .
In Tab. 1, our MM-Former performs remarkably well in on COCO for both1-shot and 5-shot setting.
Specifically, we achieve 3 .
9% improvement on 1-shot compared with
CyCTR
[34] and outperform HSNet[19] by 1 . 5% mIoU .
[Table] [Table]
Results on Pascal- 5 i .
Due to the limited number of training samples in the Pascal dataset, the POSmay easily overfit during the first training stage.
Therefore, following recent works [24, 1, 19], weinclude the transferring results that transfer the COCO-trained model to be tested on Pascal.
Note thatwhen training on the COCO dataset, the testing classes shared with the Pascal dataset are removed,so as to avoid the category information leakage.
According to Tab. 2, although our MM-Former is slightly inferior to some competitive results whentraining on Pascal dataset, we find MM-Former exhibits remarkable transferability when trainingon COCO and testing on Pascal.
Specifically, the previous state-of-the-art method HSNet showspowerful results on Pascal → Pascal but degrades when transferring from COCO to Pascal.
Instead, our
MM-Former further enhance the performance of 1-shot and 5-shot by 4 . 4% and 5 .
5% , outperforming
HSNet by 6 . 1% and 1 .
7% , respectively.
Oracle Analysis.
We also explore the room for further improvement of this new few-shot segmentation paradigm by using query ground truth (GT) during inference.
The results refer to the last rowsin Tab. 1, 2.
In detail, after the POS generates 100 mask proposals, we use the GT mask to selectone proposal mask with the highest IoU, and regard this result as the segmentation result.
Note thatthis natural selection is not the optimal solution because there may be other masks complementaryto the selected one, but it is still a good oracle to show the potential of the new learning paradigm.
According to the results, there is still a large gap between current performance and the oracle ( ≈ 20%mIoU), which suggests that our model has enormous potential for improvement whereas we haveachieved the state-of-the-art performance.
3.4 Ablation Studies
We conduct ablation studies on various choice of designs of our MM-Former to show their contributionto the final results.
Component-wise ablations, including the Mask Maching Module and the POS, areshown in Sec.
We also experimentally demonstrate the superiority of the two-stage approach in Sec. 3.4.2.
The experiments are performed with the 1-shot setting on Pascal- 5 i and COCO- 20 i .3.4.
Component-wise Ablations
To understand the effect of each component in the MM module, we start the ablations with a heuristicmatching baseline and progressively add each building block.
Baseline.
The result of the heuristic matching baseline is shown in the first row of Tab. 3a, whichdirectly selects the mask corresponding to the highest cosine similarity with the support prototype.
[Figure]
Note that when not using the learnable matching block, the results in Tab.
3a are all obtained in thesame way as the heuristic matching baseline.
We observe that the heuristic matching strategy doesnot provide strong performance, which is caused by the feature misalignment problem and fails tofuse multiple mask proposals.
Self-Alignment Block .
With the self-alignment block, the performance is improved by 4 .
3% on
Pascal and 1 . 2% on COCO, as shown by the 2 nd result in Tab. 3a, demonstrating that channel-wiseattention does help normalize the features for comparison.
However, the performance is still inferior,encouraging us to further align the support and query features with the cross-alignment block.
Cross-Alignment Block .
In the third result of Tab. 3a, we experiment with a non-parametric variantof the cross-alignment block that removes all learnable parameters in the cross-alignment block.
A significant performance drop is observed.
This is not surprising because the attention in thecross-alignment block cannot attend to proper areas due to the feature misalignment.
When learningthe cross-alignment block, indicated by the 4 th result of Tab. 3a, the performance is remarkablyimproved by 14% on Pascal and 12 .
9% on COCO, manifesting the necessity of learning the featurealignment for further matching.
Learnable Matching Block .
Surprisingly, simply using our learnable matching block can alreadyachieve decent performance (the 5 th result in Tab. 3a) compared with the baseline, thanks to itscapability to adaptively match and merge multiple mask candidates.
Mask Matching Module .
By applying all components, our MM-Former pushes the state-of-theart performance on COCO to 43 .
2% (the 7 th result in Tab. 3a).
In addition, to encourage thealignment of query and support in the feature space, we add the auxiliary loss
L co to the output of thecross-alignment block, which additionally enhances the performance by more than 1% .
[Table] Although we follow Mask2former to build our POS, several differencesare made and we evaluate the choice of design as follows.
Mask Classification .
In Mask2Former [4],a linear classifier is trained with cross-entropy loss for categorizing each mask proposal, while in our
MM-Former for few-shot segmentation, we remove it to avoid learning class-specific representationsin the first training stage.
The result in Tab.
4a shows that the classifier harms the performance due tothat the linear classifier would make the network fit the “seen” classes in the training set.
Since thischange only affects the first stage, we use the oracle results to demonstrate the effect.
Numbers of
Proposals .
In Tab. 4b, we try to vary the numbers of the mask proposals N .
Increasing the numberof N will significantly improve the oracle result and our result.
Thus we chose 100 as the defaultvalue in all other experiments.
It is worth noting that, when varying the number from 10 to 100, ourresult is improved by 5 . 4% , but the oracle result is improved by 12 . 1% , indicating the large room forimprovement with our new mask matching paradigm.
3.4.2 Effect of Decoupling Matching and Segmentation
Effect of Two-stage Training .
One may wonder what if we couple the training of POS and MMmodules.
Tab.
3c experiments on this point, and the joint optimization is inferior to the two-stagetraining strategy.
When training in an end-to-end fashion, POS would be influenced by the training ofthe MM module, degrading the quality of mask proposals.
Besides, the MM module would also beaffected by the learning with POS since POS generally converges slower than the MM module.
Effect of Different Feature Extraction .
Previous few-shot segmentation works [34, 24, 30] typically integrate matching operations with segmentation-specific techniques [38, 24, 3].
Following
Mask2Former, our POS also includes a multi-scale deformable attention (MSDeformAttn) [38].
In
Tab.
3b, we investigate using the features from the MSDeformAttn instead of the backbone featurefor the MM module.
Interestingly, although the feature after the context modeling is essential forsegmentation, it is not suitable for the matching problem and impair the matching performance.
3.5 Qualitative Analysis
[Figure]
We show some visual examples in Fig.
Without loss of generality, some support images may onlycontain part of the object ( e.g. , the 2 nd row).
Directly selecting the mask with the highest cosinesimilarity can not obtain the anticipated result.
[Figure]
learnable mask matching block to solve further improve by alleviating problem
We also provide robustness analysis in Fig.
5, which uses an anomaly support sample for segmenting the query image.
Compared with previous state-of-the-art method, HSNet, which tends tosegment the salient object in the image, our model is more robust to the anomaly inputs.
Related Work
Few-Shot Segmentation [21] is established to perform segmentation with very few labeled images.
Many recent approaches formulate few-shot segmentation from the view of metric learning [23, 8, 27].
PrototypicalNet [22] is the first to perform metric learning on few-shot segmentation.
PFENet [24]further designs an feature pyramid module to extract features from multi-levels.
Many recent methodspoint out that only a single support prototype is insufficient to represent a given category.
To addressthis problem, [32] attempt to obtain multiple prototypes via EM algorithm.
[15] utilized super-pixelsegmentation technique to generate multiple prototypes.
Another way to solve the above problemis to apply pixel-level attention mechanism.
[32, 26] attempt to use graph attention networks toutilize all foreground support pixel features.
HSNet [19] propose to learn dense matching through 4D
Convolution.
CyCTR
[37] points out that not all foreground pixels are conducive to segmentationand adopts cycle-consistency technology to filter out proper pixels to guide segmentation.
Transformers originally proposed for NLP [25] are being rapidly adapted in computer visiontask [9, 2, 29, 5, 4].
The major benefit of transformers is the ability to capture global informationusing self attention module.
DETR [2] is the first work applying Transformers on object detectiontask.
Mask2Former [4] using Transformers to unify semantic segmentation and instance segmentation.
Motivated by the design of MaskFormer, we apply transformers to segment all potential objects inone image, align support features and query features in pixel-level within our MM-Former.
Conclusion
In the paper, we present Mask Matching Transformer (MM-Former), a new perspective to tacklethe challenging few-shot segmentation task.
Different from the previous practice, MM-Former isa two-stage framework, which adopts a Potential Objects Segmentor and Mask Matching Moduleto first produce high-quality mask proposals and then blend them into the final segmentation result.
Extensive experiments on COCO- 20 i and Pascal-
i well demonstrate the effectiveness and thegeneralization advantage of the proposed MM-Former.
We hope our MM-Former can serve as solidbaseline and help advance the future research of few-shot segmentation.
Limitations and societal impact.
Our MM-Former introduces the paradigm of decompose first andthen blend to the research of few-shot segmentation, which is a totally new perspective and mayinspire future researchers to develop more advanced versions.
However, there is still a large gapbetween the current results and the oracle ( ≈ 20% mIoU).
How to further narrow this gap is ourfuture research focus.
Checklist
For all authors...
(a) Do
the main claims made
in
the abstract and introduction accurately
reflect the
paper’scontributions
and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] .
See Section 5.
Limitations andsocietal impact.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] .
SeeSection 5.
Limitations and societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] .
We have read the ethics review guidelines carefully.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] .
Code is
available in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] .
See Section 4-2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] .
All models are trained on a RTX
A6000 GPU.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] .
Our method isimplemented based on MaskFormer and Mask2Former code bases.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] .
The datasets we used in the paper are all open source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]