Feature-Proxy Transformer for Few-Shot Segmentation
Anonymous Author(s)
Abstract
Few-shot segmentation (FSS) aims at performing semantic segmentation on novelclasses given a few annotated support samples.
With a rethink of recent advances,we find that the current FSS framework has deviated far from the supervised segmentation framework:
Given the deep features, FSS methods typically use anintricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head.
Due to theintricacy of the decoder and its matching pipeline, it is not easy to follow suchan FSS framework.
This paper revives the straightforward framework of “feature extractor + linear classification head” and proposes a novel Feature-Proxy
Transformer (FPTrans) method, in which the “proxy” is the vector representinga semantic class in the linear classification head.
FPTrans has two keypoints forlearning discriminative features and representative proxies: 1) To better utilize thelimited support samples, the feature extractor makes the query interact with thesupport features from bottom to top layers using a novel prompting strategy.
2) FPTrans uses multiple local background proxies (instead of a single one) because thebackground is not homogeneous and may contain some novel foreground regions.
These two keypoints are easily integrated into the vision transformer backbone withthe prompting mechanism in the transformer.
Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation.
Althoughthe framework is straightforward, we show that FPTrans achieves competitive FSSaccuracy on par with state-of-the-art decoder-based methods.
Introduction
Few-shot learning is of significant value for semantic segmentation.
It is because the semanticsegmentation task requires pixel-wise annotation, which is notoriously cumbersome and expensive [8,27, 25].
Therefore, learning from very few samples for semantic segmentation has attracted significantresearch interest, yielding a popular topic, i e. , few-shot semantic segmentation (FSS).
Formally,
FSS aims at performing semantic segmentation on novel classes given only a few ( e.g. , one or five)densely-annotated samples (called support images) [4].
With a rethink of recent advances in FSS, current FSS methods usually require an intricate decoder,deviating far from the plain supervised segmentation framework.
More concretely, state-of-the-art
FSS methods adopt the “feature extractor + (intricate) decoder” framework (Fig. 1 (a), (b) and (c)),while the supervised segmentation methods usually adopt the “feature extractor + (simple) linearclassification head” framework (Fig.
1 (d)).
In FSS frameworks, the decoders perform sophisticatedmatching and can be summarized into three types (Fig. 1 (a), (b), and (c)), as detailed in the relatedworks in Section2.1.
Arguably, the intricacy of the decoder and its sophisticated matching pipeline
[Figure]
makes the FSS framework hard to follow.
Under this background, we think it is valuable to explore arelatively straightforward FSS framework.
This paper revives the plain framework of “feature extractor + linear classification head” for FSS andproposes a novel Feature-Proxy Transformer (FPTrans) method.
The term “proxy” denotes the vectorrepresenting a foreground class or the background in the linear classification head.
We note that in
Fig.
1 (d), given the extracted feature maps, the supervised segmentation methods simply feed theminto a linear classification head to perform pixel-wise prediction.
FPTrans adapts this straightforwardframework to the FSS task with only one modification: instead of fixing the already-learned proxiesin the classification head (Fig.
1 (d)), FPTrans uses support feature maps and the support mask togenerate the proxies on the fly.
This modification is necessary for recognizing novel classes and isconsistent with some earlier FSS methods [37, 46].
To tackle two FSS challenges ( i.e. , generalization to novel classes and very few support samples)under this simple framework, FPTrans has two keypoints for learning discriminative features and representative proxies, respectively.
1) To better utilize the limited support samples, the feature extractormakes the query interact with the support features from the bottom to top layers.
Consequently,the support sample provides extra information/clues for extracting the query features and is thusbeneficial. 2)
To promote generalization to novel classes, FPTrans uses multiple local backgroundproxies instead of a single global proxy for representing the background.
This design is importantbecause, during base training, the background is not homogeneous and may contain some novelforeground classes.
Consequently, it avoids confusing novel classes with the background of the basedata and thus benefits generalization to novel classes.
We implement the above two keypoints with a novel prompting strategy.
While using the prompt tocondition a transformer for different tasks [26, 20, 47] or different domains [11] is a common practice,our prompting strategy is significantly different and has two novel functions, i.e. , 1) prompting fordifferent (foreground or background) proxies and, 2) acting as the intermediate for query-supportinteraction.
Specifically, FPTrans simultaneously prepends multiple prompts at the input layer,with one prompt for the foreground and the other prompts for the background.
These prompts arefed into the transformer and finally become the foreground proxy and local background proxies,respectively.
During their flow from the input layer to the output layer, the hidden states of prompts inall the hidden layers are shared by the query and support for cross-attention (query-prompt attentionand support-prompt attention).
Therefore, it significantly reduces the interaction complexity from
O ( N 2 ) to O ( N ) ( N is the number of pixels on the feature maps).
Since the prompting and attentionmechanism are critical for these two keypoints, we use the transformer backbone as a natural choice.
We conduct extensive experiments and show that FPTrans achieves accuracy on par with the decoderbased FSS methods.
For example, on PASCAL-5 i [4] with one support sample, FPTrans achieves
68.81% mIoU, setting a new state of the art.
With its simple framework and competitive accuracy, wehope FPTrans can serve as a strong baseline for FSS.
To sum up, our main contributions are summarized as follows: (i)
We revive the plain “featureextractor + linear classification head” framework for FSS.
We hope the correspondingly-proposedmethod FPTrans can serve as a simple and strong FSS baseline.
(ii)
We integrate two keypointsinto FPTrans, i.e. , learning discriminative features through query-support interaction and learningrepresentative local background proxies.
These two keypoints rely on a novel prompting strategyof the transformer backbone and correspondingly tackle two FSS challenges, i.e. , very few supportsamples and generalization to novel classes.
(iii)
We conduct extensive experiments to validate theeffectiveness of FPTrans.
Experimental results show that FPTrans with a plain framework achievescompetitive accuracy, compared with state-of-the-art FSS methods with intricate decoders.
Related Work
2.1 Recent Progress on Few-Shot Segmentation
Early FSS methods directly generate class-specific classifier weights and biases [37, 36, 41] or applythe prototype learning [40, 6, 46, 38].
The recent state-of-the-art methods shift to the decoder-basedframework and can be categorized into three types according to the decoder structure.
1) Somerecent methods[34, 29, 32, 57] (Fig. 1(a)) generate prior maps based on query features and supportfeatures.
They further use CNN or transformer to refine the prior maps (which may be viewed asraw segmentation) into the final segmentation output.
Some methods [61, 59, 42, 24, 60, 50, 51,52, 58] (Fig. 1(b)) concatenate the support prototype vector and the query features and then feed theconcatenated feature maps into a subsequential CNN or transformer for prediction.
Some methodsfocus on exploring fine-grained knowledge and calculating the pixel-to-pixel matching scores betweensupport and query features to derive affinity maps [19, 45, 54, 9, 60, 55, 32] (Fig. 1(c)).
Some furtherapply 4D convolution or transformer [33, 17] to discover the latent patterns within the affinity maps.
Besides, latent information is also investigated to enhance the model [35, 62, 22, 53, 48, 30].
In contrast, this paper abandons the intricate decoder and revives the plain framework of “featureextractor + linear classification head”.
We show that this simple framework can also achieve promising
FSS results.
We note that a recent work [2] also uses the plain framework during training.
However,they rely on transductive inference during testing to mitigate the gap between the inconsistent trainingand testing schemes.
Compared with [2], the proposed FPTrans maintains its simplicity for testingand achieves superior FSS accuracy.
2.2 Backbone for Few-Shot Segmentation
Previous FSS methods usually adopt CNNs ( e.g. , VGG [39], ResNet [15]) as the backbone ( i.e. ,feature extractor) and typically fix the pretrained backbone parameters [59, 42, 60, 57, 48, 32, 24,33, 17].
However, fixing the backbone is prone to a side-effect, i.e. , insufficient adaptation to thesegmentation training data.
Some recent methods finetune the CNN backbone along with FSS trainingwith sophisticated techniques ( e.g. , model ensemble or transductive inference [23, 2]) to tackle theproblem of insufficient adaptation.
In contrast, this paper adopts the vision transformer [7, 16, 49, 31] as the backbone because werely on the attention mechanism and a novel prompting technique for query-support interaction.
Moreover, the proposed FPTrans benefits from fine-tuning the backbone parameters without bells andwhistles.
We note that using the transformer backbone does NOT necessarily improve FSS, becausethe ablation studies (in Table 5) show that replacing the CNN backbone with a transformer does
NOT bring improvements to the decoder-based FSS methods [42, 60].
Therefore, we attribute thesuperiority of our method mainly to the two unique keypoints in FPTrans.
Methods
3.1 Problem Formulation
Few-shot segmentation aims at tackling the semantic segmentation problem on novel classes under alow data regime.
Specifically, FSS usually provides a training set with categories C train and a testing set with novel categories C test ( C train ∩ C test = ∅ ).
The mainstream setting [59, 60, 42] adoptsthe episodic training and testing scheme: Each episode corresponds to a single class c ( c ∈ C trainduring training and c ∈ C test during testing), and provides a query sample { I q , Y q } and K supportsamples { I ( k ) s , Y ( k ) s } Kk =
( I is the image and Y is the label).
The superscript k will be omitted unlessnecessary.
Within each episode, the model is expected to use { I s , Y s } and I q to predict the querylabel.
In this paper, we follow this popular episodic training and testing scheme.
3.2 Preliminaries on Vision Transformer and Prompt Learning
The proposed FPTrans uses the vision transformer as its backbone and integrates a novel promptingstrategy.
Therefore, we first revisit the vision transformer and the prompting mechanism.
Vision Transformer [7](ViT) is designed for computer vision tasks based on transformer [44].
Itis composed of a patch embedding module, a transformer encoder, and an MLP head.
Given an
RGB image as the input, ViT first reshape it into N patches { a p ∈ R 3 × P × P | p = 1 , 2 , . .
, N } ( Pis the patch size) and then projects the flattened image patches into C -Dimensional embeddings byx p =
Embed ( a p ) ∈ R C , p = 1 , 2 , . . .
, N .
We denote the collection of these embedding tokens as
X
= { x p } Np =1 ∈ R N × C .
ViT has L stacked transformer blocks, each one of which consists of a
Multiheaded Self-Attention module and an MLP module (See supplementary for details).
Given X 0(the collection of embedding tokens), ViT concatenates it with a classification token x 0 cls ∈ R C andthen inputs them into the stacked transformer blocks, which is formulated as:
[Equation]
− 1 cls , X l − 1 ]) , l = 1 , 2 , . . .
, L, (1) where x lcls and X l are the outputs of the l -th block B l , and [ · , · ] is the concatenate operation.
Prompting was first introduced in NLP tasks to identify different tasks by inserting a few hintwords into input sentences [10, 21, 3, 28].
More generally, prompting techniques can efficientlycondition the transformer to different tasks [26, 12, 18, 14, 47] or domains [11] without changingany other parameters of the transformer.
To this end, prompting techniques typically prepend someprompt tokens P 0 to the input layer.
Correspondingly, Eqn (1) transforms into [ x lcls , X l , P l ] =
B l ([ x l − 1 cls , x l − 1 , P l − 1 ]) .
It can be seen that changing the prompt simultaneously changes the mappingfunction of the transformer, even if all the transformer blocks B l remain unchanged.
Our prompting strategy in FPTrans is significantly different from the prior prompting techniques.
In contrast to the popular prompting for different tasks or different domains, FPTrans prependsmultiple prompts to simultaneously activate multiple different proxies ( i.e. , the foreground and localbackground proxies), as well as to facilitate efficient query-support interactions.
Moreover, in priorworks, the prompts in the hidden layers are dependent on a single input sample.
In contrast, in
FPTrans, the prompts in the hidden layers are shared by the query and support images through asynchronization procedure.
We will illustrate these points in the following section.
3.3 Feature-Proxy Transformer
3.3.1 Overview
The proposed Feature-Proxy Transformer (FPTrans) consists of three major steps, i.e. , 1) promptgeneration, 2) feature and proxy extraction, and 3) training the classification head and inference, asillustrated in Figure 2.
The prompts are extracted from the support images within a (training or testing)episode and contain a foreground prompt and multiple local background prompts (Section 3.3.2).
Afterward, FPTrans concatenates these prompts with the image patch tokens and forwards them intothe stacked transformer blocks to extract both the features and proxies (Section 3.3.3).
Differentfrom the popular prompting technique, FPTrans shares the hidden states of the prompt tokens for thequery and its support images, using a prompt synchronization operation.
It facilitates efficient featureinteractions between support and query features.
Finally, given the extracted features and proxies, weelaborate on how to use them for training the classification head and for inference in Section 3.3.4.
3.3.2 Prompt Generation
Prompt generation is illustrated in Fig.
2 (a).
Within each episode, FPTrans uses the support image(s)to extract class-aware prompts and share them for the support and query images.
The prompts areclass-aware because we respectively extract prompts from the foreground and background.
[Figure]
According to the background partition results, we generate S local masks for the background,i.e. , { B n } Sn =1 with B n ∈ { 0 , 1 } H × W .
A partition example of S = 3 is illustrated in Fig. 2(a).
Consequently, we calculate the mean feature of the foreground u ∗ f ∈ R C and the local mean featureof background u ∗ n ∈ R C by masked average pooling:
[Equation]
where ˜ Y s ∈ { 0 , 1 } H × W is the down-sampled foreground mask of the support image.
Based on the mean features, we further expand each C -dimensional mean feature vector ( u ∗ f ∈ R Cand u ∗ n ∈ R C ) into a corresponding G × C token and then add it with extra learnable tokens by:
[Equation]
where E is the expansion from C -dimensional vector to G × C -dimensional token, and z f , z n ∈
R G × C are the learnable tokens.
Adding these learnable tokens makes prompts gain extra diversity andbecome more discriminative [56].
See supplementary for more details of this prompt augmentationtechnique.
Consequently, after the expansion and augmentation, we get a foreground prompt tokenp f ∈ R G × C and S local background prompt tokens p n ∈ R G × C ( n = 1 , 2 , · · · , S ) .
3.3.3 Feature and Proxy Extraction
Feature Extraction.
The process for extracting features is illustrated in Fig. 2(b).
Without loss ofgenerality, we take the 1-shot setting as the example for clarity.
Within each episode, a query image
I q and a support image I s are split into N patches and then flattened as { a q,p } Np =1 and { a s,p } Np =1 ,respectively.
An embedding layer projects these patches into query and support patch tokens, i.e. ,
X 0 q ∈ R N × C and X 0 s ∈ R N × C .
We recall that in Section 3.3.2, we already get multiple prompttokens, i.e. , P 0 =
[ p f , p 0 , p 1 , · · · , p S ] .
The patch tokens from a single image and the prompt tokensare concatenated as the query input and support input, i.e. , [ X 0 q , P 0 ] and [ X 0 s , P 0 ] .
The query andsupport inputs are processed by the transformer blocks, which are formulated by:
[Equation]
where l = 1 , 2 , · · · , L enumerates all the transformer blocks.
There is a novel and unique prompt synchronization (Eqn.
(6)) in FPTrans.
Specifically, Eqn.
(6)averages the query and support prompt tokens after every transformer block.
It makes the syncedprompt token P l absorb information from both the query and support patch tokens.
Therefore, in thesubsequential ( l + 1) -th block, the synced prompt token P l passes the support information to thequery tokens [ X l +1 q , P l +1 q ] and vice versa.
In a word, this simple prompt synchronization facilitatesefficient interaction between query and support features.
Proxy Extraction.
As the results of Eqn.
(4) to Eqn.
(6), the feature extractor outputs supportfeatures X Ls , query features X Lq , as well as the deep states of prompts P
L .
FPTrans uses the supportfeatures X Ls and the deep states of prompts P L to extract two types of proxies, i
e. , the feature-basedproxies and the prompt-based proxies, respectively.
• Feature-based proxies .
FPTrans uses the downsampled foreground mask ˜ Y s and the partitionedbackground masks B n (which are already provided in Section 3.3.2) to extract the feature-basedproxies by: u f = 1 | ˜
Y s | (cid:80) i F s,i ˜
Y s,i , u n = 1 |
B n | (cid:80) i F s,i B n,i n = 1 , . . .
, S .•
Prompt-based proxies .
Moreover, FPTrans uses the deep states of prompts P L to get the promptbased proxies by: v f = 1 G (cid:80) Gj
=1 p Lf,j and v n = 1 G (cid:80) Gj =1 p Ln,j .
These two types of proxies are both used for training FPTrans.
In contrast, during testing, FPTransonly uses feature-based proxies.
3.3.4 Training and Inference
FPTrans uses two classification losses (one for the feature-based proxies and the other for theprompt-based proxies) and a pairwise loss for training.
Classification losses.
Since segmentation can be viewed as a pixel-wise classification problem,
FPTrans directly compares the similarity of each query feature vector ( F q,i ) and the proxies forlinear classification.
We elaborate on the classification loss with the feature-based proxy (and theprocess with the prompt-based proxies is similar).
The predicted probability of F q,i belonging to theforeground is formulated as:
[Equation]
where sim ( · , · ) is the cosine similarity between two vectors, τ is a temperate coefficient.
The maxoperation is critical because it facilitates comparing a feature to its closest background proxy.
Given the predicted probability and the ground-truth label, FPTrans uses the standard cross-entropyloss L ce for supervising the pixel-wise classification.
In parallel to L ce , FPTrans uses the promptbased proxies in a similar procedure and derives another classification loss L ′ ce .
Pairwise loss.
In addition to the classification losses, FPTrans further employs a pairwise loss topull close all the foreground features from the query and support samples, as well as to push theforeground and background features far way against each other.
The pairwise loss is formulated as:
[Equation]
[Table]
where Z = | ( Y q,i + Y s,j ) ≥ 1 | is the normalization factor and BCE is the binary cross-entropy loss.σ is a Sigmoid layer and 1 [ · ] is the indicator function.
If two features both belong to the background( ( Y q,i + Y s,j ) = 0) , this pairwise loss function will NOT pull them close.
Ablation studies showthat pulling two foreground features close substantially improves FSS (Table 4) while pulling twobackground features actually compromises the FSS accuracy, as evidenced in the supplementary.
Overall, FPTrans sums all the three losses for training:
[Equation]
(9) where λ is a hyperparameter.
For inference, FPTrans simply uses the query feature and the featurebased proxies for the final prediction (Eqn. (7))
Experiments
4.1 Implementation Details
Datasets and Metrics.
We use two popular FSS benchmarks PASCAL-5 i [37] and COCO-20 i [42]for evaluation.
PASCAL-5 i combines PASCAL VOC 2012 [8] and SBD [13], and includes 20 classes.
Following prior works [42, 23], we split the dataset into four splits with each split 15 classes fortraining and 5 classes for testing.
COCO-20 i is constructed with COCO 2014 [27] and includes 80classes.
It is divided into 4 splits with each split 60 classes for training and 20 classes for testing.
Tocompare with previous methods, we report mean IoU (mIoU) averaged on test classes [42, 60, 23].
Training Details.
All the images are resized and cropped to 480 × 480 and augmented following [42].
We evaluate the proposed method on two vision transformer backbones, ViT-B/16 [7] and DeiT-B/16 [43].
These two backbones are both pretrained on Imagenet-1k [5].
We use SGD optimizerwith momentum 0.9, weight decay 5e-5, and a constant learning rate of 1e-3.
Using [-
-]4 A100 GPUs,we train 60 epochs with ViT and 30 epochs with DeiT backbone, using a batch size 4 for PASCAL-5 iand 16 for COCO-
i (batch size 8 in 5-shot due to the memory limitation).
When we generate thelocal background prompts (and the feature-based proxies), the background of each support imageis partitioned into 5 local parts, i.e. , S = 5 .
Each prompt consists of 12 tokens, i.e. , G = 12 .
Theweight factor λ for balancing the classification loss and pairwise loss (Eqn.
(9)) is set as 2e-2 for
PASCAL-5 i and 1e-4 for COCO-20 i .
Our baseline is implemented as the plain vision transformer.
4.2 Comparison with State-of-the-Art Methods
Main results.
Table 1 evaluates the proposed FPTrans on PASCAL-5 i , from which we draw twoobservations as follows:
First, FPTrans consistently improves the baseline on two backbones.
For example, when using the
ViT-B/16 as its baseline, FPTrans surpasses the baseline by +2.9% and +2.2% mIoU on 1-shot and
[Table]
[Table] [Table]
5-shot settings, respectively.
We note that compared with the baseline, FPTrans has three majordifferences,
i
[Table]
e. , query-support interactions to better utilize the limited support samples, multiple localbackground proxies to promote novel-class generalization, and an additional pairwise loss functionto pull close foreground features.
Ablation studies in Table 4 confirm these advantages as the mainreasons that improve the baseline
[Table]
Second, comparing FPTrans against state-of-the-art methods, we find that FPTrans achieves competitive FSS accuracy.
Under the 1-shot setting, FPTrans on DeiT-B/16 surpasses the most competitive
BAM [23] by 1.0% mIoU. Under the 5-shot setting, the superiority of FPTrans is even larger, i.e. ,+2.8% based on ViT-B/16 and +7.1% based on DeiT-B/16.
Table 2 summarizes the results on COCO-20 i .
The major observations are consistent as on PASCAL5 i .
FPTrans on ViT-B/16 and DeiT-B/16 both surpass the prior state of the art by a clear margin,setting a new state of the art.
Domain shift scenario.
Moreover, we evaluate the proposed FPTrans on the domain shift scenario,where the base training data and the testing data have a significant domain gap.
We use COCO-20 ifor training and use PASCAL-5 i for testing, following previous works [2, 33] for comparison.
Thetraining classes (in COCO-20 i ) and the novel testing classes (in PASCAL-5 i ) are not overlapped.
We summarize the average results on 4 COCO-trained models in Table 3.
FPTrans outperforms
HSNet [33] by +5.6% and +9.0% on the 1-shot and 5-shot settings, respectively.
It confirms theeffectiveness of FPTrans under the domain shift scenario.
4.3 Ablation Study
To better understand the proposed methods, we conduct ablation studies on FPTrans components.
Experiments are conducted with a ViT-B/16 backbone on the 1-shot setting unless specified otherwise.
Ablations on some major components.
We recall that FPTrans has multiple keypoints / importantdesigns, i.e. , an additional pairwise loss for training, a novel prompting strategy, and multiple localproxies.
The corresponding ablations are shown in Table 4, from which we draw three observations.
Experiments are conducted on PASCAL-5 i .
[Table] [Table]
First, the pairwise loss improves the baseline by +1.1% gains on PASCAL-5 i and +0.4% on COCO20 i .
Second, adding prompts further brings +1.0% and +2.7% gains while adding multiple backgroundproxies further brings +1.1% and +1.5% gains on PASCAL-5 i and COCO-20 i , respectively.
Third,compared with the baseline, the full FPTrans equipped with all the three components achieves overallimprovements of +2.9% and 3.6% on PASCAL-5 i and COCO-20 i , respectively.
Transformer backbone for decoder-based method.
Based on two competitive decoder-basedmethods PFENet [42] and CyCTR [60], we replace their CNN backbones with the ViT-B/16 and DeiT-B/16 backbones, as shown in Table 5.
We observe that these two methods undergo considerableaccuracy decreases after the backbone replacement.
It suggests that replacing the CNN backbonewith a transformer does not necessarily improve FSS, although the transformer backbone fits ourplain FSS framework.
[Figure]
We investigate the local background proxies by varying their numbers Sin Table 6.
It is observed that using 5 background proxies achieves the highest accuracy on both PASCAL-5 i and COCO-20 ifor our method.
Moreover, we compare ourmethod with another multi-proxy method(“Mix.”)
proposed in RPMM [52].
RPMMonline trains mixture models to generatemultiple background proxies.
In contrast,our approach for achieving multiple background is relatively simple (by partitioningthe background) and superior ( e.g. , +0.6%gains on PASCAL-
i and +1.4% gains on COCO-20 i when S
= 5 .
We infer that our partition-basedbackground proxies are more representative because it considers a realistic factor, i.e. , the far-awayregions of the background are likely to be inhomogeneous to each other.
Fig.
3 visualizes the activation maps of different proxies (one foreground proxy and three local background proxies).
It confirmsthat the local background proxies well accommodate the inhomogeneous background.
Conclusion
This paper revives the plain framework “feature extractor + linear classification head” and correspondingly proposes a novel Feature-Proxy Transformer (FPTrans) for few-shot segmentation.
During feature extraction, FPTrans makes the query interact with support features in all the transformer blocks, therefore well utilizing the limited support samples.
During proxy extraction, FPTransencodes the complex background into multiple local background proxies, therefore improving thegeneralization towards novel classes.
Given the discriminative features and the representative proxies,
FPTrans directly uses a linear classification head to compare their cosine similarity and achievesstate-of-the-art performance.
With its simplicity and competitive accuracy, we hope FPTrans canserve as a strong baseline for few-shot segmentation.
Limitations and future works.
Currently, FPTrans relies on a pretrained model for generatingprompts, which consumes extra computation resources.
However, we find that FPTrans is robust tothe model for extracting the prompts (see supplementary for details) to some extent.
Therefore, wewill seek more lightweight prompt generating models in future works.
Checklist
For all authors...
(a) Do
the main claims made
in
the abstract and introduction accurately
reflect
the paper’scontributions
and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5 and supplementary.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1 and Section 4.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See the supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[Yes] See the supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See the supplementary.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]