Feature-Proxy Transformer for Few-Shot Segmentation
Abstract
Few-shot segmentation (FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples.
With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework:
Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head.
Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework.
This paper revives the straightforward framework of “feature extractor + linear classification head” and proposes a novel Feature-Proxy
Transformer (FPTrans) method, in which the “proxy” is the vector representing a semantic class in the linear classification head.
FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from bottom to top layers using a novel prompting strategy.
2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions.
These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer.
Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation.
Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods.
Introduction
Few-shot learning is of significant value for semantic segmentation.
It is because the semantic segmentation task requires pixel-wise annotation, which is notoriously cumbersome and expensive [10, 30, 28].
Therefore, learning from very few samples for semantic segmentation has attracted significant research interest, yielding a popular topic, i.e. , few-shot semantic segmentation (FSS).
Formally,
FSS aims at performing semantic segmentation on novel classes given only a few ( e.g. , one or five) densely-annotated samples (called support images)
With a rethink of recent advances in FSS, we find current FSS methods usually require an intricate decoder, deviating far from the plain supervised segmentation framework.
More concretely, state-ofthe-art FSS methods adopt the “feature extractor + (intricate) decoder” framework (Fig. 1 (a), (b) and [Figure] (c)), while the supervised segmentation methods usually adopt the “feature extractor + (simple) linear classification head” framework (Fig.
1 (d)).
In FSS frameworks, the decoders perform sophisticated matching and can be summarized into three types (Fig. 1 (a), (b), and (c)), as detailed in the related works in Section2.1.
Arguably, the intricacy of the decoder and its sophisticated matching pipeline makes the FSS framework hard to follow.
Under this background, we think it is valuable to explore a relatively straightforward FSS framework.
This paper revives the plain framework of “feature extractor + linear classification head” for FSS and proposes a novel Feature-Proxy Transformer (FPTrans) method.
The term “proxy” denotes the vector representing a foreground class or the background in the linear classification head.
We note that in
Fig.
1 (d), given the extracted feature maps, the supervised segmentation methods simply feed them into a linear classification head to perform pixel-wise prediction.
FPTrans adapts this straightforward framework to the FSS task with only one modification: instead of fixing the already-learned proxies in the classification head (Fig.
1 (d)), FPTrans uses support feature maps and the support mask to generate the proxies on the fly.
This modification is necessary for recognizing novel classes and is consistent with some earlier FSS methods [41, 50].
To tackle two FSS challenges ( i.e. , generalization to novel classes and very few support samples) under this simple framework, FPTrans has two keypoints for learning discriminative features and representative proxies, respectively.
1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from the bottom to top layers.
Consequently, the support sample provides extra information/clues for extracting the query features and is thus beneficial.
To promote generalization to novel classes, FPTrans uses multiple local background proxies instead of a single global proxy for representing the background.
This design is important because, during base training, the background is not homogeneous and may contain some novel foreground classes.
Consequently, it avoids confusing novel classes with the background of the base data and thus benefits generalization to novel classes.
We implement the above two keypoints with a novel prompting strategy.
While using the prompt to condition a transformer for different tasks [29, 23, 51] or different domains [13] is a common practice, our prompting strategy is significantly different and has two novel functions, i.e. , 1) prompting for different (foreground or background) proxies and, 2) acting as the intermediate for query-support interaction.
Specifically, FPTrans simultaneously prepends multiple prompts at the input layer, with one prompt for the foreground and the other prompts for the background.
These prompts are fed into the transformer and finally become the foreground proxy and local background proxies, respectively.
During their flow from the input layer to the output layer, the hidden states of prompts in all the hidden layers are shared by the query and support for cross-attention (query-prompt attention and support-prompt attention).
Therefore, it significantly reduces the interaction complexity from
O ( N 2 ) to O ( N ) ( N is the number of pixels on the feature maps).
Since the prompting and attention mechanism are critical for these two keypoints, we use the transformer backbone as a natural choice.
We conduct extensive experiments and show that FPTrans achieves accuracy on par with the decoderbased FSS methods.
For example, on PASCAL-5 i [5] with one support sample, FPTrans achieves 68.81% mIoU, setting a new state of the art.
With its simple framework and competitive accuracy, we hope FPTrans can serve as a strong baseline for FSS.
To sum up, our main contributions are summarized as follows: (i)
We revive the plain “feature extractor + linear classification head” framework for FSS.
We hope the correspondingly-proposed method FPTrans can serve as a simple and strong FSS baseline.
(ii)
We integrate two keypoints into FPTrans, i.e. , learning discriminative features through query-support interaction and learning representative local background proxies.
These two keypoints rely on a novel prompting strategy of the transformer backbone and correspondingly tackle two FSS challenges, i.e. , very few support samples and generalization to novel classes.
(iii)
We conduct extensive experiments to validate the effectiveness of FPTrans.
Experimental results show that FPTrans with a plain framework achieves competitive accuracy, compared with state-of-the-art FSS methods with intricate decoders.
Related Work 2.1 Recent Progress on Few-Shot Segmentation
Early FSS methods directly generate class-specific classifier weights and biases [41, 40, 45] or apply the prototype learning [44, 8, 50, 42].
The recent state-of-the-art methods shift to the decoder-based framework and can be categorized into three types according to the decoder structure.
Some recent methods[38, 32, 36, 61] (Fig. 1(a)) generate prior maps based on query features and support features.
They further use CNN or transformer to refine the prior maps (which may be viewed as raw segmentation) into the final segmentation output.
Some methods [66, 63, 46, 27, 64, 54, 55, 56, 62] (Fig. 1(b)) concatenate the support prototype vector and the query features and then feed the concatenated feature maps into a subsequential CNN or transformer for prediction.
Some methods focus on exploring fine-grained knowledge and calculating the pixel-to-pixel matching scores between support and query features to derive affinity maps [21, 49, 58, 11, 64, 59, 36] (Fig. 1(c)).
Some further apply 4D convolution or transformer [37, 19] to discover the latent patterns within the affinity maps.
Besides, latent information is also investigated to enhance the model [39, 67, 25, 57, 52, 34].
In contrast, this paper abandons the intricate decoder and revives the plain framework of “feature extractor + linear classification head”.
We show that this simple framework can also achieve promising
FSS results.
We note that a recent work [2] also uses the plain framework during training.
However, they rely on transductive inference during testing to mitigate the gap between the inconsistent training and testing schemes.
Compared with [2], the proposed FPTrans maintains its simplicity for testing and achieves superior FSS accuracy.
2.2 Backbone for Few-Shot Segmentation
Previous FSS methods usually adopt CNNs ( e.g. , VGG [43], ResNet [17]) as the backbone ( i.e. , feature extractor) and typically fix the pretrained backbone parameters [63, 46, 64, 61, 52, 36, 27, 37, 19].
However, fixing the backbone is prone to a side-effect, i.e. , insufficient adaptation to the segmentation training data.
Some recent methods finetune the CNN backbone along with FSS training with sophisticated techniques ( e.g. , model ensemble or transductive inference [26, 2]) to tackle the problem of insufficient adaptation.
In contrast, this paper adopts the vision transformer [9, 18, 53, 35] as the backbone because we rely on the attention mechanism and a novel prompting technique for query-support interaction.
Moreover, the proposed FPTrans benefits from fine-tuning the backbone parameters without bells and whistles.
We note that using the transformer backbone does NOT necessarily improve FSS, because the ablation studies (in Table 5) show that replacing the CNN backbone with a transformer does
NOT bring improvements to the decoder-based FSS methods [46, 64].
Therefore, we attribute the superiority of our method mainly to the two unique keypoints in FPTrans.
Methods 3.1 Problem Formulation
Few-shot segmentation aims at tackling the semantic segmentation problem on novel classes under a low data regime.
Specifically, FSS usually provides a training set with categories C train and a testing set with novel categories C test ( C train ∩ C test = ∅ ).
The mainstream setting [63, 64, 46] adopts the episodic training and testing scheme: Each episode corresponds to a single class c ( c ∈ C train during training and c ∈ C test during testing), and provides a query sample { I q , Y q } and K support samples { I ( k ) s , Y ( k ) s } Kk =
( I is the image and Y is the label).
The superscript k will be omitted unless necessary.
Within each episode, the model is expected to use { I s , Y s } and I q to predict the query label.
In this paper, we follow this popular episodic training and testing scheme.
3.2 Preliminaries on Vision Transformer and Prompt Learning
The proposed FPTrans uses the vision transformer as its backbone and integrates a novel prompting strategy.
Therefore, we first revisit the vision transformer and the prompting mechanism.
Vision Transformer [9](ViT) is designed for computer vision tasks based on transformer [48] which is originally designed for sequential data [7, 3, 33].
It is composed of a patch embedding module, a transformer encoder, and an MLP head.
Given an RGB image as the input, ViT first reshape it into N patches { a p ∈ R 3 × P × P | p = 1 , 2 , . . .
, N } ( P is the patch size) and then projects the flattened image patches into C -Dimensional embeddings by x p =
Embed ( a p ) ∈ R C , p = 1 , 2 , . . .
, N .
We denote the collection of these embedding tokens as X
= { x p } Np =1 ∈ R N × C .
ViT has L stacked transformer blocks, each one of which consists of a Multiheaded Self-Attention module and an MLP module (See supplementary Section A for details).
Given X 0 (the collection of embedding tokens), ViT concatenates it with a classification token x 0 cls ∈ R C and then inputs them into the stacked transformer blocks, which is formulated as:
[Equation]
where x lcls and X l are the outputs of the l -th block B l , and [ · , · ] is the concatenate operation.
Prompting was first introduced in NLP tasks to identify different tasks by inserting a few hint words into input sentences [12, 24, 4, 31].
More generally, prompting techniques can efficiently condition the transformer to different tasks [29, 14, 20, 16, 51] or domains [13] without changing any other parameters of the transformer.
To this end, prompting techniques typically prepend some prompt tokens P 0 to the input layer.
Correspondingly, Eqn (1) transforms into [ x lcls , X l , P l ] =
B l ([ x l − 1 cls , x l − 1 , P l − 1 ]) .
It can be seen that changing the prompt simultaneously changes the mapping function of the transformer, even if all the transformer blocks B l remain unchanged.
Our prompting strategy in FPTrans is significantly different from the prior prompting techniques.
In contrast to the popular prompting for different tasks or different domains, FPTrans prepends multiple prompts to simultaneously activate multiple different proxies ( i.e. , the foreground and local background proxies), as well as to facilitate efficient query-support interactions.
Moreover, in prior works, the prompts in the hidden layers are dependent on a single input sample.
In contrast, in
FPTrans, the prompts in the hidden layers are shared by the query and support images through a synchronization procedure.
We will illustrate these points in the following section.
3.3 Feature-Proxy Transformer 3.3.1 Overview
The proposed Feature-Proxy Transformer (FPTrans) consists of three major steps, i.e. , 1) prompt generation, 2) feature and proxy extraction, and 3) training the classification head and inference, as illustrated in Figure 2.
The prompts are extracted from the support images within a (training or testing) episode and contain a foreground prompt and multiple local background prompts (Section 3.3.2).
Afterward, FPTrans concatenates these prompts with the image patch tokens and forwards them into the stacked transformer blocks to extract both the features and proxies (Section 3.3.3).
Different from the popular prompting technique, FPTrans shares the hidden states of the prompt tokens for the query and its support images, using a prompt synchronization operation.
It facilitates efficient feature [Figure] interactions between support and query features.
Finally, given the extracted features and proxies, we elaborate on how to use them for training the classification head and for inference in Section 3.3.
3.3.2 Prompt Generation
Prompt generation is illustrated in Fig.
2 (a).
Within each episode, FPTrans uses the support image(s) to extract class-aware prompts and share them for the support and query images.
The prompts are class-aware because we respectively extract prompts from the foreground and background.
To this end, we first use a pretrained plain vision transformer (Eqn. (1)) to extract deep features from the support image and get F ∗ s ∈ R C × H × W , in which H and W are the feature height and width, and the subscript s indicates the “support”. Afterward, we use the support mask to crop out the foreground regions and background regions from F ∗ s . While the foreground regions are cropped out as a whole, the background regions are cropped and partitioned into multiple ( S ) local regions, because the background is likely not to be homogeneous.
To this end, we employ a Voronoi-based method [1] to partition the background into S regions (See supplementary Section B.1 for details).
According to the background partition results, we generate S local masks for the background, i.e. , { B n } Sn =1 with B n ∈ { 0 , 1 } H × W .
A partition example of S = 3 is illustrated in Fig. 2(a).
Consequently, we calculate the mean feature of the foreground u ∗ f ∈ R C and the local mean feature of background u ∗ n ∈ R C by masked average pooling:
[Equation]
where ˜ Y s ∈ { 0 , 1 } H × W is the down-sampled foreground mask of the support image.
Based on the mean features, we further expand each C -dimensional mean feature vector ( u ∗ f ∈ R C and u ∗ n ∈ R C ) into a corresponding G × C token and then add it with extra learnable tokens by:
[Equation]
where E is the expansion from C -dimensional vector to G × C -dimensional token, and z f , z n ∈
R G × C are the learnable tokens.
Adding these learnable tokens makes prompts gain extra diversity and become more discriminative [60].
See supplementary Section B.2 for more details on this prompt augmentation technique.
Consequently, after the expansion and augmentation, we get a foreground prompt token p f ∈ R G × C and S local background prompt tokens p n ∈ R G × C ( n = 1 , 2 , · · · , S ) .
3.3.3 Feature and Proxy Extraction
Feature Extraction.
The process for extracting features is illustrated in Fig. 2(b).
Without loss of generality, we take the 1-shot setting as the example for clarity.
(See supplementary Section C for K -shot settings)
Within each episode, a query image I q and a support image I s are split into N patches and then flattened as { a q,p } Np =1 and { a s,p } Np =1 , respectively.
An embedding layer projects these patches into query and support patch tokens, i.e. , X 0 q ∈ R N × C and X 0 s ∈ R N × C .
We recall that in Section 3.3.2, we already get multiple prompt tokens, i.e. , P 0 =
[ p f , p 0 , p 1 , · · · , p S ] .
The patch tokens from a single image and the prompt tokens are concatenated as the query input and support input, i.e. , [ X 0 q , P 0 ] and [ X 0 s , P 0 ] .
The query and support inputs are processed by the transformer blocks, which are formulated by:
[Equation]
where l = 1 , 2 , · · · , L enumerates all the transformer blocks.
There is a novel and unique prompt synchronization (Eqn.
(6)) in FPTrans.
Specifically, Eqn.
(6) averages the query and support prompt tokens after every transformer block.
It makes the synced prompt token P l absorb information from both the query and support patch tokens.
Therefore, in the subsequential ( l + 1) -th block, the synced prompt token P l passes the support information to the query tokens [ X l +1 q , P l +1 q ] and vice versa.
In a word, this simple prompt synchronization facilitates efficient interaction between query and support features.
Proxy Extraction.
As the results of Eqn.
(4) to Eqn.
(6), the feature extractor outputs support features X Ls , query features X Lq , as well as the deep states of prompts P L .
FPTrans uses the support features X Ls and the deep states of prompts P L to extract two types of proxies, i.e. , the feature-based proxies and the prompt-based proxies, respectively.
• Feature-based proxies .
FPTrans uses the downsampled foreground mask ˜ Y s and the partitioned background masks B n (which are already provided in Section 3.3.2) to extract the feature-based proxies by: u f = 1 | ˜
Y s | (cid:80) i F s,i ˜
Y s,i , u n = 1 |
B n | (cid:80) i F s,i B n,i n = 1 , . . .
, S .
• Prompt-based proxies .
Moreover, FPTrans uses the deep states of prompts P L to get the promptbased proxies by: v f = 1 G (cid:80) Gj
=1 p Lf,j and v n = 1 G (cid:80) Gj =1 p Ln,j .
These two types of proxies are both used for training FPTrans.
In contrast, during testing, FPTrans only uses feature-based proxies.
3.3.4 Training and Inference
FPTrans uses two classification losses (one for the feature-based proxies and the other for the prompt-based proxies) and a pairwise loss for training.
Classification losses.
Since segmentation can be viewed as a pixel-wise classification problem,
FPTrans directly compares the similarity of each query feature vector ( F q,i ) and the proxies for linear classification.
We elaborate on the classification loss with the feature-based proxy (and the process with the prompt-based proxies is similar).
The predicted probability of F q,i belonging to the foreground is formulated as:
[Equation]
where sim ( · , · ) is the cosine similarity between two vectors, τ is a temperate coefficient.
The max operation is critical because it facilitates comparing a feature to its closest background proxy.
[Table] Given the predicted probability and the ground-truth label, FPTrans uses the standard cross-entropy loss L ce for supervising the pixel-wise classification.
In parallel to L ce , FPTrans uses the promptbased proxies in a similar procedure and derives another classification loss L ′ ce .
Pairwise loss.
In addition to the classification losses, FPTrans further employs a pairwise loss to pull close all the foreground features from the query and support samples, as well as to push the foreground and background features far away from each other.
The pairwise loss is formulated as:
[Equation]
where Z = | ( Y q,i + Y s,j ) ≥ 1 | is the normalization factor and BCE is the binary cross-entropy loss. σ is a Sigmoid layer and 1 [ · ] is the indicator function.
If two features both belong to the background ( ( Y q,i + Y s,j ) = 0) , this pairwise loss function will NOT pull them close.
Ablation studies show that pulling two foreground features close substantially improves FSS (Table 4) while pulling two background features actually compromises the FSS accuracy, as evidenced in the supplementary Section E.2.
Overall, FPTrans sums all the three losses for training:
[Equation]
where λ is a hyperparameter.
For inference, FPTrans simply uses the query feature and the featurebased proxies for the final prediction (Eqn. (7))
Experiments 4.1 Implementation Details
Datasets and Metrics.
We use two popular FSS benchmarks PASCAL-5 i [41] and COCO-20 i [46] for evaluation.
PASCAL-5 i combines PASCAL VOC 2012 [10] and SBD [15], and includesclasses.
Following prior works [46, 26], we split the dataset into four splits with each split 15 classes for training and 5 classes for testing.
COCO-20 i is constructed with COCO 2014 [30] and includes 80 classes.
It is divided into 4 splits with each split 60 classes for training and 20 classes for testing.
To compare with previous methods, we report mean IoU (mIoU) averaged on test classes [46, 64, 26].
Training Details.
All the images are resized and cropped to 480 × 480 and augmented following [46].
We evaluate the proposed method on two vision transformer backbones, ViT-B/16 [9] and
DeiT-B/16 [47].
These two backbones are both pretrained on Imagenet-1k [6]. The cross-entropy [Table] [Table]
[Table]
losses are optimized with boundary-enhanced weight maps introduced by [65].
We use the SGD optimizer with a momentum of 0.9, a weight decay of 5e- [Table] and a constant learning rate of 1e-3.
Using 4 A100 GPUs, we train 60 epochs with ViT and 30 epochs with DeiT backbone, using a batch size 4 for PASCAL-5 i and 16 for COCO-
i (batch size 8 in 5-shot due to the memory limitation).
When we generate the local background prompts (and the feature-based proxies) [Table] the background of each support image is partitioned into 5 local parts, i
[Table]
e. , S = 5 .
Each prompt consists of 12 tokens, i [Table] e. , G = 12 .
The weight factor λ for balancing the classification loss and pairwise loss (Eqn.
(9)) is set as 2e-2 for PASCAL-5 i and 1e-4 for COCO-20 i .
Our baseline is implemented as the plain vision transformer.
4.2 Comparison with State-of-the-Art Methods
Main results.
Table 1 evaluates the proposed FPTrans on PASCAL-5 i , from which we draw two observations as follows:
First, FPTrans consistently improves the baseline on two backbones.
For example, when using the
ViT-B/16 as its baseline, FPTrans surpasses the baseline by +2.9% and +2.2% mIoU on 1-shot and 5-shot settings, respectively.
We note that compared with the baseline, FPTrans has three major differences, i.e. , query-support interactions to better utilize the limited support samples, multiple local background proxies to promote novel-class generalization, and an additional pairwise loss function to pull close foreground features.
Ablation studies in Table 4 confirm these advantages as the main reasons that improve the baseline.
Second, comparing FPTrans against state-of-the-art methods, we find that FPTrans achieves competitive FSS accuracy.
Under the 1-shot setting, FPTrans on DeiT-B/16 surpasses the most competitive
BAM [26] by 1.0% mIoU. Under the 5-shot setting, the superiority of FPTrans is even larger, i.e. , +2.8% based on ViT-B/16 and +7.1% based on DeiT-B/16. {+Experiments are conducted on PASCAL-
+}
i [Table] [Table] Table 2 summarizes the results on COCO-20 i .
The major observations are consistent as on PASCAL5 i .
FPTrans on ViT-B/16 and DeiT-B/16 both surpass the prior state of the art by a clear margin, setting a new state of the art.
Domain shift scenario.
Few-shot learning has been studied under a domain shift scenario [22].
Therefore, we also evaluate the proposed FPTrans on the domain shift scenario for semantic segmentation, where the base training data and the testing data have a significant domain gap.
We use COCO-20 i for training and use PASCAL-5 i for testing, following previous works [2, 37] for comparison.
The training classes (in COCO-20 i ) and the novel testing classes (in PASCAL-5 i ) are not overlapped.
We summarize the average results on 4 COCO-trained models in Table 3.
FPTrans outperforms HSNet [37] by +5.6% and +9.0% on the 1-shot and 5-shot settings, respectively.
It confirms the effectiveness of FPTrans under the domain shift scenario. [Equation]
Ablation Study
To better understand the proposed methods, we conduct ablation studies on FPTrans components.
Experiments are conducted with a ViT-B/16 backbone on the 1-shot setting unless specified otherwise.
Ablations on some major components.
We recall that FPTrans has multiple keypoints / important designs, i.e. , an additional pairwise loss for training, a novel prompting strategy, and multiple local proxies.
The corresponding ablations are shown in Table 4, from which we draw three observations.
First, the pairwise loss improves the baseline by +1.1% gains on PASCAL-5 i and +0.4% on COCO20 i .
Second, adding prompts further brings +1.0% and +2.7% gains while adding multiple background proxies further brings +1.1% and +1.5% gains on PASCAL-5 i and COCO-20 i , respectively.
Third, compared with the baseline, the full FPTrans equipped with all the three components achieves overall improvements of +2.9% and 3.6% on PASCAL-5 i and COCO-20 i , respectively.
Transformer backbone for decoder-based method.
Based on two competitive decoder-based methods PFENet [46] and CyCTR [64], we replace their CNN backbones with the ViT-B/16 and
DeiT-B/16 backbones, as shown in Table 5.
We observe that these two methods undergo considerable accuracy decreases after the backbone replacement.
It suggests that replacing the CNN backbone with a transformer does not necessarily improve FSS, although the transformer backbone fits our plain FSS framework.
[Figure]
Investigation of the local background proxies.
We investigate the local background proxies by varying their numbers S in Table 6.
It is observed that using 5 background proxies achieves the highest accuracy on both PASCAL-5 i and COCO-20 i for our method.
Moreover, we compare our method with another multi-proxy method (“Mix.”) proposed in RPMM [56].
RPMM online trains mixture models to generate multiple background proxies.
In contrast, our approach for achieving multiple background is relatively simple (by partitioning the background) and superior ( e.g. , +0.6% gains on [-PASCAL-
-]PASCAL-5 i and +1.4% gains on COCO-20 i when S = 5 .
We infer that our partition-based background proxies are more representative because it considers a realistic factor, i.e. , the far-away regions of the background are likely to be inhomogeneous to each other.
Fig.
3 visualizes the activation maps of different proxies (one foreground proxy and three local background proxies).
It confirms that the local background proxies well accommodate the inhomogeneous background.
Conclusion
This paper revives the plain framework “feature extractor + linear classification head” and correspondingly proposes a novel Feature-Proxy Transformer (FPTrans) for few-shot segmentation.
During feature extraction, FPTrans makes the query interact with support features in all the transformer blocks, therefore well utilizing the limited support samples.
During proxy extraction, FPTrans encodes the complex background into multiple local background proxies, therefore improving the generalization towards novel classes.
Given the discriminative features and the representative proxies,
FPTrans directly uses a linear classification head to compare their cosine similarity and achieves state-of-the-art performance.
With its simplicity and competitive accuracy, we hope FPTrans can serve as a strong baseline for few-shot segmentation.
Limitations and future works.
Currently, FPTrans relies on a pretrained model for generating prompts, which consumes extra computation resources.
However, we find that FPTrans is robust to the model for extracting the prompts (see supplementary Section F for details) to some extent.
Therefore, we will seek more lightweight prompt-generating models in future works.
Acknowledgements
The work of the authors was supported by the National Natural Science Foundation of China (No.
, and the Fundamental Research Funds for the Central Universities (No. 226-2022-00087).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5 and supplementary.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and Section 4.3.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See the supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[Yes] See the supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See the supplementary.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]