Original: Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation
Revised: Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation

Original: Abstract
Revised: Abstract

Original: Few-shot semantic segmentation aims to segment the target objects in query under the condition of a few annotated support images.
Revised: Few-shot semantic segmentation aims to segment the target objects in query under the condition of a few annotated support images.

Original: Most previous works strive to mine more effective category information from the support to match with the corresponding objects in query.
Revised: Most previous works strive to mine more effective category information from the support to match with the corresponding objects in query.

Original: However, they all ignored the category information gap between query and support images.
Revised: However, they all ignored the category information gap between query and support images.

Original: If the objects in them show large intraclass diversity, forcibly migrating the category information from the support to the query is ineffective.
Revised: If the objects in them show large intraclass diversity, forcibly migrating the category information from the support to the query is ineffective.

Original: To solve this problem, we are the first to introduce an intermediate prototype for mining both deterministic category information from the support and adaptive category knowledge from the query.
Revised: To solve this problem, we are the first to introduce an intermediate prototype for mining both deterministic category information from the support and adaptive category knowledge from the query.

Original: Specifically, we design an Intermediate Prototype Mining Transformer (IPMT) to learn the prototype in an iterative way.
Revised: Specifically, we design an Intermediate Prototype Mining Transformer (IPMT) to learn the prototype in an iterative way.

Original: In each IPMT layer, we propagate the object information in both support and query features to the prototype and then use it to activate the query feature map.
Revised: In each IPMT layer, we propagate the object information in both support and query features to the prototype and then use it to activate the query feature map.

Original: By conducting this process iteratively, both the intermediate prototype and the query feature can be progressively improved.
Revised: By conducting this process iteratively, both the intermediate prototype and the query feature can be progressively improved.

Original: At last, the final query feature is used to yield precise segmentation prediction.
Revised: At last, the final query feature is used to yield precise segmentation prediction.

Original: Extensive experiments on both
Revised: Extensive experiments on both

Original: PASCAL- 5 i and COCO- 20 i datasets clearly verify the effectiveness of our IPMT and show that it outperforms previous state-of-the-art methods by a large margin.
Revised: PASCAL- 5 i and COCO- 20 i datasets clearly verify the effectiveness of our IPMT and show that it outperforms previous state-of-the-art methods by a large margin.

Original: Code is available at https://github.com/LIUYUANWEI98/IPMT
Revised: Code is available at https://github.com/LIUYUANWEI98/IPMT

Original:  Introduction
Revised:  Introduction

Original: Recent great progress on computer vision rely heavily on a large amount of annotated data, the collecting of which is a time-consuming and labor-intensive work.
Revised: Recent great progress on computer vision rely heavily on a large amount of annotated data, the collecting of which is a time-consuming and labor-intensive work.

Original: To solve this problem, few-shot learning is proposed to learn a model that can be generalized to novel categories with only a few annotated images.
Revised: To solve this problem, few-shot learning is proposed to learn a model that can be generalized to novel categories with only a few annotated images.

Original: This setting is also closer to human learning habits which can learn knowledge from scarce annotated examples and identify new categories quickly.
Revised: This setting is also closer to human learning habits which can learn knowledge from scarce annotated examples and identify new categories quickly.

Original: In this paper we focus on the few-shot semantic segmentation (FSS) task which aims to segment novel objects in the query image with a few annotated support samples.
Revised: In this paper we focus on the few-shot semantic segmentation (FSS) task which aims to segment novel objects in the query image with a few annotated support samples.

Original: Currently, a lot of works have been proposed for FSS and many of them are based on prototype learning.
Revised: Currently, a lot of works have been proposed for FSS and many of them are based on prototype learning.

Original: These methods [39, 28, 12] extract prototypes from the support set to represent the category information and then match them with the query features in a matching network to perform segmentation.
Revised: These methods [39, 28, 12] extract prototypes from the support set to represent the category information and then match them with the query features in a matching network to perform segmentation.

Original: Other graph-based methods [38, 30, 35] and transformer-based methods [19, 40] share the similar high-level idea to convey the category information from the support set to the query image.
Revised: Other graph-based methods [38, 30, 35] and transformer-based methods [19, 40] share the similar high-level idea to convey the category information from the support set to the query image.

Original: However, these methods all rely heavily on the category information extracted from the support set.
Revised: However, these methods all rely heavily on the category information extracted from the support set.

Original: Although it provides deterministic category information guidance, there may exist inherent intra-class diversity between query and support samples, which is collectively ignore by existing works.
Revised: Although it provides deterministic category information guidance, there may exist inherent intra-class diversity between query and support samples, which is collectively ignore by existing works.

Original: In Figure 1 (a), we show the distribution of some support prototypes (in orange) and the prototype generated from a query image (in magenta) of the airplane class.
Revised: In Figure 1 (a), we show the distribution of some support prototypes (in orange) and the prototype generated from a query image (in magenta) of the airplane class.

Original: We observe that for the support images that are similar with the query image (marked as “Similar support” on the right), their prototypes are close to the query prototype in the feature space, in which case the matching network can work well.
Revised: We observe that for the support images that are similar with the query image (marked as “Similar support” on the right), their prototypes are close to the query prototype in the feature space, in which case the matching network can work well.

Original: However, for the support images that have large diversity in pose and appearance compared with the query (marked as “Diverse support” on the left), the distance between the support and query prototypes will be faraway.
Revised: However, for the support images that have large diversity in pose and appearance compared with the query (marked as “Diverse support” on the left), the distance between the support and query prototypes will be faraway.

Original: In such a case, if we forcibly migrate the category information in the support prototype to the query, a large category information bias is inevitably introduced.
Revised: In such a case, if we forcibly migrate the category information in the support prototype to the query, a large category information bias is inevitably introduced.

Original: Therefore, our work aims to relieve this problem by introducing an intermediate prototype that could bridge the category information gap between query and support images through our proposed
Revised: Therefore, our work aims to relieve this problem by introducing an intermediate prototype that could bridge the category information gap between query and support images through our proposed

Original: Intermediate Prototype Mining Transformer (IPMT).
Revised: Intermediate Prototype Mining Transformer (IPMT).

Original: Each IPMT layer consists of two steps, i .
Revised: Each IPMT layer consists of two steps, i .

Original: e .,
Revised: e .,

Original: Intermediate Prototype Mining (IPM) and Query Activation (QA).
Revised: Intermediate Prototype Mining (IPM) and Query Activation (QA).

Original: In IPM we learn the intermediate prototype via combing both the deterministic category information from the support images and the adaptive category knowledge from the query image.
Revised: In IPM we learn the intermediate prototype via combing both the deterministic category information from the support images and the adaptive category knowledge from the query image.

Original: Then, we use the learned prototype to activate the query feature map in QA.
Revised: Then, we use the learned prototype to activate the query feature map in QA.

Original: Furthermore, our IPMT is used in an iterative way to progressively improve the quality of the learned prototype and the activated query feature.
Revised: Furthermore, our IPMT is used in an iterative way to progressively improve the quality of the learned prototype and the activated query feature.

Original: As such, the intermediate prototype can successfully reduce the category information gap with the query set, which is verified by the distribution in Figure 1 (b).
Revised: As such, the intermediate prototype can successfully reduce the category information gap with the query set, which is verified by the distribution in Figure 1 (b).

Original: In summary, our main contributions can be concluded as: 1) To the best of our knowledge, this is the first time to focus on the intra-class diversity between support and query in FSS, and we propose the idea of intermediate prototype to relieve the existing category information gap issue.
Revised: In summary, our main contributions can be concluded as: 1) To the best of our knowledge, this is the first time to focus on the intra-class diversity between support and query in FSS, and we propose the idea of intermediate prototype to relieve the existing category information gap issue.

Original: We propose a novel IPMT to explicitly mine the intermediate prototype which contains both the deterministic information from the support set and the adaptive category knowledge from the query.
Revised: We propose a novel IPMT to explicitly mine the intermediate prototype which contains both the deterministic information from the support set and the adaptive category knowledge from the query.

Original: We present an iterative learning scheme to fully explore the intermediate category information hidden in both support and query and update the query feature.
Revised: We present an iterative learning scheme to fully explore the intermediate category information hidden in both support and query and update the query feature.

Original: 4) Extensive experiments on PASCAL- 5 i and
Revised: 4) Extensive experiments on PASCAL- 5 i and

Original: COCO- 20 i show that our proposed IPMT brings a significant improvement over state-of-the-art methods.
Revised: COCO- 20 i show that our proposed IPMT brings a significant improvement over state-of-the-art methods.

Original:  Related Work 2.1 Visual Transformer
Revised:  Related Work 2.1 Visual Transformer

Original: Recently, transformer is introduced into the field of artificial intelligence and has attracted significant attention from many researchers.
Revised: Recently, transformer is introduced into the field of artificial intelligence and has attracted significant attention from many researchers.

Original: [29] first proposes the transformer architecture composed of selfattention and feed-forward layers, and also achieves remarkable performance in natural language processing.
Revised: [29] first proposes the transformer architecture composed of selfattention and feed-forward layers, and also achieves remarkable performance in natural language processing.

Original: Very recently, transformer has been found to be able to obtain excellent results in computer vision.
Revised: Very recently, transformer has been found to be able to obtain excellent results in computer vision.

Original: Specifically, [5] firstly introduces transformer into vision tasks and proposes vision transformer (ViT) by treating non-overlapped image patches as a series of tokens.
Revised: Specifically, [5] firstly introduces transformer into vision tasks and proposes vision transformer (ViT) by treating non-overlapped image patches as a series of tokens.

Original: Subsequently, a lot of works are devoted to tailoring the transformer structure to be more suitable for computing vision tasks.
Revised: Subsequently, a lot of works are devoted to tailoring the transformer structure to be more suitable for computing vision tasks.

Original: [18] utilizes shifted window self-attention and patch merging to reduce the computational cost and aggregate features, respectively.
Revised: [18] utilizes shifted window self-attention and patch merging to reduce the computational cost and aggregate features, respectively.

Original: [33] imitates the feature pyramid structure in CNNs and proposes the pyramid vision transformer, which reduces the computational cost of self-attention.
Revised: [33] imitates the feature pyramid structure in CNNs and proposes the pyramid vision transformer, which reduces the computational cost of self-attention.

Original: Other transformer-based methods are proposed for various tasks, such as object detection[42, 3], semantic segmentation[41], panoptic segmentation[31], multiple object tracking[27] and so on.
Revised: Other transformer-based methods are proposed for various tasks, such as object detection[42, 3], semantic segmentation[41], panoptic segmentation[31], multiple object tracking[27] and so on.

Original: [3] proposes an transformer-based end-to-end object detection framework and utilizes object queries to locate the objects.
Revised: [3] proposes an transformer-based end-to-end object detection framework and utilizes object queries to locate the objects.

Original: [7] replaces the multi-head attention with twin attention to interact with the context both on row and column features for instance segmentation.
Revised: [7] replaces the multi-head attention with twin attention to interact with the context both on row and column features for instance segmentation.

Original: In the above works, learnable
Revised: In the above works, learnable

Original: Queries * are utilized to aggregate the context-information from feature maps for further regression or segmentation.
Revised: Queries * are utilized to aggregate the context-information from feature maps for further regression or segmentation.

Original: [4] introduces the masked attention to extract relevant features instead of global features by limiting the cross-attention regions within predicted masks.
Revised: [4] introduces the masked attention to extract relevant features instead of global features by limiting the cross-attention regions within predicted masks.

Original: Motivated by these works, we use a learnable prototype as the intermediate prototype to dig out adaptive category information from both query and support images.
Revised: Motivated by these works, we use a learnable prototype as the intermediate prototype to dig out adaptive category information from both query and support images.

Original: 2.2 Few-shot Semantic Segmentation
Revised: 2.2 Few-shot Semantic Segmentation

Original: FSS is a natural extension of semantic segmentation in the condition of a few annotated samples.
Revised: FSS is a natural extension of semantic segmentation in the condition of a few annotated samples.

Original: The typical paradigm proposed by SG-one [25] is using two-branch networks.
Revised: The typical paradigm proposed by SG-one [25] is using two-branch networks.

Original: A conditioning branch extracts the category context from the support images and another segmentation branch segments the query image under the guidance of the former.
Revised: A conditioning branch extracts the category context from the support images and another segmentation branch segments the query image under the guidance of the former.

Original: Following this paradigm, many approaches [36, 12, 37, 14, 28, 34] are proposed to explore how to fully excavate the category information from the support images.
Revised: Following this paradigm, many approaches [36, 12, 37, 14, 28, 34] are proposed to explore how to fully excavate the category information from the support images.

Original: For mining more abundant category information from the support images, ASGNet[36] and PMM[12] construct multiple prototypes using parameter-free methods, i.e. superpixel-guided clustering and the expectation-maximum algorithm, to cluster the support foreground features into multiple prototypes and then activate different areas in the query image.
Revised: For mining more abundant category information from the support images, ASGNet[36] and PMM[12] construct multiple prototypes using parameter-free methods, i.e. superpixel-guided clustering and the expectation-maximum algorithm, to cluster the support foreground features into multiple prototypes and then activate different areas in the query image.

Original: SCL[37] utilizes the missing parts in the initial segmentation result of the support images to form auxiliary support vectors and then merge them in a cross-guidance module to obtain a better prediction.
Revised: SCL[37] utilizes the missing parts in the initial segmentation result of the support images to form auxiliary support vectors and then merge them in a cross-guidance module to obtain a better prediction.

Original: In PFENet[28], a prior mask is generated by calculating the cosine-similarity between support and query features in high-level.
Revised: In PFENet[28], a prior mask is generated by calculating the cosine-similarity between support and query features in high-level.

Original: Then, a feature enrichment module is applied to perform dense comparison on different feature scales obtained by adaptive pooling.
Revised: Then, a feature enrichment module is applied to perform dense comparison on different feature scales obtained by adaptive pooling.

Original: Furthermore, MMNet[34] introduces meta-class memory to store the meta-information during training and applies it into novel classes during the inference stage.
Revised: Furthermore, MMNet[34] introduces meta-class memory to store the meta-information during training and applies it into novel classes during the inference stage.

Original: However, all above methods ignore the inherent intra-class differences between query and support images and transfer the support information to the query image forcibly.
Revised: However, all above methods ignore the inherent intra-class differences between query and support images and transfer the support information to the query image forcibly.

Original: Our work aims to relieve this problem and propose the intermediate prototype to bridge the category information gap between query and support images.
Revised: Our work aims to relieve this problem and propose the intermediate prototype to bridge the category information gap between query and support images.

Original: 2.3 Transformer-based Few-shot Semantic Segmentation
Revised: 2.3 Transformer-based Few-shot Semantic Segmentation

Original: [19] introduces the multi-head attention as an attention module to transfer the classifier weights from support to query.
Revised: [19] introduces the multi-head attention as an attention module to transfer the classifier weights from support to query.

Original: However, it does not take full advantage of the transformer on incorporating long-range dependencies.
Revised: However, it does not take full advantage of the transformer on incorporating long-range dependencies.

Original: [40] proposes a Cycle-Consistent TRansformer (CyCTR) module to select relevant pixel-level support features to perform cross-attention with the query feature.
Revised: [40] proposes a Cycle-Consistent TRansformer (CyCTR) module to select relevant pixel-level support features to perform cross-attention with the query feature.

Original: In our work, instead of performing cross-attention between support and query features, we leverage a learnable
Revised: In our work, instead of performing cross-attention between support and query features, we leverage a learnable

Original: Query as the intermediate prototype to aggregate the category information from both support and query images and refine the query feature using this prototype.
Revised: Query as the intermediate prototype to aggregate the category information from both support and query images and refine the query feature using this prototype.

Original:  Problem Definition
Revised:  Problem Definition

Original: In FSS, the whole dataset is divided into two disjoint subsets D train and D test based on the object categories they contain.
Revised: In FSS, the whole dataset is divided into two disjoint subsets D train and D test based on the object categories they contain.

Original: An FSS model is expected to learn the meta knowledge on D train with sufficient labeled images and generalize to unseen categories on D test with scarce labeled images.
Revised: An FSS model is expected to learn the meta knowledge on D train with sufficient labeled images and generalize to unseen categories on D test with scarce labeled images.

Original: Following the previous meta-learning paradigm, we execute the episodic training strategy to train our model.
Revised: Following the previous meta-learning paradigm, we execute the episodic training strategy to train our model.

Original: Specifically, these two subsets are both partitioned into numerous episodes, each of which randomly samples K + 1 image-mask pairs.
Revised: Specifically, these two subsets are both partitioned into numerous episodes, each of which randomly samples K + 1 image-mask pairs.

Original: For one episode, K pairs compose the support set S =
Revised: For one episode, K pairs compose the support set S =

Original: { ( I si , M si ) }
Revised: { ( I si , M si ) }

Original: Ki =1 and the rest one pair composes the query set Q = {
Revised: Ki =1 and the rest one pair composes the query set Q = {

Original: ( I q , M q ) } , where
Revised: ( I q , M q ) } , where

Original: I ∗ ∈
Revised: I ∗ ∈

Original: R H
Revised: R H

Original: × W × 3 and M ∗ ∈
Revised: × W × 3 and M ∗ ∈

Original: R H × W denote the RGB images and their corresponding binary masks, respectively.
Revised: R H × W denote the RGB images and their corresponding binary masks, respectively.

Original: In each episode sampled from D train , the model is trained to predict the query mask supervised by M q under the guidance of the support set.
Revised: In each episode sampled from D train , the model is trained to predict the query mask supervised by M q under the guidance of the support set.

Original: Then, the trained model is evaluated on
Revised: Then, the trained model is evaluated on

Original: D test to segment unseen categories straightly without any further optimization.
Revised: D test to segment unseen categories straightly without any further optimization.

Original:  A Review of Transformer
Revised:  A Review of Transformer

Original: We first review the typical transformer model here.
Revised: We first review the typical transformer model here.

Original: As the form in [29], a transformer layer mainly consists of an attention block and a multi-layer perception (MLP) block.
Revised: As the form in [29], a transformer layer mainly consists of an attention block and a multi-layer perception (MLP) block.

Original: The former is used to aggregate global contexts and the latter performs embedding updating.
Revised: The former is used to aggregate global contexts and the latter performs embedding updating.

Original: Attention Block.
Revised: Attention Block.

Original: Given an input token sequence X ∈ R L 1 × C and a context token sequence
Revised: Given an input token sequence X ∈ R L 1 × C and a context token sequence

Original: Y ∈ R L 2 × C , where L 1 , L 2 are the length of the two sequences and C is the channel dimension of their embeddings, respectively, the attention block first computes the attention weight matrix:
Revised: Y ∈ R L 2 × C , where L 1 , L 2 are the length of the two sequences and C is the channel dimension of their embeddings, respectively, the attention block first computes the attention weight matrix:

Original: [Equation]
Revised: [Equation]

Original: where W q and W k ∈ R C × d are linear transformation weight matrixes and √ d is the scale factor.
Revised: where W q and W k ∈ R C × d are linear transformation weight matrixes and √ d is the scale factor.

Original: Then, A ( X , Y ) is normalized and then used to aggregate the global context from Y :
Revised: Then, A ( X , Y ) is normalized and then used to aggregate the global context from Y :

Original: [Equation]
Revised: [Equation]

Original: where W v ∈ R C
Revised: where W v ∈ R C

Original: ×
Revised: ×

Original: C is another linear transformation weight matrix.
Revised: C is another linear transformation weight matrix.

Original: If X , Y are the same feature, the attention operation is called self-attention which propagates contexts among different tokens.
Revised: If X , Y are the same feature, the attention operation is called self-attention which propagates contexts among different tokens.

Original: If they are not the same, it is named cross-attention which conveys relevant information from Y to X .
Revised: If they are not the same, it is named cross-attention which conveys relevant information from Y to X .

Original: Usually multi-head attention [29] is used to boost the model performance.
Revised: Usually multi-head attention [29] is used to boost the model performance.

Original: Multi-layer Perception Block.
Revised: Multi-layer Perception Block.

Original: After the attention block, a MLP block is applied to each token separately and identically to further transform the token embeddings.
Revised: After the attention block, a MLP block is applied to each token separately and identically to further transform the token embeddings.

Original: Specifically, MLP is implemented using two linear projection layers with a ReLU activation in between.
Revised: Specifically, MLP is implemented using two linear projection layers with a ReLU activation in between.

Original: Given a token sequence
Revised: Given a token sequence

Original: X ∈ R L 1 × C as the input, it is formulated as:
Revised: X ∈ R L 1 × C as the input, it is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where W ∗ and b ∗ denote the linear transformation weight matrixes and biases, respectively.
Revised: where W ∗ and b ∗ denote the linear transformation weight matrixes and biases, respectively.

Original: We follow [29] and set the channel dimensions of the first and the second layer to be 4 C and C , respectively.
Revised: We follow [29] and set the channel dimensions of the first and the second layer to be 4 C and C , respectively.

Original: Note that layer normalization [1] and residual connections are omitted here for simplicity.
Revised: Note that layer normalization [1] and residual connections are omitted here for simplicity.

Original:  Intermediate Prototype Mining Transformer
Revised:  Intermediate Prototype Mining Transformer

Original: We now present our proposed Intermediate Prototype Mining Transformer (IPMT), as Figureshown, for few-shot semantic segmentation.
Revised: We now present our proposed Intermediate Prototype Mining Transformer (IPMT), as Figureshown, for few-shot semantic segmentation.

Original: Each IPMT layer consists of two steps, i .
Revised: Each IPMT layer consists of two steps, i .

Original: e ., Intermediate
Revised: e ., Intermediate

Original: Prototype Mining (IPM) and Query Activation (QA).
Revised: Prototype Mining (IPM) and Query Activation (QA).

Original: IPM is used to mine the intermediate prototype from both support and query features while QA is designed to activate the query feature map using the learned prototype.
Revised: IPM is used to mine the intermediate prototype from both support and query features while QA is designed to activate the query feature map using the learned prototype.

Original: We adopt a duplex segmentation loss (DSL) to supervise the learning of the intermediate prototype in each IPMT layer.
Revised: We adopt a duplex segmentation loss (DSL) to supervise the learning of the intermediate prototype in each IPMT layer.

Original: Furthermore, we propose to perform the intermediate prototype mining in an iterative way, thus boosting the quality of the learned prototype and the segmentation results progressively.
Revised: Furthermore, we propose to perform the intermediate prototype mining in an iterative way, thus boosting the quality of the learned prototype and the segmentation results progressively.

Original: Next, we will describe them in details.
Revised: Next, we will describe them in details.

Original: 5.1 Intermediate Prototype Mining
Revised: 5.1 Intermediate Prototype Mining

Original: Our IPM has a learnable prototype G to extract adaptive category information from both query and support images using Masked Attention (MA).
Revised: Our IPM has a learnable prototype G to extract adaptive category information from both query and support images using Masked Attention (MA).

Original: The prototype G ∈ R 1 × C is initially a category- and image-agnostic vector that encodes general segmentation prior and will be updated by MA in each episode, encoding adaptive category information for the target category in that episode.
Revised: The prototype G ∈ R 1 × C is initially a category- and image-agnostic vector that encodes general segmentation prior and will be updated by MA in each episode, encoding adaptive category information for the target category in that episode.

Original: Masked Attention.
Revised: Masked Attention.

Original: We leverage cross-attention to update G by using both support and query features as the context.
Revised: We leverage cross-attention to update G by using both support and query features as the context.

Original: Furthermore, to make G only focus on the target regions and extract the category information without noises, we follow [4] and use support and query masks to limit the attended region in the attention matrix.
Revised: Furthermore, to make G only focus on the target regions and extract the category information without noises, we follow [4] and use support and query masks to limit the attended region in the attention matrix.

Original: Specifically, given a flattened support or query feature
Revised: Specifically, given a flattened support or query feature

Original: F ∈ R hw × C and a corresponding binary segmentation mask P ∈ R h × w , we first compute the attention weight matrix
Revised: F ∈ R hw × C and a corresponding binary segmentation mask P ∈ R h × w , we first compute the attention weight matrix

Original: A ( G , F ) ∈ R 1 × hw .
Revised: A ( G , F ) ∈ R 1 × hw .

Original: Then, an attention mask is computed by following [4]:
Revised: Then, an attention mask is computed by following [4]:

Original: [Equation]
Revised: [Equation]

Original: where i denotes the location index.
Revised: where i denotes the location index.

Original: Next, we use ˆP to modulate the attention weights, leading to the masked attention:
Revised: Next, we use ˆP to modulate the attention weights, leading to the masked attention:

Original: [Equation]
Revised: [Equation]

Original: where Vec ( · ) is the vectorization operation.
Revised: where Vec ( · ) is the vectorization operation.

Original: As such, the normalized attention weights on the background regions are close to zero, making sure the prototype only be updated by relevant contexts of the desired category.
Revised: As such, the normalized attention weights on the background regions are close to zero, making sure the prototype only be updated by relevant contexts of the desired category.

Original: In our IPM, on one hand, we use the support feature F s and the ground-truth support mask M s to update G , providing deterministic category information since M s is definitely accurate.
Revised: In our IPM, on one hand, we use the support feature F s and the ground-truth support mask M s to update G , providing deterministic category information since M s is definitely accurate.

Original: On the other hand, we also leverage the query feature F q and a query prediction mask P q to provide query-adaptive category knowledge for G , thus reducing the category information gap between support and query images.
Revised: On the other hand, we also leverage the query feature F q and a query prediction mask P q to provide query-adaptive category knowledge for G , thus reducing the category information gap between support and query images.

Original: After that, an MLP block is further used on the learned prototype.
Revised: After that, an MLP block is further used on the learned prototype.

Original: The whole process can be formulated as:
Revised: The whole process can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Please note that the two masked attention operations do not share weights since the two segmentation masks have different uncertainty.
Revised: Please note that the two masked attention operations do not share weights since the two segmentation masks have different uncertainty.

Original: For the K -shot setting, we simply average the outputs of
Revised: For the K -shot setting, we simply average the outputs of

Original: MaskAttn on support.
Revised: MaskAttn on support.

Original: 5.2 Query Activation
Revised: 5.2 Query Activation

Original: In this step, QA is used to activate the target regions in the query feature map F q under the guidance of the learned prototype G .
Revised: In this step, QA is used to activate the target regions in the query feature map F q under the guidance of the learned prototype G .

Original: Previous works [28, 39] have demonstrated that it is an essential operation for FSS to pass the category information to the query feature map and provide specific segmentation cues.
Revised: Previous works [28, 39] have demonstrated that it is an essential operation for FSS to pass the category information to the query feature map and provide specific segmentation cues.

Original: Specifically, G is expanded and concatenated with F q ∈ R h × w × C to activate the target regions:
Revised: Specifically, G is expanded and concatenated with F q ∈ R h × w × C to activate the target regions:

Original: [Equation]
Revised: [Equation]

Original: where ⊚ represents the concatenation operation, and F actv is a simple activation network which consists of a 1 × 1 convolutional layer, a ReLU layer, and a 3 × 3 convolutional layer.
Revised: where ⊚ represents the concatenation operation, and F actv is a simple activation network which consists of a 1 × 1 convolutional layer, a ReLU layer, and a 3 × 3 convolutional layer.

Original: Additionally, we also follow [40] to use a multi-head deformable self-attention layer [42] for further aggregating context information in the query feature.
Revised: Additionally, we also follow [40] to use a multi-head deformable self-attention layer [42] for further aggregating context information in the query feature.

Original: 5.3 Duplex Segmentation Loss
Revised: 5.3 Duplex Segmentation Loss

Original: To facilitate the learning of the adaptive category information in G , we use it to generate two segmentation masks on both support and query images and calculate two segmentation losses.
Revised: To facilitate the learning of the adaptive category information in G , we use it to generate two segmentation masks on both support and query images and calculate two segmentation losses.

Original: Specifically, motivated by [4], we use G to generate a mask embedding and then conduct multiplication with the image feature maps for obtaining segmentation masks.
Revised: Specifically, motivated by [4], we use G to generate a mask embedding and then conduct multiplication with the image feature maps for obtaining segmentation masks.

Original: The mask generation (MG) process is formulated as:
Revised: The mask generation (MG) process is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where W m ∈ R C
Revised: where W m ∈ R C

Original: ×
Revised: ×

Original: C is a linear projection weight matrix for generating the mask embedding.
Revised: C is a linear projection weight matrix for generating the mask embedding.

Original: Next, the standard binary cross-entropy (BCE) loss is calculated between the generated masks and the ground truth, i .
Revised: Next, the standard binary cross-entropy (BCE) loss is calculated between the generated masks and the ground truth, i .

Original: e ., M q and M s , as our duplex segmentation loss to optimize the prototype learning process:
Revised: e ., M q and M s , as our duplex segmentation loss to optimize the prototype learning process:

Original: [Equation]
Revised: [Equation]

Original: Here, α is a hyperparameter to balance the losses between query and support predictions.
Revised: Here, α is a hyperparameter to balance the losses between query and support predictions.

Original: 5.4 Iterative Prototype Mining
Revised: 5.4 Iterative Prototype Mining

Original: Since one IPMT layer can update the intermediate prototype G , the query feature map F q , and the query segmentation mask P q , we can iteratively perform this process and obtain better and better G and F q , finally making the segmentation results effectively boosted.
Revised: Since one IPMT layer can update the intermediate prototype G , the query feature map F q , and the query segmentation mask P q , we can iteratively perform this process and obtain better and better G and F q , finally making the segmentation results effectively boosted.

Original: Suppose we have L iterative
Revised: Suppose we have L iterative

Original: IPMT layers, then for each layer l we have:
Revised: IPMT layers, then for each layer l we have:

Original: [Equation]
Revised: [Equation]

Original: which can be broken down into the following steps:
Revised: which can be broken down into the following steps:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, since the masked attention requires a binary mask as the input, we use 0.5 as the threshold to generate P q l .
Revised: Here, since the masked attention requires a binary mask as the input, we use 0.5 as the threshold to generate P q l .

Original:  Few-shot Semantic Segmentation Model
Revised:  Few-shot Semantic Segmentation Model

Original: Following previous works [28, 40], we input query and support images into a fixed and shared encoder backbone such as the ResNet family [9] to obtain multi-level features.
Revised: Following previous works [28, 40], we input query and support images into a fixed and shared encoder backbone such as the ResNet family [9] to obtain multi-level features.

Original: Then, we concatenate the outputs of the third and fourth encoder blocks together and then adopt a 1 × 1 convolutional layer to generate middle-level query and support features, respectively.
Revised: Then, we concatenate the outputs of the third and fourth encoder blocks together and then adopt a 1 × 1 convolutional layer to generate middle-level query and support features, respectively.

Original: We also calculate the similarity between the high-level query and support features at the fifth encoder block to produce a prior mask and use masked average pooling on the support feature map to obtain a support prototype.
Revised: We also calculate the similarity between the high-level query and support features at the fifth encoder block to produce a prior mask and use masked average pooling on the support feature map to obtain a support prototype.

Original: Next, the query feature map, prior mask, and the expanded prototype are concatenated and transformed using a 1 × 1 convolutional layer to obtain the initial query feature F q 0 .
Revised: Next, the query feature map, prior mask, and the expanded prototype are concatenated and transformed using a 1 × 1 convolutional layer to obtain the initial query feature F q 0 .

Original: We also concatenate the middle-level support feature with the expanded prototype to generate the support feature F s .
Revised: We also concatenate the middle-level support feature with the expanded prototype to generate the support feature F s .

Original: All the above processes are common methods in FSS and termed prototype activation (PA) in Figure 2.
Revised: All the above processes are common methods in FSS and termed prototype activation (PA) in Figure 2.

Original: For more details please refer to [28].
Revised: For more details please refer to [28].

Original: As for the iterative learning, we feed F q 0 into two convolutional layers to obtain the initial query segmentation prediction P q 0 .
Revised: As for the iterative learning, we feed F q 0 into two convolutional layers to obtain the initial query segmentation prediction P q 0 .

Original: The initial intermediate prototype G 0 is randomly initialized at the beginning of the training and then optimized on the training set.
Revised: The initial intermediate prototype G 0 is randomly initialized at the beginning of the training and then optimized on the training set.

Original: Next, we feed G 0 , F s , F q 0 , M s , and
Revised: Next, we feed G 0 , F s , F q 0 , M s , and

Original: P q 0 into our iterative IPMT layers to perform intermediate prototype mining.
Revised: P q 0 into our iterative IPMT layers to perform intermediate prototype mining.

Original: After L iterations, the final activated query feature F q L is used to predict the final segmentation result via two convolutional layers.
Revised: After L iterations, the final activated query feature F q L is used to predict the final segmentation result via two convolutional layers.

Original: The dice loss [20] is used here to optimize the training of the whole model.
Revised: The dice loss [20] is used here to optimize the training of the whole model.

Original: [Table]
Revised: [Table]

Original:  Experiments 7.1 Datasets and Evaluation Metrics
Revised:  Experiments 7.1 Datasets and Evaluation Metrics

Original: Datasets.
Revised: Datasets.

Original: To make a fair comparison with previous works
Revised: To make a fair comparison with previous works

Original: [Table]
Revised: [Table]

Original: our model is evaluated on two few-shot semantic segmentation benchmark datasets,
Revised: our model is evaluated on two few-shot semantic segmentation benchmark datasets,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e ., the PASCAL- 5 i dataset[25] and the COCO- 20 i dataset [22].
Revised: e ., the PASCAL- 5 i dataset[25] and the COCO- 20 i dataset [22].

Original: PASCAL- 

Revised: PASCAL-

Original: i is constructed based on the PASCAL VOC 2012 dataset [6] and additional annotations from SDS [8].
Revised: i is constructed based on the PASCAL VOC 2012 dataset [6] and additional annotations from SDS [8].

Original: It contains 20 categories in total and these categories are partitioned into four folds as in [32] for cross validation, where each fold contains five categories.
Revised: It contains 20 categories in total and these categories are partitioned into four folds as in [32] for cross validation, where each fold contains five categories.

Original: COCO- 20 i is a larger datasets based on the MSCOCO [13] dataset.
Revised: COCO- 20 i is a larger datasets based on the MSCOCO [13] dataset.

Original: Similar to the division in PASCAL- 5 i , the 80 categories in MSCOCO are also partitioned into four folds for cross validation, where each fold includes 20 categories.
Revised: Similar to the division in PASCAL- 5 i , the 80 categories in MSCOCO are also partitioned into four folds for cross validation, where each fold includes 20 categories.

Original: For both datasets, we train our model on three folds and evaluate it on the remaining one fold and perform cross validation.
Revised: For both datasets, we train our model on three folds and evaluate it on the remaining one fold and perform cross validation.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: Following previous methods [25, 26, 15, 17], we adopt the class mean intersection over union (mIoU) as a primary evaluation metric.
Revised: Following previous methods [25, 26, 15, 17], we adopt the class mean intersection over union (mIoU) as a primary evaluation metric.

Original: In addition, we also report the results of foreground-background IoU (FB-IoU) for comparison.
Revised: In addition, we also report the results of foreground-background IoU (FB-IoU) for comparison.

Original: 7.2 Implementation Details
Revised: 7.2 Implementation Details

Original: Following previous works, we adopt ResNet-50 and ResNet-101 [9] as our encoder backbone.
Revised: Following previous works, we adopt ResNet-50 and ResNet-101 [9] as our encoder backbone.

Original: Note that they are initialized by the weights pre-trained on ImageNet [24] and froze during training.
Revised: Note that they are initialized by the weights pre-trained on ImageNet [24] and froze during training.

Original: Our proposed IPMT is implemented using PyTorch [23] and all the experiments are conducted on one
Revised: Our proposed IPMT is implemented using PyTorch [23] and all the experiments are conducted on one

Original: NVIDIA RTX 3090 GPU.
Revised: NVIDIA RTX 3090 GPU.

Original: We use the same data augmentation setting as [28] for fair comparisons.
Revised: We use the same data augmentation setting as [28] for fair comparisons.

Original: Our model is trained for 200 epochs on PASCAL- 5 i while 50 epochs on COCO- 20 i , respectively, with the batchsize set to 4.
Revised: Our model is trained for 200 epochs on PASCAL- 5 i while 50 epochs on COCO- 20 i , respectively, with the batchsize set to 4.

Original: Two optimizers ( i .
Revised: Two optimizers ( i .

Original: e ., SGD and AdamW) are used to train our model.
Revised: e ., SGD and AdamW) are used to train our model.

Original: The former is used to optimize the convolutional layers with the initial learning rate, weight decay, and momentum set to 2 .
Revised: The former is used to optimize the convolutional layers with the initial learning rate, weight decay, and momentum set to 2 .

Original: 5 × 10 − 3 , 0.0001, and 0.9, respectively.
Revised: 5 × 10 − 3 , 0.0001, and 0.9, respectively.

Original: The latter is used to optimize the transformer layers by setting the learning rate to 1 × 10 − 4 and the weight decay to 1 × 10 − 2 .
Revised: The latter is used to optimize the transformer layers by setting the learning rate to 1 × 10 − 4 and the weight decay to 1 × 10 − 2 .

Original: We also use the polynomial annealing policy with the power set to 0.9 to decay the learning rate.
Revised: We also use the polynomial annealing policy with the power set to 0.9 to decay the learning rate.

Original: For hyper-parameters in our IPMT, the number of multi-heads and the channel dimension of the image features are set to 8 and 256, respectively.
Revised: For hyper-parameters in our IPMT, the number of multi-heads and the channel dimension of the image features are set to 8 and 256, respectively.

Original: The weight α in DSL is set to 0.3 since a larger weight should be given for the more reliable category knowledge from the support images.
Revised: The weight α in DSL is set to 0.3 since a larger weight should be given for the more reliable category knowledge from the support images.

Original: During the evaluation, we follow [28] to randomly sample 1000 support-query pairs on PASCAL- 5 i andpairs on COCO- 20 i , respectively.
Revised: During the evaluation, we follow [28] to randomly sample 1000 support-query pairs on PASCAL- 5 i andpairs on COCO- 20 i , respectively.

Original: 7.3 Comparison with State
Revised: 7.3 Comparison with State

Original: -of-the-art Methods
Revised: -of-the-art Methods

Original: Quantitative comparison.
Revised: Quantitative comparison.

Original: As shown in Table 1 and 2, we compare our method with previous works on both PASCAL- 5 i and COCO- 20 i , respectively.
Revised: As shown in Table 1 and 2, we compare our method with previous works on both PASCAL- 5 i and COCO- 20 i , respectively.

Original: It can be found that our IPMT surpasses all other approaches by a large margin and achieves new state-of-the-art results.
Revised: It can be found that our IPMT surpasses all other approaches by a large margin and achieves new state-of-the-art results.

Original: On PASCAL- 5 i , when using ResNet-50 as the backbone, our proposed IPMT improves the mIoU score by 2.6 under the
Revised: On PASCAL- 5 i , when using ResNet-50 as the backbone, our proposed IPMT improves the mIoU score by 2.6 under the

Original: [Table]
Revised: [Table]

Original: 1-shot setting compared with the previous best result.
Revised: 1-shot setting compared with the previous best result.

Original: Additionally, we also achieve 1.8 mIoU (under the 1-shot setting) and 2.2 mIoU (5-shot setting) improvements using the ResNet-101 backbone.
Revised: Additionally, we also achieve 1.8 mIoU (under the 1-shot setting) and 2.2 mIoU (5-shot setting) improvements using the ResNet-101 backbone.

Original: As for COCO- 20 i , our method with the ResNet-50 backbone outperforms the previous best results by 1.6 and 0.6 mIoU under the two settings, respectively.
Revised: As for COCO- 20 i , our method with the ResNet-50 backbone outperforms the previous best results by 1.6 and 0.6 mIoU under the two settings, respectively.

Original: When using the ResNet-101 backbone, our
Revised: When using the ResNet-101 backbone, our

Original: IPMT improves the mIoU score by 3.5 and 4.7 over the previous best results.
Revised: IPMT improves the mIoU score by 3.5 and 4.7 over the previous best results.

Original: These improvements demonstrate the competitiveness of our model on complex data.
Revised: These improvements demonstrate the competitiveness of our model on complex data.

Original: In addition, we report the comparison with some advanced approaches in terms of the FB-IoU score in Table 3, which also shows our superiority.
Revised: In addition, we report the comparison with some advanced approaches in terms of the FB-IoU score in Table 3, which also shows our superiority.

Original: Quantitative Result.
Revised: Quantitative Result.

Original: We visualize some prediction results of our method and a support-only FSS method ( i .
Revised: We visualize some prediction results of our method and a support-only FSS method ( i .

Original: e ., CyCTR [40]) in Figure 3.
Revised: e ., CyCTR [40]) in Figure 3.

Original: It can be observed that our results (the 3 row) could effectively relieve the segmentation error caused by the inherent intra-class diversity compared with the results of only using the support information (the 2th row).
Revised: It can be observed that our results (the 3 row) could effectively relieve the segmentation error caused by the inherent intra-class diversity compared with the results of only using the support information (the 2th row).

Original: 7.4 Ablation Study
Revised: 7.4 Ablation Study

Original: In this section, we report ablation study results on PASCAL- 

Revised: In this section, we report ablation study results on PASCAL-

Original: i with the ResNet-50 backbone under the 1-shot setting.
Revised: i with the ResNet-50 backbone under the 1-shot setting.

Original: Effectiveness of IPM.
Revised: Effectiveness of IPM.

Original: To demonstrate the necessity of bridging the category information gap between query and support images using the proposed IPM, we conduct ablation study by learning the category information only from support or query images.
Revised: To demonstrate the necessity of bridging the category information gap between query and support images using the proposed IPM, we conduct ablation study by learning the category information only from support or query images.

Original: Furthermore, we report the performance comparison both with and without iteration.
Revised: Furthermore, we report the performance comparison both with and without iteration.

Original: All the ablation studies are conducted with both DSL and
Revised: All the ablation studies are conducted with both DSL and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: QA for a fair comparison.
Revised: QA for a fair comparison.

Original: As shown in Table 4, ‘Support only’ and ‘Query only’ mean the learnable prototype is only updated by the support feature and the query feature, respectively.
Revised: As shown in Table 4, ‘Support only’ and ‘Query only’ mean the learnable prototype is only updated by the support feature and the query feature, respectively.

Original: ‘Intermediate’ means using both support and query information in our IPM.
Revised: ‘Intermediate’ means using both support and query information in our IPM.

Original: We observe that our method surpasses other schemes by 1.6 mIoU when not using iteration.
Revised: We observe that our method surpasses other schemes by 1.6 mIoU when not using iteration.

Original: This improvement even increases to 3.4 when using iteration.
Revised: This improvement even increases to 3.4 when using iteration.

Original: These results clearly demonstrate the effectiveness of our IPM and also indicate that using iteration is an effective booster for our IPM.
Revised: These results clearly demonstrate the effectiveness of our IPM and also indicate that using iteration is an effective booster for our IPM.

Original: Ablation on Different Numbers of IPMT Layers.
Revised: Ablation on Different Numbers of IPMT Layers.

Original: We vary the number of IPMT layers from 1 to 5 and report the results in Table 5.
Revised: We vary the number of IPMT layers from 1 to 5 and report the results in Table 5.

Original: It shows that using more layers can gradually improve the model performance and using five layers achieves as much as 2.7 mIoU improvement.
Revised: It shows that using more layers can gradually improve the model performance and using five layers achieves as much as 2.7 mIoU improvement.

Original: We did not try more layers considering the accuracy-efficiency trade-off.
Revised: We did not try more layers considering the accuracy-efficiency trade-off.

Original: Effectiveness of DSL and QA.
Revised: Effectiveness of DSL and QA.

Original: Since the effectiveness of the IPM has been proved in Table 4, here we conduct an ablation study to validate the effectiveness of QA and DSL.
Revised: Since the effectiveness of the IPM has been proved in Table 4, here we conduct an ablation study to validate the effectiveness of QA and DSL.

Original: We remove all the three components from our model as the baseline model (only with PA), which only uses the support prototype to directly segment the target object like [28].
Revised: We remove all the three components from our model as the baseline model (only with PA), which only uses the support prototype to directly segment the target object like [28].

Original: As Table 6 shows, compared to the baseline, solely using IPM leads to 5.3 mIoU drop.
Revised: As Table 6 shows, compared to the baseline, solely using IPM leads to 5.3 mIoU drop.

Original: However, when DSL is added, our model achieves 4.1 mIoU improvement over the baseline.
Revised: However, when DSL is added, our model achieves 4.1 mIoU improvement over the baseline.

Original: This phenomenon is reasonable because there is no guarantee that the learnable prototype in IPM will learn intermediate category knowledge without DSL.
Revised: This phenomenon is reasonable because there is no guarantee that the learnable prototype in IPM will learn intermediate category knowledge without DSL.

Original: Meanwhile, using the QA to activate the query feature map leads to further 2.5 mIoU improvement.
Revised: Meanwhile, using the QA to activate the query feature map leads to further 2.5 mIoU improvement.

Original: These results clearly verify the effectiveness of our proposed QA and DSL.
Revised: These results clearly verify the effectiveness of our proposed QA and DSL.

Original: Prototype Comparison.
Revised: Prototype Comparison.

Original: We first visualize the overall distribution of the support (orange points) and intermediate (blue points) prototypes given two query (magenta points) images in Figure 4 (a) and (c).
Revised: We first visualize the overall distribution of the support (orange points) and intermediate (blue points) prototypes given two query (magenta points) images in Figure 4 (a) and (c).

Original: It is clearly observed that our intermediate prototypes are closer to the query prototypes than the support ones are in the feature space, hence verifying that our method effectively relieves the intra-class diversity issue and bridges the category information gap between query and support images.
Revised: It is clearly observed that our intermediate prototypes are closer to the query prototypes than the support ones are in the feature space, hence verifying that our method effectively relieves the intra-class diversity issue and bridges the category information gap between query and support images.

Original: We also show the support images for some specific points and the query images in Figure(b) and (d).
Revised: We also show the support images for some specific points and the query images in Figure(b) and (d).

Original: Please note that the support and intermediate prototypes of the same support image are shown as two points with the same shape.
Revised: Please note that the support and intermediate prototypes of the same support image are shown as two points with the same shape.

Original: We find that for these support images which have clear intra-class differences from the query image, our generated intermediate prototypes are successfully pulled to a closer feature space with the query prototype.
Revised: We find that for these support images which have clear intra-class differences from the query image, our generated intermediate prototypes are successfully pulled to a closer feature space with the query prototype.

Original: Additionally, for evaluating the intra-class diversity objectively, we adopt the Euclidean distance as a metric to measure the distance between the query prototype and the support prototype ( D qs ) in each episode.
Revised: Additionally, for evaluating the intra-class diversity objectively, we adopt the Euclidean distance as a metric to measure the distance between the query prototype and the support prototype ( D qs ) in each episode.

Original: Then, to further demonstrate the effectiveness of our method, we also measure the distance between the query prototype and the intermediate prototype ( D qi ) and the distance between the intermediate prototype and the support prototype ( D is ).
Revised: Then, to further demonstrate the effectiveness of our method, we also measure the distance between the query prototype and the intermediate prototype ( D qi ) and the distance between the intermediate prototype and the support prototype ( D is ).

Original: The average distances of each fold and the mean of all the categories on the PASCAL- 5 i are shown in Table 7.
Revised: The average distances of each fold and the mean of all the categories on the PASCAL- 5 i are shown in Table 7.

Original: From the table, we can clearly see that D qi is smaller than D qs on all folds, which means that our mined intermediate prototype is more similar to the query than the support is.
Revised: From the table, we can clearly see that D qi is smaller than D qs on all folds, which means that our mined intermediate prototype is more similar to the query than the support is.

Original: This also demonstrates that our method effectively reduces the distance between the mined prototype and the query and mitigates the intra-class diversity problem.
Revised: This also demonstrates that our method effectively reduces the distance between the mined prototype and the query and mitigates the intra-class diversity problem.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we focus on the intra-class diversity between query and support and introduce an intermediate prototype to bridge the category information gap between them.
Revised: In this paper, we focus on the intra-class diversity between query and support and introduce an intermediate prototype to bridge the category information gap between them.

Original: The core idea is to use the intermediate prototype to aggregate the support-deterministic and query-adaptive category information by our designed Intermediate Prototype Mining Transformer (IPMT) in an iterative way.
Revised: The core idea is to use the intermediate prototype to aggregate the support-deterministic and query-adaptive category information by our designed Intermediate Prototype Mining Transformer (IPMT) in an iterative way.

Original: Surprisingly, despite its simplicity, our method outperforms previous state-of-the-art results by a large margin on two FSS benchmark datasets.
Revised: Surprisingly, despite its simplicity, our method outperforms previous state-of-the-art results by a large margin on two FSS benchmark datasets.

Original: We hope our work could inspire future research to concentrate more on the intra-class diversity in FSS.
Revised: We hope our work could inspire future research to concentrate more on the intra-class diversity in FSS.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was supported in part by the National Key R&D Program of China under Grant 2020AAA0105701; the National Natural Science Foundation of China under Grant 62036011, U20B2065, 32036005, 6202781, 62136007.
Revised: This work was supported in part by the National Key RD Program of China under Grant 2020AAA0105701, the National Natural Science Foundation of China under Grant 62136007, 62071388, 62036011 and U20B2065, the Key Research and Development Program of Shaanxi Province under Grant 2021ZDLGY01-08.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section 1.
Revised: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code will be provided in the supplemental material.
Revised: [Yes] The code will be provided in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 7.1 and 7.2.
Revised: [Yes] See Section 7.1 and 7.2.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We followed previous FSS works and reported the averaged results over 5 independent runs but did not report error bars.
Revised: [No] We followed previous FSS works and reported the averaged results over 5 independent runs but did not report error bars.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 7.2. 4.
Revised: [Yes] See Section 7.2. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 7.1.
Revised: [Yes] See Section 7.1.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [No] The data is open source.
Revised: [No] The data is open source.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N /A]
Revised: [N/A]

Original: Appendix A Limitation and Societal Impacts
Revised: Appendix A Limitation and Societal Impacts

Original: Limitation.
Revised: Limitation.

Original: We observe that the performance of our method does not achieve a significant improvement under the 5-shot setting.
Revised: We observe that the performance of our method does not achieve a significant improvement under the 5-shot setting.

Original: We argue that this is reasonable because as the number of the support images increases, their internal diversity probably increases.
Revised: We argue that this is reasonable because as the number of the support images increases, their internal diversity probably increases.

Original: As a result, the intra-class diversity between support and query images probably decreases and their category information gap will be narrowed.
Revised: As a result, the intra-class diversity between support and query images probably decreases and their category information gap will be narrowed.

Original: Thus, the ability of our method to correct the support-query deviation will be weakened.
Revised: Thus, the ability of our method to correct the support-query deviation will be weakened.

Original: However, our work still performs very effectively under the 1-shot setting and provides a new perspective for future research.
Revised: However, our work still performs very effectively under the 1-shot setting and provides a new perspective for future research.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: The method proposed in this paper is applied for few-shot semantic segmentation which has many applications in robot vision systems, image recognition software, etc.
Revised: The method proposed in this paper is applied for few-shot semantic segmentation which has many applications in robot vision systems, image recognition software, etc.

Original: Compared with previous methods, although our method achieves a large performance improvement, it is still hard to come up with the human cognitive ability.
Revised: Compared with previous methods, although our method achieves a large performance improvement, it is still hard to come up with the human cognitive ability.

Original: For some safety-critical fields, such as medical treatment and autonomous driving, it is risky if we apply few-shot semantic segmentation methods directly.
Revised: For some safety-critical fields, such as medical treatment and autonomous driving, it is risky if we apply few-shot semantic segmentation methods directly.

Original: B Visualization of Extensive Ablative Analysis
Revised: B Visualization of Extensive Ablative Analysis

Original: Effectiveness of IPM.
Revised: Effectiveness of IPM.

Original: As a supplement, we show some qualitative comparison results to prove the effectiveness of our IPM in an intuitionistic way.
Revised: As a supplement, we show some qualitative comparison results to prove the effectiveness of our IPM in an intuitionistic way.

Original: From left to right in Figure 5, the 3 rd and 4 th columns show the predictions of only using the query feature to update the learnable prototype.
Revised: From left to right in Figure 5, the 3 rd and 4 th columns show the predictions of only using the query feature to update the learnable prototype.

Original: We find that the predicted masks are often confused with other distracting objects.
Revised: We find that the predicted masks are often confused with other distracting objects.

Original: Predictions generated by only using the support feature are shown in the 5 th and 6 th columns, which also can not handle the intra-class difference between support and query images.
Revised: Predictions generated by only using the support feature are shown in the 5 th and 6 th columns, which also can not handle the intra-class difference between support and query images.

Original: Using the intermediate prototype proposed in IPM can relieve the bias of the unidirectional category knowledge.
Revised: Using the intermediate prototype proposed in IPM can relieve the bias of the unidirectional category knowledge.

Original: The last column shows that we get the best results when using the intermediate prototype with iterations.
Revised: The last column shows that we get the best results when using the intermediate prototype with iterations.

Original: Visualization of Using Different Numbers of IPMT Layers.
Revised: Visualization of Using Different Numbers of IPMT Layers.

Original: We further show some qualitative results of stacking different IPMT layers in Figure 6 to prove the effectiveness of our proposed iterative prototype mining scheme.
Revised: We further show some qualitative results of stacking different IPMT layers in Figure 6 to prove the effectiveness of our proposed iterative prototype mining scheme.

Original: From the 3 rd column to the 7 th column, we observe that the accuracy of the predictions is gradually improved by stacking more IPMT layers and the distraction from other objects is relieved.
Revised: From the 3 rd column to the 7 th column, we observe that the accuracy of the predictions is gradually improved by stacking more IPMT layers and the distraction from other objects is relieved.

Original: When setting the number of the layers to five, the prediction quality is the best.
Revised: When setting the number of the layers to five, the prediction quality is the best.

Original: C Additional Qualitative Results
Revised: C Additional Qualitative Results

Original: We give more qualitative results in Figure 7 to show the good performance of our IPMT.
Revised: We give more qualitative results in Figure 7 to show the good performance of our IPMT.