Original: S UPPRESSING THE H ETEROGENEITY : A S TRONG F EATURE E XTRACTOR FOR F EW - SHOT S EGMENTATION
Revised: S UPPRESSING THE H ETEROGENEITY : A S TRONG F EATURE E XTRACTOR FOR F EW - SHOT S EGMENTATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper tackles the Few-shot Semantic Segmentation (FSS) task with focus on learning the feature extractor.
Revised: This paper tackles the Few-shot Semantic Segmentation (FSS) task with focus on learning the feature extractor.

Original: Somehow the feature extractor has been overlooked by recent state-of-the-art methods, which directly use a deep model pretrained on
Revised: Somehow the feature extractor has been overlooked by recent state-of-the-art methods, which directly use a deep model pretrained on

Original: ImageNet for feature extraction (without further fine-tuning).
Revised: ImageNet for feature extraction (without further fine-tuning).

Original: Under this background, we think the FSS feature extractor deserves exploration and observe the heterogeneity ( i.e. , the intra-class diversity in the raw images) as a critical challenge hindering the intra-class feature compactness.
Revised: Under this background, we think the FSS feature extractor deserves exploration and observe the heterogeneity ( i.e. , the intra-class diversity in the raw images) as a critical challenge hindering the intra-class feature compactness.

Original: The heterogeneity has three levels from coarse to fine: 1) Sample-level: the inevitable distribution gap between the support and query images makes them heterogeneous from each other. 2)
Revised: The heterogeneity has three levels from coarse to fine: 1) Sample-level: the inevitable distribution gap between the support and query images makes them heterogeneous from each other. 2)

Original: Region-level: the background in FSS actually contains multiple regions with different semantics.
Revised: Region-level: the background in FSS actually contains multiple regions with different semantics.

Original: 3) Patch-level: some neighboring patches belonging to a same class may appear quite different from each other.
Revised: 3) Patch-level: some neighboring patches belonging to a same class may appear quite different from each other.

Original: Motivated by these observations, we propose a feature extractor with Multi-level Heterogeneity Suppressing (MuHS).
Revised: Motivated by these observations, we propose a feature extractor with Multi-level Heterogeneity Suppressing (MuHS).

Original: MuHS leverages the attention mechanism in transformer backbone to effectively suppress all these three-level heterogeneity.
Revised: MuHS leverages the attention mechanism in transformer backbone to effectively suppress all these three-level heterogeneity.

Original: Concretely, MuHS reinforces the attention / interaction between different samples (query and support), different regions and neighboring patches by constructing cross-sample attention, cross-region interaction and a novel masked image segmentation (inspired by the recent masked image modeling), respectively.
Revised: Concretely, MuHS reinforces the attention / interaction between different samples (query and support), different regions and neighboring patches by constructing cross-sample attention, cross-region interaction and a novel masked image segmentation (inspired by the recent masked image modeling), respectively.

Original: We empirically show that 1) MuHS brings consistent improvement for various FSS heads and 2) using a simple linear classification head, MuHS sets new states of the art on multiple FSS datasets, validating the importance of FSS feature learning.
Revised: We empirically show that 1) MuHS brings consistent improvement for various FSS heads and 2) using a simple linear classification head, MuHS sets new states of the art on multiple FSS datasets, validating the importance of FSS feature learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Few-shot semantic segmentation (FSS) aims to generalize the semantic segmentation model from base classes to novel classes, using very few support samples.
Revised: Few-shot semantic segmentation (FSS) aims to generalize the semantic segmentation model from base classes to novel classes, using very few support samples.

Original: FSS depicts a potential to reduce the notoriously expensive pixel-wise annotation and has thus drawn great research interest.
Revised: FSS depicts a potential to reduce the notoriously expensive pixel-wise annotation and has thus drawn great research interest.

Original: However, we observe that the current research has been biased towards partial component of the FSS framework.
Revised: However, we observe that the current research has been biased towards partial component of the FSS framework.

Original: Concretely, an FSS framework typically consists of a feature extractor and a matching head, while the recent state-of-the-art methods (Zhang et al.
Revised: Concretely, an FSS framework typically consists of a feature extractor and a matching head, while the recent state-of-the-art methods (Zhang et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: (2020b)
Revised: (2020b)

Original: ; Li et al.
Revised: ; Li et al.

Original:  ; Xie et al. (2021b); Wu et al. (2021); Zhang et al. (2021a); Wang et al.
Revised: (2021a) ; Xie et al. (2021b); Wu et al. (2021); Zhang et al. (2021a); Li et al.

Original: all focus on the matching head.
Revised: all focus on the matching head.

Original: They pay NO effort on learning the feature extractor and adopt a ImageNet-pretrained model without any fine-tuning.
Revised: They pay NO effort on learning the feature extractor and adopt a ImageNet-pretrained model without any fine-tuning.

Original: Under this background, we think the FSS feature extractor deserves exploration and take a rethink on the corresponding challenge.
Revised: Under this background, we think the FSS feature extractor deserves exploration and take a rethink on the corresponding challenge.

Original: Some prior literature (Tian et al.
Revised: Some prior literature (Tian et al.

Original: (2020b)
Revised: (2020b)

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021b)) argue that the challenge is mainly because the limited support samples are insufficient for finetuning a large feature extractor ( e.g. , ResNet-50 (He et al.
Revised: (2021b)) argue that the challenge is mainly because the limited support samples are insufficient for finetuning a large feature extractor ( e.g. , ResNet-50 (He et al.

Original: (2016))), therefore leading to the overfitting problem.
Revised: (2016))), therefore leading to the overfitting problem.

Original: We hold a different perspective and observe the heterogeneity ( i.e. , the intra-class diversity in the raw images) as a critical challenge hindering the intra-class compactness of FSS features.
Revised: We hold a different perspective and observe the heterogeneity ( i.e. , the intra-class diversity in the raw images) as a critical challenge hindering the intra-class compactness of FSS features.

Original: Although the heterogeneity is not a unique problem in FSS ( e.g. , it does exist in the  canonical segmentation as well), its challenge is significantly amplified by the few-shot setting.
Revised: Although the heterogeneity is not a unique problem in FSS ( e.g. , it does exist in the [Figure] canonical segmentation as well), its challenge is significantly amplified by the few-shot setting.

Original: In our viewpoint, the heterogeneity has three levels from coarse to fine:
Revised: In our viewpoint, the heterogeneity has three levels from coarse to fine:

Original: • Sample-level heterogeneity exists between the query and support images due to their distribution gap.
Revised: • Sample-level heterogeneity exists between the query and support images due to their distribution gap.

Original: For example, in

Original:  Fig. 1 (a), the foreground objects (“cow” and “cattle” in the support and query images look quite different, although they both belong to a same semantic class “cow”.
Revised: For example, in Fig. 1 (a), the foreground objects (“cow” and “cattle”) in the support and query images look quite different, although they both belong to a same semantic class “cow”.

Original: [Figure]

Original: • Region-level heterogeneity exists (mostly) in the background, which actually contains multiple regions with different semantics.
Revised: • Region-level heterogeneity exists (mostly) in the background, which actually contains multiple regions with different semantics.

Original: In Fig.
Revised: In Fig.

Original: 1 (b), “horse” in the image is a foreground region when the support object is another horse.
Revised: 1 (b), “horse” in the image is a foreground region when the support object is another horse.

Original: However, when the support object shifts to a “rider”, the horse in the image should be merged into the background, resulting in the region-level heterogeneity.
Revised: However, when the support object shifts to a “rider”, the horse in the image should be merged into the background, resulting in the region-level heterogeneity.

Original: • Patch-level heterogeneity exists among neighboring patches which belong to a same semantic class but have significant appearance variations.
Revised: • Patch-level heterogeneity exists among neighboring patches which belong to a same semantic class but have significant appearance variations.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 1 (c), the upper and lower body of a single person are in different colors, therefore introducing patch-level heterogeneity.
Revised: 1 (c), the upper and lower body of a single person are in different colors, therefore introducing patch-level heterogeneity.

Original: Motivated by these observations, we propose an FSS feature extractor with Multi-level Heterogeneity Suppressing (MuHS).
Revised: Motivated by these observations, we propose an FSS feature extractor with Multi-level Heterogeneity Suppressing (MuHS).

Original: MuHS adopts the transformer backbone and leverages the attention mechanism to suppress all these three-level heterogeneity.
Revised: MuHS adopts the transformer backbone and leverages the attention mechanism to suppress all these three-level heterogeneity.

Original: Our choice of using the transformer backbone is natural: the attention mechanism provides strong potential for constructing long-range dependencies across samples, regions and patches.
Revised: Our choice of using the transformer backbone is natural: the attention mechanism provides strong potential for constructing long-range dependencies across samples, regions and patches.

Original: Concretely, MuHS reinforces the attention / interaction between different samples (query and support), different regions and neighboring patches by constructing cross-sample attention, cross-region interaction and a novel masked image segmentation, respectively.
Revised: Concretely, MuHS reinforces the attention / interaction between different samples (query and support), different regions and neighboring patches by constructing cross-sample attention, cross-region interaction and a novel masked image segmentation, respectively.

Original: To be more specific, these attention / interaction are as below:
Revised: To be more specific, these attention / interaction are as below:

Original: (i) Cross-Sample Attention .
Revised: (i) Cross-Sample Attention .

Original: In popular transformers, the attention is within each single sample and does not cross multiple samples.
Revised: In popular transformers, the attention is within each single sample and does not cross multiple samples.

Original: In contrast, MuHS constructs cross-sample attention with a novel design of “linking” tokens.
Revised: In contrast, MuHS constructs cross-sample attention with a novel design of “linking” tokens.

Original: In each transformer layer, we use some linking tokens to connect all the patch tokens from the query and support samples simultaneously, therefore efficiently propagating information across different samples.
Revised: In each transformer layer, we use some linking tokens to connect all the patch tokens from the query and support samples simultaneously, therefore efficiently propagating information across different samples.

Original: (ii) Cross-Region Interaction .
Revised: (ii) Cross-Region Interaction .

Original: In popular transformers, the attention usually encourages feature interaction (absorption) between similar patch tokens.
Revised: In popular transformers, the attention usually encourages feature interaction (absorption) between similar patch tokens.

Original: In contrary to this common practice, MuHS enforces additional feature absorption between patch tokens from dissimilar regions in the background.
Revised: In contrary to this common practice, MuHS enforces additional feature absorption between patch tokens from dissimilar regions in the background.

Original: Such a cross-region interaction smooths the background and suppresses the region-level heterogeneity.
Revised: Such a cross-region interaction smooths the background and suppresses the region-level heterogeneity.

Original: (iii) Masked Image Segmentation .
Revised: (iii) Masked Image Segmentation .

Original: Inspired by the recent masked image modeling (MIM), MuHS randomly masks some patch tokens and makes partial prediction for the existing patches.
Revised: Inspired by the recent masked image modeling (MIM), MuHS randomly masks some patch tokens and makes partial prediction for the existing patches.

Original: Afterwards, MuHS fills trainable mask tokens and encourages the deep model to make the holistic prediction for complete patches, yielding a novel masked image segmentation.
Revised: Afterwards, MuHS fills trainable mask tokens and encourages the deep model to make the holistic prediction for complete patches, yielding a novel masked image segmentation.

Original: The learned capacity of inferring the semantics of the masked patches from neighboring patches suppresses the patch-level heterogeneity.
Revised: The learned capacity of inferring the semantics of the masked patches from neighboring patches suppresses the patch-level heterogeneity.

Original: In MuHS, the above three components respectively mitigate a corresponding type of heterogeneity and achieve complementary benefits for few-shot semantic segmentation.
Revised: In MuHS, the above three components respectively mitigate a corresponding type of heterogeneity and achieve complementary benefits for few-shot semantic segmentation.

Original: Empirically, we show that using MuHS to replace the frozen feature extractor (pretrained on ImageNet) brings consistent improvement for multiple popular FSS heads.
Revised: Empirically, we show that using MuHS to replace the frozen feature extractor (pretrained on ImageNet) brings consistent improvement for multiple popular FSS heads.

Original: Importantly, since the MuHS feature has relatively good intra-class compactness, we simply cooperate it with a linear classification head and achieve new state of the art on multiple FSS datasets.
Revised: Importantly, since the MuHS feature has relatively good intra-class compactness, we simply cooperate it with a linear classification head and achieve new state of the art on multiple FSS datasets.

Original: For example, on PASCAL- 5 i , MuHS achieves 69.1% mIoU under 1-shot setting.
Revised: For example, on PASCAL- 5 i , MuHS achieves 69.1% mIoU under 1-shot setting.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: First, we shift the FSS research focus from the matching head to the feature extractor and reveal the heterogeneity as an important challenge.
Revised: First, we shift the FSS research focus from the matching head to the feature extractor and reveal the heterogeneity as an important challenge.

Original: Second.
Revised: Second.

Original: we propose Multi-level Heterogeneity Suppressing (MuHS).
Revised: we propose Multi-level Heterogeneity Suppressing (MuHS).

Original: MuHS utilizes novel crosssample attention, cross-region interaction and masked image segmentation to suppress the heterogeneity from three levels.
Revised: MuHS utilizes novel crosssample attention, cross-region interaction and masked image segmentation to suppress the heterogeneity from three levels.

Original: Third, we conduct extensive experiments to validate the effectiveness of the proposed MuHS.
Revised: Third, we conduct extensive experiments to validate the effectiveness of the proposed MuHS.

Original: Experimental results confirm that MuHS is compatible to multiple FSS heads and achieves new state of the art using a simple linear classification head.
Revised: Experimental results confirm that MuHS is compatible to multiple FSS heads and achieves new state of the art using a simple linear classification head.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Few-shot Segmentation Methods With Focus On Matching Head.
Revised: Few-shot Segmentation Methods With Focus On Matching Head.

Original: The recent state-of-the-art
Revised: The recent state-of-the-art

Original: FSS methods (Zhang et al.
Revised: FSS methods (Zhang et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: (2020b)
Revised: (2020b)

Original: ; Li et al.
Revised: ; Li et al.

Original:  ; Xie et al. (2021b); Wu et al. (2021); Zhang et al. (2021a); Lang et al. (2022); Yang et al.
Revised: (2021a) ; Xie et al. (2021b); Wu et al. (2021); Jiao et al. (2022); Lang et al. (2022); Yang et al.

Original: (2020); Siam et al.
Revised: (2020); Siam et al.

Original: (2019)) focus on learning the matching head (based on a frozen CNN feature extractor).
Revised: (2019)) focus on learning the matching head (based on a frozen CNN feature extractor).

Original: Some methods (Zhang et al.
Revised: Some methods (Zhang et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: (2020b)
Revised: (2020b)

Original: ; Li et al.
Revised: ; Li et al.

Original:  ; Xie et al. (2021b); Lang et al. (2022)) generate a prior-map based on the similarity between samples (query and support) and adopt convolution based matching heads to further improve the segmentation accuracy.
Revised: (2021a) ; Xie et al. (2021b); Lang et al. (2022)) generate a prior-map based on the similarity between samples (query and support) and adopt convolution based matching heads to further improve the segmentation accuracy.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021b)
Revised: (2021b)

Original: ; Lu et al.
Revised: ; Lu et al.

Original: (2021) proposed transformer-based matching head and perform attention mechanism to aggregate features from support to query.
Revised: (2021) proposed transformer-based matching head and perform attention mechanism to aggregate features from support to query.

Original: Moreover, some methods (Min et al.
Revised: Moreover, some methods (Min et al.

Original: ; Hong et al.
Revised: ; Hong et al.

Original: propose 4D convolutions to fully extract multi-level features.
Revised: propose 4D convolutions to fully extract multi-level features.

Original: Different from these recent progresses, this paper focuses on learning the feature extractor.
Revised: Different from these recent progresses, this paper focuses on learning the feature extractor.

Original: The proposed MuHS feature extractor brings general improvement for a battery of matching heads and achieves state-of-the-art accuracy with a simple linear classification head.
Revised: The proposed MuHS feature extractor brings general improvement for a battery of matching heads and achieves state-of-the-art accuracy with a simple linear classification head.

Original: Transformers for Visual Recognition.
Revised: Transformers for Visual Recognition.

Original: Recently, transformers are introduced to computer vision tasks, e.g. , image classification (Dosovitskiy et al.
Revised: Recently, transformers are introduced to computer vision tasks, e.g. , image classification (Dosovitskiy et al.

Original: Vaswani et al.
Revised: Vaswani et al.

Original: Liu et al.
Revised: Liu et al.

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021)), segmentation (Wang et al.
Revised: (2021)), segmentation (Wang et al.

Original: ; Xie et al. (2021a)), detection (Carion et al.
Revised: ; Xie et al. (2021a); Li et al. (2021b)), detection (Carion et al.

Original: (2020); Zhu et al.
Revised: Zhu et al.

Original: Bar et al.
Revised: Bar et al.

Original: and have shown promising performance.
Revised: and have shown promising performance.

Original: Under FSS scenario, we observe three-level heterogeneity ( i.e., sample-level, region-level, patchlevel), which hinders intra-class compactness of FSS features.
Revised: Under FSS scenario, we observe three-level heterogeneity ( i.e., sample-level, region-level, patchlevel), which hinders intra-class compactness of FSS features.

Original: We think the attention mechanism in the transformer provides strong potential for constructing long-range dependencies across samples, regions and patches.
Revised: We think the attention mechanism in the transformer provides strong potential for constructing long-range dependencies across samples, regions and patches.

Original: Therefore, the proposed MuHS adopts the transformer network as its backbone and utilizes the characteristics of transformer to suppress all these three-level heterogeneity in a unified framework.
Revised: Therefore, the proposed MuHS adopts the transformer network as its backbone and utilizes the characteristics of transformer to suppress all these three-level heterogeneity in a unified framework.

Original: Masked Image Modeling.
Revised: Masked Image Modeling.

Original: Masked modeling methods (Devlin et al.
Revised: Masked modeling methods (Devlin et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: (2018; 2019)) are wildly utilized in NLP tasks.
Revised: (2018; 2019)) are wildly utilized in NLP tasks.

Original: BERT (Devlin et al.
Revised: BERT (Devlin et al.

Original: (2018)) utilizes a “masked language model” (MLM) to randomly masks input and predict the original vocabulary id of the masked tokens.
Revised: (2018)) utilizes a “masked language model” (MLM) to randomly masks input and predict the original vocabulary id of the masked tokens.

Original: Motivated by BERT, BEIT (Bao et al.
Revised: Motivated by BERT, BEIT (Bao et al.

Original: (2021)) proposes “masked image modeling” (MIM) to perform self-supervised learning on Vision task.
Revised: (2021)) proposes “masked image modeling” (MIM) to perform self-supervised learning on Vision task.

Original: It randomly masks some proportion of image patches and replaces them with a mask tokens.
Revised: It randomly masks some proportion of image patches and replaces them with a mask tokens.

Original: Recently, SimMIM (Xie et al.
Revised: Recently, SimMIM (Xie et al.

Original: , MAE (He et al.
Revised: , MAE (He et al.

Original: (2022)) simplify the MIM designs and improves transformers.
Revised: (2022)) simplify the MIM designs and improves transformers.

Original: An important component of MuHS is inspired by the masked image modeling to suppress the patchlevel heterogeneity.
Revised: An important component of MuHS is inspired by the masked image modeling to suppress the patchlevel heterogeneity.

Original: Different from the popular self-supervised scheme, MIS is fully supervised.
Revised: Different from the popular self-supervised scheme, MIS is fully supervised.

Original: It masks out some query patches at the input and yet maintains the holistic prediction for segmentation, yielding a novel Masked Image Segmentation (MIS) learning. 
Revised: It masks out some query patches at the input and yet maintains the holistic prediction for segmentation, yielding a novel Masked Image Segmentation (MIS) learning. [Figure]

Original:  M ETHODS 3. O VERVIEW
Revised:  M ETHODS 3. O VERVIEW

Original: In the few-shot segmentation (FSS) task, we have a training set and a testing set with no labeling overlap.
Revised: In the few-shot segmentation (FSS) task, we have a training set and a testing set with no labeling overlap.

Original: The testing set consists of a support set S and a query set Q , which are from the same novel category (unseen in the training set).
Revised: The testing set consists of a support set S and a query set Q , which are from the same novel category (unseen in the training set).

Original: To set up a N -shot scenario, we randomly sample
Revised: To set up a N -shot scenario, we randomly sample

Original: [Figure] N support images {I 1 s , ..., I Ns } with corresponding masks {Y 1 s , ..., Y Ns } from S to recognize the same-semantic region in the query image I q ∈ Q .
Revised: N support images {I 1 s , ..., I Ns } with corresponding masks {Y 1 s , ..., Y Ns } from S to recognize the same-semantic region in the query image I q ∈ Q .

Original: During training, we follow the popular meta-learning scheme (Tian et al.
Revised: During training, we follow the popular meta-learning scheme (Tian et al.

Original: (2020a);
Revised: (2020a);

Original: Pambala et al.
Revised: Pambala et al.

Original: and construct a meta-task, i.e. , sampling N support and one query from the training set into each episode.
Revised: and construct a meta-task, i.e. , sampling N support and one query from the training set into each episode.

Original: The proposed MuHS feature extractor is based on the transformer backbone consisting of L transformer blocks, as illustrated in Fig.
Revised: The proposed MuHS feature extractor is based on the transformer backbone consisting of L transformer blocks, as illustrated in Fig.

Original: 2 (a).
Revised: 2 (a).

Original: Given an input image, we split it into non-overlapping image patches, linearly embed them into patch tokens, and feed the patch tokens into the MuHS feature extractor.
Revised: Given an input image, we split it into non-overlapping image patches, linearly embed them into patch tokens, and feed the patch tokens into the MuHS feature extractor.

Original: We utilize X q , X s to denote the embedding of query tokens and support tokens, respectively.
Revised: We utilize X q , X s to denote the embedding of query tokens and support tokens, respectively.

Original: Specifically for the query image, we randomly discard some patch tokens.
Revised: Specifically for the query image, we randomly discard some patch tokens.

Original: Based on the transformer backbone, MuHS has three major components as below:
Revised: Based on the transformer backbone, MuHS has three major components as below:

Original: 1) MuHS enforces a Cross-Sample Attention between the support and query through some linking tokens.
Revised: 1) MuHS enforces a Cross-Sample Attention between the support and query through some linking tokens.

Original: These linking tokens are trainable and update themselves by absorbing all query and support tokens simultaneously.
Revised: These linking tokens are trainable and update themselves by absorbing all query and support tokens simultaneously.

Original: In the following transformer block, the updated state of the linking tokens are absorbed by the query and support, respectively.
Revised: In the following transformer block, the updated state of the linking tokens are absorbed by the query and support, respectively.

Original: Therefore, it facilitates interaction between the query and support with relatively low computational cost.
Revised: Therefore, it facilitates interaction between the query and support with relatively low computational cost.

Original: The details are in Section 3.2. 2)
Revised: The details are in Section 3.2. 2)

Original: Within each transformer block, MuHS appends an additional Cross-Region Interaction upon the original self-attention layer.
Revised: Within each transformer block, MuHS appends an additional Cross-Region Interaction upon the original self-attention layer.

Original: While the original self-attention encourages interaction among similar patch tokens, the Cross-Region Interaction promotes interaction among dissimilar regions in the background.
Revised: While the original self-attention encourages interaction among similar patch tokens, the Cross-Region Interaction promotes interaction among dissimilar regions in the background.

Original: The details are in Section 3.3.3)
Revised: The details are in Section 3.3.

Original: Based on the output of the MuHS feature extractor, we use a matching head ( e.g. , the linear classification head) to make partial prediction for the existing patches of the query image (note that some query patches are discarded at the input layer).
Revised: Based on the output of the MuHS feature extractor, we use a matching head ( e.g. , the linear classification head) to make partial prediction for the existing patches of the query image (note that some query patches are discarded at the input layer).

Original: Afterwards, we fill trainable mask tokens to the incomplete query patch tokens, and input them into a Masked Image Segmentation model consisting of M transformer decoder.
Revised: Afterwards, we fill trainable mask tokens to the incomplete query patch tokens, and input them into a Masked Image Segmentation model consisting of M transformer decoder.

Original: The Masked Image Segmentation aims to make holistic prediction of the input query, regardless of the discarded patches.
Revised: The Masked Image Segmentation aims to make holistic prediction of the input query, regardless of the discarded patches.

Original: The details are in Section 3.4.
Revised: The details are in Section 3.4.

Original:  C ROSS -S AMPLE A TTENTION
Revised: 3. C ROSS -S AMPLE A TTENTION

Original: The Cross-Sample Attention mechanism constructs interaction between support and query so as to suppress the sample-level heterogeneity, as illustrated in Fig.
Revised: The Cross-Sample Attention mechanism constructs interaction between support and query so as to suppress the sample-level heterogeneity, as illustrated in Fig.

Original: 2 (c).
Revised: 2 (c).

Original: Specifically, we use linking tokens X link ∈ R C × D ( C embedding vectors with D dimensions ) to interact query and the support.
Revised: Specifically, we use linking tokens X link ∈ R C × D ( C embedding vectors with D dimensions ) to interact query and the support.

Original: At the input block, we initialize two linking tokens X 0 link with the mean features of the foreground and background region of the support sample.
Revised: At the input block, we initialize two linking tokens X 0 link with the mean features of the foreground and background region of the support sample.

Original: Then the linking tokens are updated block by block through cross-attention, which is formulated as:
Revised: Then the linking tokens are updated block by block through cross-attention, which is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where X i +1 link denotes the updated linking tokens, { , } denotes the concatenation operation.
Revised: where X i +1 link denotes the updated linking tokens, { , } denotes the concatenation operation.

Original: Que ,
Revised: Que ,

Original: Key , Val are the operation to calculate query embedding, key embedding and value embedding of support tokens X is , query tokens
Revised: Key , Val are the operation to calculate query embedding, key embedding and value embedding of support tokens X is , query tokens

Original: X iq and linking tokens X ilink in i -th MuHS Block, respectively.
Revised: X iq and linking tokens X ilink in i -th MuHS Block, respectively.

Original: Given the updated linking tokens, we then update the support and the query patch tokens by:
Revised: Given the updated linking tokens, we then update the support and the query patch tokens by:

Original: [Equation]
Revised: [Equation]

Original: where X is , X iq are the D dimensions embedding of support and query tokens in i -th Block.
Revised: where X is , X iq are the D dimensions embedding of support and query tokens in i -th Block.

Original: X i +1 link denotes the embedding of Linking tokens updated by Eq. 1.
Revised: X i +1 link denotes the embedding of Linking tokens updated by Eq. 1.

Original: Since the linking tokens already absorb information from all the query and support tokens (Eq. 1), in the subsequent Eq. 2, they propagate the absorbed information onto the support and the query tokens, therefore facilitating a mediate interaction between the support and query samples.
Revised: Since the linking tokens already absorb information from all the query and support tokens (Eq. 1), in the subsequent Eq. 2, they propagate the absorbed information onto the support and the query tokens, therefore facilitating a mediate interaction between the support and query samples.

Original: Compared with directly constructing patch-to-patch attention between the query and support sample, our solution with linking tokens has the advantage of high efficiency.
Revised: Compared with directly constructing patch-to-patch attention between the query and support sample, our solution with linking tokens has the advantage of high efficiency.

Original: Specifically, the patch-to-patch attention incurs quadratic complexity against the patch token numbers, while using the linking tokens only incurs linear complexity.
Revised: Specifically, the patch-to-patch attention incurs quadratic complexity against the patch token numbers, while using the linking tokens only incurs linear complexity.

Original: 3. C ROSS -R EGION I NTERACTION
Revised: 3. C ROSS -R EGION I NTERACTION

Original: The Cross-Region Interaction (Fig. 

Revised: The Cross-Region Interaction (Fig.

Original: (b)) is appended upon the self-attention layer in Eq.
Revised: (b)) is appended upon the self-attention layer in Eq.

Original: 2 within each MuHS block
Revised: 2 within each MuHS block

Original: to encourage the interaction between different regions in the background.
Revised: to encourage the interaction between different regions in the background.

Original: To this end, we use the ground-truth label to split the background in each query image (during training) into multiple regions.
Revised: To this end, we use the ground-truth label to split the background in each query image (during training) into multiple regions.

Original: Specifically, some regions in the background actually belong to some annotated foreground classes but are merged into the background because the current training episode focuses on a different foreground class.
Revised: Specifically, some regions in the background actually belong to some annotated foreground classes but are merged into the background because the current training episode focuses on a different foreground class.

Original: We denote these regions as temporary background, and the remaining part (which has no foreground annotations) as constant background.
Revised: We denote these regions as temporary background, and the remaining part (which has no foreground annotations) as constant background.

Original: Correspondingly, we use x [ tb ] and x [ cb ] to distinguish the tokens from the temporal and the constant background regions, respectively, and use x [ f ] to represent the tokens in the foreground-of-interest in the current episode.
Revised: Correspondingly, we use x [ tb ] and x [ cb ] to distinguish the tokens from the temporal and the constant background regions, respectively, and use x [ f ] to represent the tokens in the foreground-of-interest in the current episode.

Original: The Cross-Region Interaction compares the cosine distance between x [ tb ] and x
Revised: The Cross-Region Interaction compares the cosine distance between x [ tb ] and x

Original: [ cb ] after the selfattention layer (Eq. 2) and makes x [ cb ] absorb information from the dissimilar x [ tb ] ∈ X [ tb ] by:
Revised: [ cb ] after the selfattention layer (Eq. 2) and makes x [ cb ] absorb information from the dissimilar x [ tb ] ∈ X [ tb ] by:

Original: [Equation]
Revised: [Equation]

Original: (3) where x [ cb ] ′ denotes updated constant background tokens embedding.
Revised: where x [ cb ] ′ denotes updated constant background tokens embedding.

Original: Region-level triplet loss.
Revised: Region-level triplet loss.

Original: Besides the above Cross-Region Interaction smoothing the background across different regions through attention, we further use a region-level triplet loss to pull close the constant background tokens and temporal background tokens on the last MuHS transformer block.
Revised: Besides the above Cross-Region Interaction smoothing the background across different regions through attention, we further use a region-level triplet loss to pull close the constant background tokens and temporal background tokens on the last MuHS transformer block.

Original: The region-level triplet loss is enforced on the final output state of the background tokens by:
Revised: The region-level triplet loss is enforced on the final output state of the background tokens by:

Original: [Equation]
Revised: [Equation]

Original: where D ( ., . ) is the cosine distance between two tokens.
Revised: where D ( ., . ) is the cosine distance between two tokens.

Original: 3. M ASKED I MAGE S EGMENTATION
Revised: 3. M ASKED I MAGE S EGMENTATION

Original: The Masked Image Segmentation (MIS) model is appended upon the feature extractor of MuHS.
Revised: The Masked Image Segmentation (MIS) model is appended upon the feature extractor of MuHS.

Original: It makes two types of prediction, i.e. , a partial prediction from a matching head and a holistic prediction from an additional “decoder + matching head”.
Revised: It makes two types of prediction, i.e. , a partial prediction from a matching head and a holistic prediction from an additional “decoder + matching head”.

Original: The details are as below:
Revised: The details are as below:

Original: • Partial prediction : We recall that MuHS randomly discards some patches of the input query image during training.
Revised: • Partial prediction : We recall that MuHS randomly discards some patches of the input query image during training.

Original: Correspondingly, the output tokens X Lq from the L -layer MuHS feature extractor are incomplete.
Revised: Correspondingly, the output tokens X Lq from the L -layer MuHS feature extractor are incomplete.

Original: Given these existing output patches we use a matching head (linear classification head) to make partial prediction in Fig.
Revised: Given these existing output patches, we use a matching head (linear classification head) to make partial prediction in Fig.

Original: 2 (a)
Revised: 2 (a).

Original: Specifically
Revised: Specifically,

Original: , we
Revised: we

Original: calculate the foreground / background mean features with the support tokens X Ls and the ground- truth object mask to correspondingly derive the foreground / background proxies w f and w b .
Revised: calculate the foreground / background mean features with the support tokens X Ls and the ground-truth object mask to correspondingly derive the foreground / background proxies w f and w b .

Original: During training, the supervision on a specified query token x
Revised: During training, the supervision on a specified query token x

Original: Lq is:
Revised: Lq is:

Original: [Equation]
Revised: [Equation]

Original: where y q is corresponding label of the query
Revised: where y q is corresponding label of the query

Original: token x Lq ( y q =1, if y q belongs to the foreground; otherwise y q =0).
Revised: token x Lq ( y q =1, if y q belongs to the foreground; otherwise y q =0).

Original: • Holistic prediction : In addition, we fill mask tokens with trainable positional embeddings to construct the full set of the query patches (as shown in Fig.
Revised: • Holistic prediction : In addition, we fill mask tokens with trainable positional embeddings to construct the full set of the query patches (as shown in Fig.

Original: 2 (a)).
Revised: 2 (a)).

Original: Therefore, the output tokens X Mq from the Masked Image Segmentation model are complete.
Revised: Therefore, the output tokens X Mq from the Masked Image Segmentation model are complete.

Original: We make holistic prediction for all the query patches from an additional matching head, the weight matrix from which is calculated by foreground / background mean features with the support tokens X Ms from the Masked Image Segmentation model and the ground-truth object mask.
Revised: We make holistic prediction for all the query patches from an additional matching head, the weight matrix from which is calculated by foreground / background mean features with the support tokens X Ms from the Masked Image Segmentation model and the ground-truth object mask.

Original: The holistic prediction is supervised under cross-entropy loss L holistic , which is similar as in Eq. 5 and is thus omitted here.
Revised: The holistic prediction is supervised under cross-entropy loss L holistic , which is similar as in Eq. 5 and is thus omitted here.

Original: We note that Masked Image Segmentation model is only utilized for training
Revised: We note that Masked Image Segmentation model is only utilized for training.

Original: During testing, we feed all the query patches into MuHS feature extractor and compare each query token against these proxies to classify each patch into the foreground / background.
Revised: During testing, we feed all the query patches into MuHS feature extractor and compare each query token against these proxies to classify each patch into the foreground / background.

Original: Overall Training.
Revised: Overall Training.

Original: During the training stage, we successively perform the feature extractor and
Revised: During the training stage, we successively perform the feature extractor and

Original: Masked Image Segmentation model for the query prediction.
Revised: Masked Image Segmentation model for the query prediction.

Original: The overall training loss is as follows:
Revised: The overall training loss is as follows:

Original: [Equation]
Revised: [Equation]

Original: where α and β are weighting factors.
Revised: where α and β are weighting factors.

Original:  E XPERIMENTS 4. D ATASETS .
Revised:  E XPERIMENTS 4. D ATASETS .

Original: We evaluate the proposed MuHS on two datasets: PASCAL- 5 i (Shaban et al.
Revised: We evaluate the proposed MuHS on two datasets: PASCAL- 5 i (Shaban et al.

Original: and COCO20 i (Nguyen & Todorovic (2019)).
Revised: and COCO20 i (Nguyen & Todorovic (2019)).

Original: PASCAL- 

Revised: PASCAL-

Original: i consists of PASCAL VOC 2012 (Everingham et al.
Revised: i consists of PASCAL VOC 2012 (Everingham et al.

Original: (2010)) and additionally annotations from SDS (Hariharan et al.
Revised: (2010)) and additionally annotations from SDS (Hariharan et al.

Original: We divide 20 classes into 4 splits and each split has 5 classes.
Revised: We divide 20 classes into 4 splits and each split has 5 classes.

Original: During evaluation on one split (5 classes), we have other three splits (15 classes) for training.
Revised: During evaluation on one split (5 classes), we have other three splits (15 classes) for training.

Original: We randomly sample 1000 pairs of support and query in each split testing.
Revised: We randomly sample 1000 pairs of support and query in each split testing.

Original: COCO- 20 i is built from COCO2014 (Lin et al.
Revised: COCO- 20 i is built from COCO2014 (Lin et al.

Original: We divide 80 classes into 4 splits and each split has 20 classes.
Revised: We divide 80 classes into 4 splits and each split has 20 classes.

Original: During evaluation on one split (20 classes), we have other three splits (60 classes) for training.
Revised: During evaluation on one split (20 classes), we have other three splits (60 classes) for training.

Original: We randomly sample 20000 pairs of support and query in each split testing.
Revised: We randomly sample 20000 pairs of support and query in each split testing.

Original: Following previous works (Tian et al.
Revised: Following previous works (Tian et al.

Original: (2020b)
Revised: (2020b)

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021b))
Revised: (2021b))

Original: , we compare the performance on testing splits by using mean intersection over union (mIoU) as our evaluation metrics. 
Revised: , we compare the performance on testing splits by using mean intersection over union (mIoU) as our evaluation metrics. [Table] [Table]

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: We adopt DeiT-B/16 (Touvron et al.
Revised: We adopt DeiT-B/16 (Touvron et al.

Original: (pretrained on Imagenet (Deng et al.
Revised: (pretrained on Imagenet (Deng et al.

Original: as our backbone.
Revised: as our backbone.

Original: We use SGD optimizer and set the learning rate as 9e-4.
Revised: We use SGD optimizer and set the learning rate as 9e-4.

Original: We randomly crop images to 480 × 480 and follow the data augmentation in PFENet (Tian et al.
Revised: We randomly crop images to 480 × 480 and follow the data augmentation in PFENet (Tian et al.

Original: (2020b)).
Revised: (2020b)).

Original: For PASCAL- 5 i , we train 50 epochs with batch size 4.
Revised: For PASCAL- 5 i , we train 50 epochs with batch size 4.

Original: For COCO- 20 i , we train 30 epochs and set batch size to 16.
Revised: For COCO- 20 i , we train 30 epochs and set batch size to 16.

Original: The proposed MuHS is trained on Pytorch with 4 NVIDIA 
Revised: The proposed MuHS is trained on Pytorch with 4 NVIDIA A100 GPUS.

Original: [Table] [Table]
Revised: More details are in Sec A.2.

Original: 4. C OMPARISON WITH THE STATE OF THE ARTS
Revised: 4. C OMPARISON WITH THE STATE OF THE ARTS

Original: We compare MuHS with the existing state of the arts on PASCAL- 5 i and COCO- 20 i .
Revised: We compare MuHS with the existing state of the arts on PASCAL- 5 i and COCO- 20 i .

Original: The results on two datasets are summarized in Table.
Revised: The results on two datasets are summarized in Table.

Original: 1 and Table.
Revised: 1 and Table.

Original: 2, respectively.
Revised: 2, respectively.

Original: From Table.
Revised: From Table.

Original: 1, we clearly observe the superiority of MuHS on PASCAL- 5 i .
Revised: 1, we clearly observe the superiority of MuHS on PASCAL- 5 i .

Original: First, comparing MuHS against the DeiT-B baseline, we find MuHS achieves considerable improvements.
Revised: First, comparing MuHS against the DeiT-B baseline, we find MuHS achieves considerable improvements.

Original: For example, under the 1-shot and 5-shot settings, MuHS outperforms the DeiT-B baseline by +5.2%
Revised: For example, under the 1-shot and 5-shot settings, MuHS outperforms the DeiT-B baseline by +5.2%

Original: and +1.9% mIoU, respectively.
Revised: and +1.9% mIoU, respectively.

Original: Second, MuHS surpasses all the state-of-the-art methods by a clear margin (especially under the 5-shot setting).
Revised: Second, MuHS surpasses all the state-of-the-art methods by a clear margin (especially under the 5-shot setting).

Original: For example, MuHS clearly outperforms the most competing method BAM (Lang et al.
Revised: For example, MuHS clearly outperforms the most competing method BAM (Lang et al.

Original: by +1.3% and +5.8% mIoU on 1-shot and 5-shot, respectively.
Revised: by +1.3% and +5.8% mIoU on 1-shot and 5-shot, respectively.

Original: We note that all the competing methods use sophisticated matching heads, while our MuHS only uses a simple linear classification head.
Revised: We note that all the competing methods use sophisticated matching heads, while our MuHS only uses a simple linear classification head.

Original: We thus infer that the superiority of our MuHS mainly comes from the discriminative features, indicating that MuHs is a strong feature extractor for FSS.
Revised: We thus infer that the superiority of our MuHS mainly comes from the discriminative features, indicating that MuHs is a strong feature extractor for FSS.

Original: The observation on COCO-20 i (Table. 2) is similar, i.e. , MuHS improves the DeiT-B baseline and presents superiority against all the competing methods.
Revised: The observation on COCO-20 i (Table. 2) is similar, i.e. , MuHS improves the DeiT-B baseline and presents superiority against all the competing methods.

Original: For example, under the 5-shot settings,
Revised: For example, under the 5-shot settings,

Original: MuHS surpasses the DeiT-B baseline by +3.3% mIoU and surpasses BAM (Lang et al.
Revised: MuHS surpasses the DeiT-B baseline by +3.3% mIoU and surpasses BAM (Lang et al.

Original: by +5.5%, mIoU, respectively.
Revised: by +5.5%, mIoU, respectively.

Original: 4. C OMPATIBILITY TO STATE - OF - THE - ART MATCHING HEADS .

Original: Since the proposed MuHS focuses on learning the feature extractor, we are interested at its compatibility to different FSS matching heads.

Original: Specifically, we investigate two popular FSS heads, i.e. ,

Original: Res Res DeiT-B
Revised: Res50 Res101 DeiT-B

Original: MuHS 64.
Revised: MuHS 64.

Original: Res Res DeiT-B
Revised: Res50 Res101 DeiT-B

Original: MuHS
Revised: MuHS

Original: [Figure]
Revised: [Figure]

Revised: PFENet (

Revised: [Figure]

Revised: ) CyCTR

Original: Components
Revised: Components

Original: PASCAL- 

Revised: PASCAL-

Original: i
Revised: i

Original: COCO- 20 i 1-shot 5-shot 1-shot 5-shot
Revised: COCO- 20 i 1-shot 5-shot 1-shot 5-shot

Original: Baseline 63.
Revised: Baseline 63.

Original: Baseline + Cross-sample Attention 65.
Revised: Baseline + Cross-Sample Attention 65.

Original: Baseline + Region-level Interaction 66.
Revised: Baseline + Cross-Region Interaction 66.

Original: Baseline + Masked Image Segmentation 66.
Revised: Baseline + Masked Image Segmentation 66.

Original: All (MuHS) 69.
Revised: All (MuHS) 69.

Revised: 4. C OMPATIBILITY TO STATE - OF - THE - ART MATCHING HEADS .

Revised: Since the proposed MuHS focuses on learning the feature extractor, we are interested at its compatibility to different FSS matching heads.

Revised: Specifically, we investigate two popular FSS heads, i.e. ,

Original: PFENet (Tian et al.
Revised: PFENet (Tian et al.

Original: (2020b))
Revised: (2020b))

Original: and CyCTR (Zhang et al.
Revised: and CyCTR (Zhang et al.

Original: (2021b)).
Revised: (2021b)).

Original: We compare our MuHS feature against two frozen CNN feature ( i.e., ResNet50, ResNet101, following their original practice), as well as the DeiT-B feature.
Revised: We compare our MuHS feature against two frozen CNN feature ( i.e., ResNet50, ResNet101, following their original practice), as well as the DeiT-B feature.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: 3, from which we draw three observations as below:
Revised: 3, from which we draw three observations as below:

Original: First, compared with the frozen CNN features, the frozen DeiT-B features considerably decreases accuracy for both the PFENet and CyCTR heads.
Revised: First, compared with the frozen CNN features, the frozen DeiT-B features considerably decreases accuracy for both the PFENet and CyCTR heads.

Original: For example, the frozen DeiT-B feature is inferior than the frozen ResNet50 feature by ↓ 3 .
Revised: For example, the frozen DeiT-B feature is inferior than the frozen ResNet50 feature by ↓ 3 .

Original: 1% , ↓ 3 .
Revised: 1% , ↓ 3 .

Original: 2% mIoU on PFENet and CyCTR, respectively.
Revised: 2% mIoU on PFENet and CyCTR, respectively.

Original: It is somehow surprising given that the transformer feature usually achieves better discriminative ability than the CNN counterparts.
Revised: It is somehow surprising given that the transformer feature usually achieves better discriminative ability than the CNN counterparts.

Original: However, we think the above observation is actually reasonable, because both the PFENet and CyCTR heads (as well as most matching heads in prior literature) are specifically designed for CNN features and lack consideration for the transformer features.
Revised: However, we think the above observation is actually reasonable, because both the PFENet and CyCTR heads (as well as most matching heads in prior literature) are specifically designed for CNN features and lack consideration for the transformer features.

Original: Second, our MuHS features significantly outperform the CNN features, although the employed heads is specifically designed for the CNN features.
Revised: Second, our MuHS features significantly outperform the CNN features, although the employed heads is specifically designed for the CNN features.

Original: For example, using the PFENet head, MuHS feature surpasses the ResNet101 feature by ↑ 7 .
Revised: For example, using the PFENet head, MuHS feature surpasses the ResNet101 feature by ↑ 7 .

Original: We infer that although there are some weaknesses for cooperating the transformer feature with these heads (as in observation 1), the benefits of suppressing the heterogeneity in MuHS dominate.
Revised: We infer that although there are some weaknesses for cooperating the transformer feature with these heads (as in observation 1), the benefits of suppressing the heterogeneity in MuHS dominate.

Original: Therefore, MuHS brings improvement over the
Revised: Therefore, MuHS brings improvement over the

Original: CNN features and shows good compatibility against these matching heads.
Revised: CNN features and shows good compatibility against these matching heads.

Original: Third, comparing the achieved mIoU of “MuHS + PFENet (CyCTR) head” against the “MuHS + linear classification head” (in Table. 1), we find that the latter is slightly better.
Revised: Third, comparing the achieved mIoU of “MuHS + PFENet (CyCTR) head” against the “MuHS + linear classification head” (in Table. 1), we find that the latter is slightly better.

Original: This result is consistent with the above two observations: MuHS is the superior feature for the PFENet and CyCTR heads (observation 2), while those two heads are not the superior heads for MuHS and other transformer features (observation 1).
Revised: This result is consistent with the above two observations: MuHS is the superior feature for the PFENet and CyCTR heads (observation 2), while those two heads are not the superior heads for MuHS and other transformer features (observation 1).

Original: Therefore, we recommend using the simple linear classification head for
Revised: Therefore, we recommend using the simple linear classification head for

Original: MuHS currently.
Revised: MuHS currently.

Original: That being said, we are optimistic towards a good embrace between MuHS and future matching heads in the FSS community.
Revised: That being said, we are optimistic towards a good embrace between MuHS and future matching heads in the FSS community.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: MuHS adopts Cross-Sample Attention, Region-level Interaction and Masked Image Segmentation to reinforce the interaction between different samples, regions and neighboring patches and suppress
Revised: MuHS adopts Cross-Sample Attention, Cross-Region Interaction and Masked Image Segmentation to reinforce the interaction between different samples, regions and neighboring patches and suppress

Original: [Figure]
Revised: [Figure]

Original: the heterogeneity in FSS.
Revised: the heterogeneity in FSS.

Original: We investigate their benefits through ablation, as shown in Table 3.
Revised: We investigate their benefits through ablation, as shown in Table 3.

Original: We draw two observations:
Revised: We draw two observations:

Original: First, all the three modules can bring considerable improvements, independently.
Revised: First, all the three modules can bring considerable improvements, independently.

Original: For example, under 1-shot on PASCAL- 5 i , adding Cross-sample Attention, Region-level Interaction and Masked
Revised: For example, under 1-shot on PASCAL- 5 i , adding Cross-Sample Attention, Cross-Region Interaction and Masked

Original: Image Segmentation alone improves the baseline by + 1 . 9% , + 2 . 4%
Revised: Image Segmentation alone improves the baseline by + 1 . 9% , + 2 . 4%

Original: and + 3 . 0% , respectively.
Revised: and + 3 . 0% , respectively.

Original: Second, MuHS integrating all the three components achieves further improvement, e.g., + 5 .
Revised: Second, MuHS integrating all the three components achieves further improvement, e.g., + 5 .

Original: 2% on
Revised: 2% on

Original: PASCAL- 5 i under 1-shot setting.
Revised: PASCAL- 5 i under 1-shot setting.

Original: It indicates that these three components suppressing different heterogeneity achieve complementary benefits.
Revised: It indicates that these three components suppressing different heterogeneity achieve complementary benefits.

Original: We supplement more ablation studies in Sec A.5.
Revised: We supplement more ablation studies in Sec A.5.

Original: Visualization of heterogeneity suppression.
Revised: Visualization of heterogeneity suppression.

Original: We visualize some segmentation results in Fig.
Revised: We visualize some segmentation results in Fig.

Original: 4 to intuitively understand how the proposed MuHS suppresses the heterogeneity and improves FSS.
Revised: 4 to intuitively understand how the proposed MuHS suppresses the heterogeneity and improves FSS.

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 4 (a), due to the sample-level heterogeneity (between cattle and cow), the baseline fails to recall many foreground pixels.
Revised: 4 (a), due to the sample-level heterogeneity (between cattle and cow), the baseline fails to recall many foreground pixels.

Original: In contrast, the proposed MuHS significantly improves the recall on the foreground pixels due to its capacity of suppressing the sample-level heterogeneity.
Revised: In contrast, the proposed MuHS significantly improves the recall on the foreground pixels due to its capacity of suppressing the sample-level heterogeneity.

Original: In Fig.
Revised: In Fig.

Original: 4 (b), the region-level heterogeneity makes the baseline to classify many pixels on the horse into the foreground (person).
Revised: 4 (b), the region-level heterogeneity makes the baseline to classify many pixels on the horse into the foreground (person).

Original: In contrast, MuHS smooths the background and thus remove the distraction from the horse.
Revised: In contrast, MuHS smooths the background and thus remove the distraction from the horse.

Original: In Fig.
Revised: In Fig.

Original: 4 (c), MuHS successfully merges the upper part of body into the foreground “person” by suppressing patch-level heterogeneity.
Revised: 4 (c), MuHS successfully merges the upper part of body into the foreground “person” by suppressing patch-level heterogeneity.

Original: Additionally, we visualize the feature distribution before and after eliminating heterogeneity in Sec A.4 and plot the convergence curves in Sec A.6.
Revised: Additionally, we visualize the feature distribution before and after eliminating heterogeneity in Sec A.4 and plot the convergence curves in Sec A.6.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes a feature extractor with Multi-level Heterogeneity Suppressing (MuHS) for fewshot semantic segmentation(FSS).
Revised: This paper proposes a feature extractor with Multi-level Heterogeneity Suppressing (MuHS) for fewshot semantic segmentation(FSS).

Original: Based on the transformer backbone, MuHS sets up novel crosssample attention, cross-region interaction and the masked image segmentation.
Revised: Based on the transformer backbone, MuHS sets up novel crosssample attention, cross-region interaction and the masked image segmentation.

Original: The cross-sample attention efficiently propagates information across different samples.
Revised: The cross-sample attention efficiently propagates information across different samples.

Original: The cross-region interaction facilitates feature absorption between dissimilar regions within the background.
Revised: The cross-region interaction facilitates feature absorption between dissimilar regions within the background.

Original: The masked image segmentation utilizes the contextual information to infer the labels for discarded (masked) patch tokens so as to reinforce the capacity of contextual inference.
Revised: The masked image segmentation utilizes the contextual information to infer the labels for discarded (masked) patch tokens so as to reinforce the capacity of contextual inference.

Original: These three modules respectively suppress the heterogeneity from three different levels, therefore improving the intra-class compactness of the FSS features.
Revised: These three modules respectively suppress the heterogeneity from three different levels, therefore improving the intra-class compactness of the FSS features.

Original: Extensive experiments on two popular FSS datasets demonstrate the effectiveness of MuHS and the achieved results are on par with the state of the art.
Revised: Extensive experiments on two popular FSS datasets demonstrate the effectiveness of MuHS and the achieved results are on par with the state of the art.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper can help to improve the semantic segmentation accuracy with the limited labeled samples.
Revised: This paper can help to improve the semantic segmentation accuracy with the limited labeled samples.

Original: It may be applied to automatic driving system to improve safety when the system needs to recognize unseen objects.
Revised: It may be applied to automatic driving system to improve safety when the system needs to recognize unseen objects.

Original: We will explore more applications of few-shot segmentation.
Revised: We will explore more applications of few-shot segmentation.

Original: Moreover, we will try to improve the reliability of few-shot segmentation systems to reduce potential problems.
Revised: Moreover, we will try to improve the reliability of few-shot segmentation systems to reduce potential problems.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The MuHS is reproducible.
Revised: The MuHS is reproducible.

Original: In the main text, we describe the two datasets for evaluation, ( i e.,
Revised: In the main text, we describe the two datasets for evaluation, ( i e.,

Original: PASCAL- 5 i and COCO- 20 i ) and the details about the experimental implementation.
Revised: PASCAL- 5 i and COCO- 20 i ) and the details about the experimental implementation.

Original: We supply the analysis of some hyper parameters in appendix.
Revised: We supply the analysis of some hyper parameters in appendix.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In the appendix, we supply the details which are not described in the main text due to space limitation.
Revised: In the appendix, we supply the details which are not described in the main text due to space limitation.

Original: In Section A.1, we analyze the impact of some hyper-parameters.
Revised: In Section A.1, we analyze the impact of some hyper-parameters.

Original: In Section A.2, we add more implementation details.
Revised: In Section A.2, we add more implementation details.

Original: In Section A.3, we adopt one more dataset to evaluate performance.
Revised: In Section A.3, we adopt one more dataset to evaluate performance.

Original: In Section A.4, we compare the feature distribution before and after eliminating heterogeneity.
Revised: In Section A.4, we compare the feature distribution before and after eliminating heterogeneity.

Original: In
Revised: In

Original: Section A.5, we supply more ablation experiments.
Revised: Section A.5, we supply more ablation experiments.

Original: In Section A.6, we plot the convergence curves of the proposed MuHS and recent state-of-the-art methods.
Revised: In Section A.6, we plot the convergence curves of the proposed MuHS and recent state-of-the-art methods.

Original: A. H YPER - PARAMETERS ANALYSIS .
Revised: A. H YPER - PARAMETERS ANALYSIS .

Original: We analyze the impact of important hyper-parameters, i.e., α and β in Eq. 6.
Revised: We analyze the impact of important hyper-parameters, i.e., α and β in Eq. 6.

Original: and investigate the impact of model depth and masked ratio of the proposed Masked Image Segmentation (MIS) in
Revised: and investigate the impact of model depth and masked ratio of the proposed Masked Image Segmentation (MIS) in

Original: Section 3.4.
Revised: Section 3.4.

Original: The experiments are reported on split-0 of PASCAL- 

Revised: The experiments are reported on split-0 of PASCAL-

Original: i , under 1-shot setting.
Revised: i , under 1-shot setting.

Original: 5-shot 1-shot

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 5 (a), we evaluate the impact of α , which controls the weight of triplet loss in Eq.
Revised: 5 (a), we evaluate the impact of α , which controls the weight of triplet loss in Eq.

Original: We observe that the accuracy first increases (when α increases from 0 to 0.6) and then decreases (when α further increases to 1.0).
Revised: We observe that the accuracy first increases (when α increases from 0 to 0.6) and then decreases (when α further increases to 1.0).

Original: We set α = 0.6 as the weight factor.
Revised: We set α = 0.6 as the weight factor.

Original: In Fig.
Revised: In Fig.

Original: 5 (b), we evaluate the impact of β , which controls the weight of holistic prediction loss in Eq.
Revised: 5 (b), we evaluate the impact of β , which controls the weight of holistic prediction loss in Eq.

Original: We observe that the accuracy first increases (when β increases from 0 to 0.5) and then decreases (when β further increases to 1.0).
Revised: We observe that the accuracy first increases (when β increases from 0 to 0.5) and then decreases (when β further increases to 1.0).

Original: We set β = 0.5 as the weight factor.
Revised: We set β = 0.5 as the weight factor.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In Table 4, we analyse the impact of Masked Image Segmentation (MIS) model depth.
Revised: In Table 4, we analyse the impact of Masked Image Segmentation (MIS) model depth.

Original: We observe that the 7-layer MIS model can achieve the best accuracy.
Revised: We observe that the 7-layer MIS model can achieve the best accuracy.

Original: In Table 5, we analyse the impact of masked ratio for MIS model.
Revised: In Table 5, we analyse the impact of masked ratio for MIS model.

Original: We observe that randomly masking out 7% patches can achieve the best accuracy.
Revised: We observe that randomly masking out 7% patches can achieve the best accuracy.

Original: A. I MPLEMENTATION DETAILS .
Revised: A. I MPLEMENTATION DETAILS .

Original: We supply the implementation details of how to transform a patch token into pixel-wise mask map and the scheme to generate the discarded patches.
Revised: We supply the implementation details of how to transform a patch token into pixel-wise mask map and the scheme to generate the discarded patches.

Original: • To transform a patch into pixel-wise mask map, we follow the common practice (PANet (Wang et al.
Revised: • To transform a patch into pixel-wise mask map, we follow the common practice (PANet (Wang et al.

Original: , PFENet (Tian et al.
Revised: , PFENet (Tian et al.

Original: (2020b)))
Revised: (2020b)))

Original: of spatially up-sampling.
Revised: of spatially up-sampling.

Original: Specially, we first obtain the softmax scores for each patch token through the classification head.
Revised: Specially, we first obtain the softmax scores for each patch token through the classification head.

Original: Then, the score maps are upsampled through bi-linear interpolation to match the size of the input image.
Revised: Then, the score maps are upsampled through bi-linear interpolation to match the size of the input image.

Original: Finally, we use argmax operation to generate pixel-wise mask map.
Revised: Finally, we use argmax operation to generate pixel-wise mask map.

Original: • To generate the discarded patches, we randomly shuffle the patch tokens and then mask the rear of the token sequence.
Revised: • To generate the discarded patches, we randomly shuffle the patch tokens and then mask the rear of the token sequence.

Original: This operation is the same as in other MIM methods ( e.g. , MAE (He et al.
Revised: This operation is the same as in other MIM methods ( e.g. , MAE (He et al.

Original: A. E VALUATION ON MORE DATASETS .
Revised: A. E VALUATION ON MORE DATASETS .

Original: We evaluate one more dataset, i.e., Cityscapes (Cordts et al.
Revised: We evaluate one more dataset, i.e., Cityscapes (Cordts et al.

Original: (2016)), an urban street-scene dataset.
Revised: (2016)), an urban street-scene dataset.

Original: We use 15 classes (out of the commonly-used 19 classes) to construct the base set and use the other 4 classes (i.e., “sky”, “person”, “car”, “bicycle”) for the novel set.
Revised: We use 15 classes (out of the commonly-used 19 classes) to construct the base set and use the other 4 classes (i.e., “sky”, “person”, “car”, “bicycle”) for the novel set.

Original: [Table]
Revised: [Table]

Original: Based on this newly-generated few-shot segmentation benchmark, we compare the proposed MuHS against the baseline and a most recent state-of-the-art method CyCTR(Zhang et al.
Revised: Based on this newly-generated few-shot segmentation benchmark, we compare the proposed MuHS against the baseline and a most recent state-of-the-art method CyCTR(Zhang et al.

Original: (2021b)).
Revised: (2021b)).

Original: From the Table 6, we observe MuHS significantly surpasses the baseline and CyCTR by +12.1% and +10% mAP, respectively.
Revised: From the Table 6, we observe MuHS significantly surpasses the baseline and CyCTR by +12.1% and +10% mAP, respectively.

Original: We note that the cityscapes is challenging for few-shot segmentation task.
Revised: We note that the cityscapes is challenging for few-shot segmentation task.

Original: We infer it is because in cityscapes, the number of semantic classes appearing in a single image is much larger, therefore increasing the challenge from region-level heterogeneity.
Revised: We infer it is because in cityscapes, the number of semantic classes appearing in a single image is much larger, therefore increasing the challenge from region-level heterogeneity

Original:  A. C OMPARISON OF THE FEATURE DISTRIBUTION .
Revised: . A. C OMPARISON OF THE FEATURE DISTRIBUTION .

Original: To better understand how MuHS suppresses the three types of heterogeneity, we use t-SNE visualization to compare the feature distribution before and after MuHS in Fig.
Revised: To better understand how MuHS suppresses the three types of heterogeneity, we use t-SNE visualization to compare the feature distribution before and after MuHS in Fig.

Original: [Figure]
Revised: [Figure]

Original: We correspondingly draw two observations.
Revised: We correspondingly draw two observations.

Original: First, the intra-class distributions of both background and the foreground become more compact, indicating that MuHS effectively suppresses the regionlevel and patch-level heterogeneity.
Revised: First, the intra-class distributions of both background and the foreground become more compact, indicating that MuHS effectively suppresses the regionlevel and patch-level heterogeneity.

Original: Second, the foreground from the support and query images become closer, indicating that MuHS effectively suppresses the sample-level heterogeneity.
Revised: Second, the foreground from the support and query images become closer, indicating that MuHS effectively suppresses the sample-level heterogeneity.

Original: These observations are consistent with the segmentation examples in Fig.
Revised: These observations are consistent with the segmentation examples in Fig.

Original: This experiment also intuitively validates that MuHS improves intra-class feature compactness.
Revised: This experiment also intuitively validates that MuHS improves intra-class feature compactness.

Original: A. A BLATIONS ON MORE VARIANTS .
Revised: A. A BLATIONS ON MORE VARIANTS .

Original: We recall that all the three components ( i.e. , cross-sample attention, region-level interaction and masked image segmentation) can bring considerable improvements, independently and integrating all of them achieves further improvements.
Revised: We recall that all the three components ( i.e. , cross-sample attention, cross-region interaction and masked image segmentation) can bring considerable improvements, independently and integrating all of them achieves further improvements.

Original: For better investigating the efficiency, we supply more ablation studies by adding three more variants, as shown in Table 7.
Revised: For better investigating the efficiency, we supply more ablation studies by adding three more variants, as shown in Table 7.

Original: Each variant combines two components out of cross-sample attention, region-level interaction and masked image segmentation can still improve the baseline.
Revised: Each variant combines two components out of cross-sample attention, cross-region interaction and masked image segmentation can still improve the baseline.

Original: [Figure]

Original: [Table]
Revised: [Table]

Revised: A.

Revised: C ONVERGENCE CURVES .

Original: In Fig.
Revised: In Fig.

Original: 7, we plot the convergence curves of CyCTR, PFENet and MuHS.
Revised: 7, we plot the convergence curves of CyCTR, PFENet and MuHS.

Original: Compared with CyCTR and PFENet, MuHS costs 4 × less training epochs (50 epochs) with fewer time (8 hours) and achieves higher accuracy on PASCAL-5 i .
Revised: Compared with CyCTR and PFENet, MuHS costs 4 × less training epochs (50 epochs) with fewer time (8 hours) and achieves higher accuracy on PASCAL-5 i .

Original: The comparison is based on Pytorch and NVIDIA
Revised: The comparison is based on Pytorch and NVIDIA

Original: A
Revised: A

Original: GPU.
Revised: GPU.

Original: [Figure]
Revised: [Figure]