Original: O PEN -V OCABULARY P ANOPTIC /U NIVERSAL S EGMENTATION WITH M ASK CLIP
Revised: O PEN -V OCABULARY P ANOPTIC /U NIVERSAL S EGMENTATION WITH M ASK CLIP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we tackle an emerging computer vision task
Revised: In this paper, we tackle an emerging computer vision task

Original: , open-vocabulary panop
Revised: , open-vocabulary panop

Original: - tic segmentation, that aims to perform panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time.
Revised: - tic segmentation, that aims to perform panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time.

Original: We first build a baseline method by directly adopting pre-trained CLIP models without finetuning nor distillation.
Revised: We first build a baseline method by directly adopting pre-trained CLIP models without finetuning nor distillation.

Original: We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual Encoder, which is an encoder-only module that seamless integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction.
Revised: We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual Encoder, which is an encoder-only module that seamless integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction.

Original: MaskCLIP learns to efficiently and effectively utilize pre-trained dense/local CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process.
Revised: MaskCLIP learns to efficiently and effectively utilize pre-trained dense/local CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process.

Original: We obtain encouraging results for open-vocabulary panoptic/instance segmentation and state-of-the-art results for semantic segmentation on ADE20K and PASCAL datasets.
Revised: We obtain encouraging results for open-vocabulary panoptic/instance segmentation and state-of-the-art results for semantic segmentation on ADE20K and PASCAL datasets.

Original: We show qualitative illustration for MaskCLIP with online custom categories
Revised: We show qualitative illustration for MaskCLIP with online custom categories

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Panoptic segmentation (Kirillov et al., 2019b) or image parsing (Tu et al., 2005) integrates the task of semantic segmentation (Tu, 2008) for background regions (e.g. “stuff” like “road”, “sky”) and instance segmentation (He et al., 2017) for foreground objects (e.g. “things” such as “person”, “table”).
Revised: Panoptic segmentation (Kirillov et al., 2019b) or image parsing (Tu et al., 2005) integrates the task of semantic segmentation (Tu, 2008) for background regions (e.g. “stuff” like “road”, “sky”) and instance segmentation (He et al., 2017) for foreground objects (e.g. “things” such as “person”, “table”).

Original: Existing panoptic segmentation methods (Kirillov et al., 2019b;a; Li et al., 2019; Xiong et al., 2019; Lazarow et al., 2020) and instance segmentation approaches (He et al., 2017) deal with a fixed set of category definitions, which are essentially represented by categorical labels without semantic relations.
Revised: Existing panoptic segmentation methods (Kirillov et al., 2019b;a; Li et al., 2019; Xiong et al., 2019; Lazarow et al., 2020) and instance segmentation approaches (He et al., 2017) deal with a fixed set of category definitions, which are essentially represented by categorical labels without semantic relations.

Original: DEtection TRansformer (DETR) (Carion et al., 2020) is a pioneering work that builds a Transformer-based architecture for both object detection and panoptic segmentation.
Revised: DEtection TRansformer (DETR) (Carion et al., 2020) is a pioneering work that builds a Transformer-based architecture for both object detection and panoptic segmentation.

Original: The deep learning field is moving rapidly towards the open-world/zero-shot settings (Bendale & Boult, 2015) where computer vision tasks such as classification (Radford et al., 2021), object detection (Li et al., 2022b; Zareian et al., 2021; Zang et al., 2022;
Revised: The deep learning field is moving rapidly towards the open-world/zero-shot settings (Bendale & Boult, 2015) where computer vision tasks such as classification (Radford et al., 2021), object detection (Li et al., 2022b; Zareian et al., 2021; Zang et al., 2022;

Original: Gu et al., 2022;
Revised: Gu et al., 2022;

Original: Cai et al., 2022), semantic labeling (Li et al., 2022a; Ghiasi et al., 2022), and image retrieval (Bendale & Boult, 2015; Hinami &
Revised: Cai et al., 2022), semantic labeling (Li et al., 2022a; Ghiasi et al., 2022), and image retrieval (Bendale & Boult, 2015; Hinami &

Original: Satoh, 2018; Zareian et al., 2021; Hinami & Satoh, 2018; Kamath et al., 2021) perform recognition and detection for categories beyond those in the training set.
Revised: Satoh, 2018; Zareian et al., 2021; Hinami & Satoh, 2018; Kamath et al., 2021) perform recognition and detection for categories beyond those in the training set.

Original: In this paper, we take the advantage of the existence of pre-trained CLIP image and text embedding models (Radford et al., 2021), that are mapped to the same space.
Revised: In this paper, we take the advantage of the existence of pre-trained CLIP image and text embedding models (Radford et al., 2021), that are mapped to the same space.

Original: We first build a baseline method for open-vocabulary panoptic segmentation using CLIP models without training.
Revised: We first build a baseline method for open-vocabulary panoptic segmentation using CLIP models without training.

Original: We then develop a new algorithm, MaskCLIP, that is a Transformer-based approach efficiently and effectively utilizing pre-trained dense/local CLIP features without heavy re-training.
Revised: We then develop a new algorithm, MaskCLIP, that is a Transformer-based approach efficiently and effectively utilizing pre-trained dense/local CLIP features without heavy re-training.

Original: The key component of MaskCLIP is a Relative Mask Attention (RMA) module that seamlessly integrates the mask tokens with a pretrained ViT-based CLIP backbone.
Revised: The key component of MaskCLIP is a Relative Mask Attention (RMA) module that seamlessly integrates the mask tokens with a pretrained ViT-based CLIP backbone.

Original: MaskCLIP is distinct and advantageous compared with existing approaches in three aspects: 1) A canonical background and instance segmentation representation by the mask token representation with a unique encoder-only strategy that tightly couples a pre-trained
Revised: MaskCLIP is distinct and advantageous compared with existing approaches in three aspects: 1) A canonical background and instance segmentation representation by the mask token representation with a unique encoder-only strategy that tightly couples a pre-trained

Original: CLIP image feature encoder with the mask token encoder.
Revised: CLIP image feature encoder with the mask token encoder.

Original: 2) MaskCLIP avoids the challenging student-teacher distillation processes such as OVR-CNN (Zareian et al., 2021) and ViLD
Revised: 2) MaskCLIP avoids the challenging student-teacher distillation processes such as OVR-CNN (Zareian et al., 2021) and ViLD

Original: (Gu et al., 2022) that face limited number of teacher objects to train; 3) MaskCLIP also learns to refine masks beyond simple pooling in e.g. OpenSeg (Ghiasi et al., 2022).
Revised: (Gu et al., 2022) that face limited number of teacher objects to train; 3) MaskCLIP also learns to refine masks beyond simple pooling in e.g. OpenSeg (Ghiasi et al., 2022).

Original: The contributions of our work are listed as follows.
Revised: The contributions of our work are listed as follows.

Original: • We develop a new algorithm, MaskCLIP, to perform open-vocabulary panoptic segmentation building on top of canonical background and instance mask representation with a cascade mask proposal and refinement process.
Revised: • We develop a new algorithm, MaskCLIP, to perform open-vocabulary panoptic segmentation building on top of canonical background and instance mask representation with a cascade mask proposal and refinement process.

Original: • We device the MaskCLIP Visual Encoder under an encoder-only strategy by tightly coupling a pre-trained CLIP image feature encoder with the mask token encoder, to allow for the direct formulation of the mask feature representation for semantic/instance segmentation+refinement, and class prediction.
Revised: • We device the MaskCLIP Visual Encoder under an encoder-only strategy by tightly coupling a pre-trained CLIP image feature encoder with the mask token encoder, to allow for the direct formulation of the mask feature representation for semantic/instance segmentation+refinement, and class prediction.

Original: Within the MaskCLIP Visual Encoder, there is a new module called Relative Mask Attention (RMA)module that performs mask refinement.
Revised: Within the MaskCLIP Visual Encoder, there is a new module called Relative Mask Attention (RMA) that performs mask refinement.

Original: • MaskCLIP expands the scope of the existing CLIP models to open-vocabulary panoptic segmentation by demonstrating encouraging and competitive results for open-vocabulary- panoptic, instance, and semantic segmentation.
Revised: • MaskCLIP expands the scope of the existing CLIP models to open-vocabulary panoptic segmentation by demonstrating encouraging and competitive results for open-vocabulary- panoptic, instance, and semantic segmentation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: [Table]
Revised: [Table]

Original: Open vocabulary.
Revised: Open vocabulary.

Original: The open vocabulary setting is gaining increasing popularity lately as traditional fully supervised setting cannot handle unseen classes during testing, while real world vision applications like scene understanding, self driving and robotics are commonly required to predict unseen classes.
Revised: The open vocabulary setting is gaining increasing popularity lately as traditional fully supervised setting cannot handle unseen classes during testing, while real world vision applications like scene understanding, self driving and robotics are commonly required to predict unseen classes.

Original: Previous open-vocabulary attempts have been primarily made for object detection.
Revised: Previous open-vocabulary attempts have been primarily made for object detection.

Original: ViLD (Gu et al., 2022) trains a student model to distill the knowledge of CLIP.
Revised: ViLD (Gu et al., 2022) trains a student model to distill the knowledge of CLIP.

Original: RegionCLIP (Zhong et al., 2022) finetunes the pretrained CLIP model to match the image areas with corresponding texts.
Revised: RegionCLIP (Zhong et al., 2022) finetunes the pretrained CLIP model to match the image areas with corresponding texts.

Original: OV-DETR (Zang et al., 2022) uses CLIP as an external model to obtain the query embedding from CLIP model.
Revised: OV-DETR (Zang et al., 2022) uses CLIP as an external model to obtain the query embedding from CLIP model.

Original: Recently there is also work made for open-vocabulary semantic segmentation (Ghiasi et al., 2022).
Revised: Recently there is also work made for open-vocabulary semantic segmentation (Ghiasi et al., 2022).

Original: Panoptic segmentation.
Revised: Panoptic segmentation.

Original: Existing panoptic segmentation (or image parsing (Tu et al., 2005)) methods (Kirillov et al., 2019b;a; Li et al., 2019; Xiong et al., 2019; Lazarow et al., 2020) perform training and testing based on a fixed set of category labels.
Revised: Existing panoptic segmentation (or image parsing (Tu et al., 2005)) methods (Kirillov et al., 2019b;a; Li et al., 2019; Xiong et al., 2019; Lazarow et al., 2020) perform training and testing based on a fixed set of category labels.

Original: Open-set panoptic segmentation (Hwang et al., 2021) is an exemplar based approach that requires categories to be known in advance, which is narrower than the open-vocabulary setting where categories of interest can be freely specified in inference.
Revised: Open-set panoptic segmentation (Hwang et al., 2021) is an exemplar based approach that requires categories to be known in advance, which is narrower than the open-vocabulary setting where categories of interest can be freely specified in inference.

Original: Open-vocabulary panoptic segmentation: an emerging task.
Revised: Open-vocabulary panoptic segmentation: an emerging task.

Original: As open-set, open-world, zero-shot, and open-vocabulary are relatively new concepts that have no commonly accepted definitions, thus, different algorithms are often not directly comparable with differences in problem definition/setting, training data, and testing scope.
Revised: As open-set, open-world, zero-shot, and open-vocabulary are relatively new concepts that have no commonly accepted definitions, thus, different algorithms are often not directly comparable with differences in problem definition/setting, training data, and testing scope.

Original: Table 1 gives a summary for the recent open-vocabulary applications.
Revised: Table 1 gives a summary for the recent open-vocabulary applications.

Original: XPM (Huynh et al., 2022) utilizes vision-language cross modal data to generate pseudo-mask supervision to train a student model for instance segmentation, and thus, it may not be fully openvocabulary to allow for arbitrary object specifications in the inference time.
Revised: XPM (Huynh et al., 2022) utilizes vision-language cross modal data to generate pseudo-mask supervision to train a student model for instance segmentation, and thus, it may not be fully openvocabulary to allow for arbitrary object specifications in the inference time.

Original: LSeg (Li et al., 2022a) also has limited open-vocabulary aspect as the learned CNN image features in LSeg are not exposed to representations beyond the training labeling categories.
Revised: LSeg (Li et al., 2022a) also has limited open-vocabulary aspect as the learned CNN image features in LSeg are not exposed to representations beyond the training labeling categories.

Original: OpenSeg (Ghiasi et al., 2022) is potentially applicable for instance/panoptic segmentation, but OpenSeg is formulated to be trained on captions which lack instance-level information that is fundamental for panoptic segmentation.
Revised: OpenSeg (Ghiasi et al., 2022) is potentially applicable for instance/panoptic segmentation, but OpenSeg is formulated to be trained on captions which lack instance-level information that is fundamental for panoptic segmentation.

Original: The direct image feature pooling strategy in OpenSeg is potentially another limiting factor towards the open-vocabulary panoptic segmentation.
Revised: The direct image feature pooling strategy in OpenSeg is potentially another limiting factor towards the open-vocabulary panoptic segmentation.

Original: Nevertheless, no results for open-vocabulary panoptic/instance segmentation are reported in (Ghiasi et al., 2022).
Revised: Nevertheless, no results for open-vocabulary panoptic/instance segmentation are reported in (Ghiasi et al., 2022).

Original: CLIP model distillation/reuse.
Revised: CLIP model distillation/reuse.

Original: After its initial release, the CLIP model (Radford et al., 2021) that is learned from large-scale image-text paired captioning datasets has received a tremendous amount of attention.
Revised: After its initial release, the CLIP model (Radford et al., 2021) that is learned from large-scale image-text paired captioning datasets has received a tremendous amount of attention.

Original: Some other similar vision-language models have also been proposed later e.g. ALIGN (Jia et al., 2021), GLIP (Li et al., 2022b).
Revised: Some other similar vision-language models have also been proposed later e.g. ALIGN (Jia et al., 2021), GLIP (Li et al., 2022b).

Original: Many algorithms have been developed lately (Zang et al., 2022; Wang et al., 2022; Zhong et al., 2022; Luo et al., 2021;
Revised: Many algorithms have been developed lately (Zang et al., 2022; Wang et al., 2022; Zhong et al., 2022; Luo et al., 2021;

Original: Patashnik et al., 2021;
Revised: Patashnik et al., 2021;

Original: Shen et al., 2022) trying knowledge distillation from the CLIP model to benefit the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images.
Revised: Shen et al., 2022) trying knowledge distillation from the CLIP model to benefit the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images.

Original: Here, we directly adopt the backbone of CLIP image model to train for open-vocabulary panoptic segmentation.
Revised: Here, we directly adopt the backbone of CLIP image model to train for open-vocabulary panoptic segmentation.

Original: There have been attempts (Rao et al., 2022; Zhou et al., 2022) that use the dense CLIP features to represent pixel-wise feature as teacher model to train student model for semantic segmentation.
Revised: There have been attempts (Rao et al., 2022; Zhou et al., 2022) that use the dense CLIP features to represent pixel-wise feature as teacher model to train student model for semantic segmentation.

Original:  M ETHOD
Revised:  M ETHOD

Original: [Figure]
Revised: [Figure]

Original: Our pipeline, shown in Figure 1, contains two stages.
Revised: Our pipeline, shown in Figure 1, contains two stages.

Original: The first stage is a class-agnostic mask proposal network.
Revised: The first stage is a class-agnostic mask proposal network.

Original: The second stage is MaskCLIP Visual Encoder which is built on the CLIP (Radford et al., 2021) ViT architecture.
Revised: The second stage is MaskCLIP Visual Encoder which is built on the CLIP (Radford et al., 2021) ViT architecture.

Original: It takes the images and the coarse masks from the first stage as the input and will output refined masks along with the corresponding dense image features for further classification using the text embeddings from the CLIP text encoder.
Revised: It takes the images and the coarse masks from the first stage as the input and will output refined masks along with the corresponding dense image features for further classification using the text embeddings from the CLIP text encoder.

Original: 3. C LASS -A GNOSTIC M ASK P ROPOSAL N ETWORK
Revised: 3. C LASS -A GNOSTIC M ASK P ROPOSAL N ETWORK

Original: Our Class-Agnostic Mask Proposal Network is built on instance/segmentation models such as
Revised: Our Class-Agnostic Mask Proposal Network is built on instance/segmentation models such as

Original: MaskRCNN(He et al., 2017) and Mask2Former(Cheng et al., 2022).
Revised: MaskRCNN(He et al., 2017) and Mask2Former(Cheng et al., 2022).

Original: To make the model classagnostic, we remove the class supervision during training.
Revised: To make the model classagnostic, we remove the class supervision during training.

Original: The classification head thus becomes a binary classification for either positive or negative in these models.
Revised: The classification head thus becomes a binary classification for either positive or negative in these models.

Original: 3. M ASK CLIP V ISUAL E NCODER
Revised: 3. M ASK CLIP V ISUAL E NCODER

Original: Similar to CLIP, our MaskCLIP Visual Encoder also predicts the image features.
Revised: Similar to CLIP, our MaskCLIP Visual Encoder also predicts the image features.

Original: Unlike the CLIP
Revised: Unlike the CLIP

Original: Visual Encoder that only uses on class token to output the feature of the whole image.
Revised: Visual Encoder that only uses on class token to output the feature of the whole image.

Original: Our MaskCLIP
Revised: Our MaskCLIP

Original: Visual Encoder uses another M Mask Class Tokens to output the partial/dense features for each corresponding area of the image given the masks.
Revised: Visual Encoder uses another M Mask Class Tokens to output the partial/dense features for each corresponding area of the image given the masks.

Original: The Mask Class Tokens use attention masks and
Revised: The Mask Class Tokens use attention masks and

Original: Relative Mask Attention to obtain the partial/dense features which we discuss in the following two parts.
Revised: Relative Mask Attention to obtain the partial/dense features which we discuss in the following two parts.

Original: 3.2. M ASK C LASS T OKENS
Revised: 3.2. M ASK C LASS T OKENS

Original: In order to obtain dense image features for the corresponding masks or bounding boxes for further recognition or distillation, an easy way to do this is simply masking or cropping the image and then sending the obtained image to the pretrained image encoder.
Revised: In order to obtain dense image features for the corresponding masks or bounding boxes for further recognition or distillation, an easy way to do this is simply masking or cropping the image and then sending the obtained image to the pretrained image encoder.

Original: This method has been widely used in several open vocabulary object detection methods (Zhong et al., 2022; Gu et al., 2022).
Revised: This method has been widely used in several open vocabulary object detection methods (Zhong et al., 2022; Gu et al., 2022).

Original: The problem is that it’s not computation efficient ( N masks/boxes will lead to N images and they will be computed through the image encoder independently) and also loses the ability to see the global image context information which is very important for recognizing some objects and stuff.
Revised: The problem is that it’s not computation efficient ( N masks/boxes will lead to N images and they will be computed through the image encoder independently) and also loses the ability to see the global image context information which is very important for recognizing some objects and stuff.

Original: For masking, another problem is that masks are in different shapes and simply masking the image will cause the resulted image to have transparent background which usually doesn’t exist in real images that are used for training in large language-vision models e.g. CLIP.
Revised: For masking, another problem is that masks are in different shapes and simply masking the image will cause the resulted image to have transparent background which usually doesn’t exist in real images that are used for training in large language-vision models e.g. CLIP.

Original: To solve this, we propose Mask Class Tokens for efficient feature extraction from images without losing the global image context information.
Revised: To solve this, we propose Mask Class Tokens for efficient feature extraction from images without losing the global image context information.

Original: In the original CLIP ViT-based visual encoder framework, the input of the network is N image tokens and 1 class token.
Revised: In the original CLIP ViT-based visual encoder framework, the input of the network is N image tokens and 1 class token.

Original: The final output of the class token will be used for the relation computation with the text embeddings.
Revised: The final output of the class token will be used for the relation computation with the text embeddings.

Original: Our newly introduced M Mask
Revised: Our newly introduced M Mask

Original: Class Tokens will be alongside with the image tokens and the class token.
Revised: Class Tokens will be alongside with the image tokens and the class token.

Original: The embeddings’ weights of the Mask Class Token are provided by the class token in the pretrained CLIP ViT model and are fixed.
Revised: The embeddings’ weights of the Mask Class Token are provided by the class token in the pretrained CLIP ViT model and are fixed.

Original: Each Mask Class Token will output a corresponding dense image feature similar to the class token which outputs the feature of the whole image.
Revised: Each Mask Class Token will output a corresponding dense image feature similar to the class token which outputs the feature of the whole image.

Original: To achieve this, we design an attention mask as following
Revised: To achieve this, we design an attention mask as following

Original: [Equation]
Revised: [Equation]

Original: in which M is the number of Mask Class Tokens, N is the number of image tokens, T m × n is an m × n True matrix, F m × n is an m × n False matrix and M ′ is defined as following:
Revised: in which M is the number of Mask Class Tokens, N is the number of image tokens, T m × n is an m × n True matrix, F m × n is an m × n False matrix and M ′ is defined as following:

Original: [Equation]
Revised: [Equation]

Original: where True means that this position is masked out i.e. not allowed to attend and False otherwise.
Revised: where True means that this position is masked out i.e. not allowed to attend and False otherwise.

Original: In our mask attention matrix M , F ( N +1) × ( N +1) shows the N Image Tokens and one Class Token are attending each other as in the original CLIP.
Revised: In our mask attention matrix M , F ( N +1) × ( N +1) shows the N Image Tokens and one Class Token are attending each other as in the original CLIP.

Original: T ( N +1)
Revised: T ( N +1)

Original: × M shows that the N Image Tokens and one Class Token are not attending the M Mask Class Tokens.
Revised: × M shows that the N Image Tokens and one Class Token are not attending the M Mask Class Tokens.

Original: M ′ M × N shows that the Mask Class
Revised: M ′ M × N shows that the Mask Class

Original: Tokens are attending the Image Tokens given the corresponding masks.
Revised: Tokens are attending the Image Tokens given the corresponding masks.

Original: F M × 1 shows that the M
Revised: F M × 1 shows that the M

Original: Mask Class Tokens are attending the Class Token.
Revised: Mask Class Tokens are attending the Class Token.

Original: T M × M shows that the M Mask Class Tokens are not interacting with each other.
Revised: T M × M shows that the M Mask Class Tokens are not interacting with each other.

Original: In this way, each Mask Class Token will learn from the corresponding mask area of the images.
Revised: In this way, each Mask Class Token will learn from the corresponding mask area of the images.

Original: The image tokens are also interacting with each other which means the global information won’t lose.
Revised: The image tokens are also interacting with each other which means the global information won’t lose.

Original: And it’s also very efficient since we don’t need to do redundant computing for each mask or finetune the pretrained model.
Revised: And it’s also very efficient since we don’t need to do redundant computing for each mask or finetune the pretrained model.

Original: However, the mask information are not fully utilized and they cannot be refined either.
Revised: However, the mask information are not fully utilized and they cannot be refined either.

Original: But we will see in the experiments later that simply adpoting Mask Class Tokens to the pretrained CLIP model without any finetuning will already serve as a competitive baseline.
Revised: But we will see in the experiments later that simply adpoting Mask Class Tokens to the pretrained CLIP model without any finetuning will already serve as a competitive baseline.

Original: 3.2. R ELATIVE M ASK A TTENTION
Revised: 3.2. R ELATIVE M ASK A TTENTION

Original: [Figure]
Revised: [Figure]

Original: To further utilize the mask information and refine the coarse masks, we propose Relative Mask
Revised: To further utilize the mask information and refine the coarse masks, we propose Relative Mask

Original: Attention mechanism in our transformer.
Revised: Attention mechanism in our transformer.

Original: Our key design principle is try not to change the CLIP features directly as this would destroy the learned relationship between the image features and text features in the CLIP model.
Revised: Our key design principle is try not to change the CLIP features directly as this would destroy the learned relationship between the image features and text features in the CLIP model.

Original: Therefore, we adopt a way to only change the attention matrix in the transformer to learn a better linear combination of the values in the attention layers according to the mask information.
Revised: Therefore, we adopt a way to only change the attention matrix in the transformer to learn a better linear combination of the values in the attention layers according to the mask information.

Original: As in Figure 2, our proposed Relative Mask Attention Mechanism only changes the attention matrix and refines the masks.
Revised: As in Figure 2, our proposed Relative Mask Attention Mechanism only changes the attention matrix and refines the masks.

Original: M r is defined in Eq.
Revised: M r is defined in Eq.

Original: A ′ :
Revised: A ′ :

Original: M, − N : is defined in Eq.
Revised: M, − N : is defined in Eq.

Original: f M is the class-agnostic mask proposal network.
Revised: f M is the class-agnostic mask proposal network.

Original: f 1 and f 2 are two downsampling networks that encode the images/masks to image tokens/mask patch tokens sharing the same architecture.
Revised: f 1 and f 2 are two downsampling networks that encode the images/masks to image tokens/mask patch tokens sharing the same architecture.

Original: f r is a two-layer convolutional network that maps the attention matrix to a mask residual.
Revised: f r is a two-layer convolutional network that maps the attention matrix to a mask residual.

Original: Similar to relative positional encoding, we use a relative attention mechanism here.
Revised: Similar to relative positional encoding, we use a relative attention mechanism here.

Original: Let D be the dimension of the token embedding, for each Mask Class Token T MC i ∈
Revised: Let D be the dimension of the token embedding, for each Mask Class Token T MC i ∈

Original: R D with a corresponding mask K i ∈
Revised: R D with a corresponding mask K i ∈

Original: R H × W whose shape is the same as the image, we use a similar way as for the images to get mask patch tokens T MP ∈ R M × N × D in the computation of the attention.
Revised: R H × W whose shape is the same as the image, we use a similar way as for the images to get mask patch tokens T MP ∈ R M × N × D in the computation of the attention.

Original: In our attention matrix, the Mask Class Tokens attending image tokens part will then be as following:
Revised: In our attention matrix, the Mask Class Tokens attending image tokens part will then be as following:

Original: [Equation]
Revised: [Equation]

Original: where T IM ∈ R N × D is image tokens, T MC ∈ R M × D is Mask Class Tokens, T MP ∈ R M × N × D is
Revised: where T IM ∈ R N × D is image tokens, T MC ∈ R M × D is Mask Class Tokens, T MP ∈ R M × N × D is

Original: Mask Patch Tokens ϕ Q , ϕ K , ϕ Q m , ϕ K m are linear transformations, ⊙ is element-wise product and (cid:80)
Revised: Mask Patch Tokens ϕ Q , ϕ K , ϕ Q m , ϕ K m are linear transformations, ⊙ is element-wise product and (cid:80)

Original: Dc ( · ) c is the sum of the embedding dimension.
Revised: Dc ( · ) c is the sum of the embedding dimension.

Original: ϕ
Revised: ϕ

Original: K m (
Revised: K m (

Original: T IM ) ∈ R N × D will first be broadcast to
Revised: T IM ) ∈ R N × D will first be broadcast to

Original: R M × N × D before doing element-wise production.
Revised: R M × N × D before doing element-wise production.

Original: The attention will also in turn be used for the refinement of the masks.
Revised: The attention will also in turn be used for the refinement of the masks.

Original: The vanilla attention can be seen as a relationship between each mask area and all the image patches.
Revised: The vanilla attention can be seen as a relationship between each mask area and all the image patches.

Original: Thus we utilize this to help our coarse masks be more accurate.
Revised: Thus we utilize this to help our coarse masks be more accurate.

Original: The updating process of the masks is as following:
Revised: The updating process of the masks is as following:

Original: [Equation]
Revised: [Equation]

Original: where M c , M r ∈ R N × H × W denotes the coarse mask and refined mask respectively, f r is a learnable non-linear function that maps the attention matrix to a mask residual, σ and σ − 1 are sigmoid and inverse sigmoid functions respectively.
Revised: where M c , M r ∈ R N × H × W denotes the coarse mask and refined mask respectively, f r is a learnable non-linear function that maps the attention matrix to a mask residual, σ and σ − 1 are sigmoid and inverse sigmoid functions respectively.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this part, we train our proposed MaskCLIP method using COCO (Lin et al., 2014) training data and test on other datasets (ADE20K (Zhou et al., 2019; 2017), PASCAL Context (Mottaghi et al., 2014),
Revised: In this part, we train our proposed MaskCLIP method using COCO (Lin et al., 2014) training data and test on other datasets (ADE20K (Zhou et al., 2019; 2017), PASCAL Context (Mottaghi et al., 2014),

Original: LVIS) under the open vocabulary setting.
Revised: LVIS) under the open vocabulary setting.

Original: Due to the novel setting of the open vocabulary panoptic segmentation task, we also compare our performance on open vocabulary semantic segmentation with previous methods.
Revised: Due to the novel setting of the open vocabulary panoptic segmentation task, we also compare our performance on open vocabulary semantic segmentation with previous methods.

Original: Apart from the quantitative results, we also provide qualitative results to validate our method that has good ability to learn dense image features and can support user-specified arbitrary categories.
Revised: Apart from the quantitative results, we also provide qualitative results to validate our method that has good ability to learn dense image features and can support user-specified arbitrary categories.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: COCO: COCO (Lin et al., 2014) includes 133 classes where 80 classes are things and 53 classes are stuff or background.
Revised: COCO: COCO (Lin et al., 2014) includes 133 classes where 80 classes are things and 53 classes are stuff or background.

Original: There are 118k training images and 5k validation images.
Revised: There are 118k training images and 5k validation images.

Original: In our experiments, we first train the class-agnostic mask proposal network on COCO training dataset using the annotations of panoptic masks.
Revised: In our experiments, we first train the class-agnostic mask proposal network on COCO training dataset using the annotations of panoptic masks.

Original: Then we train our models on COCO training images in a supervised manner.
Revised: Then we train our models on COCO training images in a supervised manner.

Original: ADE20K: ADE20K
Revised: ADE20K: ADE20K

Original: (Zhou et al., 2019; 2017) contains 20,210 images and annotations for training and 2000 images and annotations for validation.
Revised: (Zhou et al., 2019; 2017) contains 20,210 images and annotations for training and 2000 images and annotations for validation.

Original: It serves both panoptic segmentation and semantic segmentation.
Revised: It serves both panoptic segmentation and semantic segmentation.

Original: The full version (A-847) (Zhou et al., 2019) includes 847 classes and the short version (A-150) (Zhou et al., 2017) includes 150 classes.
Revised: The full version (A-847) (Zhou et al., 2019) includes 847 classes and the short version (A-150) (Zhou et al., 2017) includes 150 classes.

Original: We use the validation set in ADE20K for testing without any training on this dataset in which case we can test our model’s capability of open vocabulary segmentation.
Revised: We use the validation set in ADE20K for testing without any training on this dataset in which case we can test our model’s capability of open vocabulary segmentation.

Original: PASCAL Context: PASCAL Context (Mottaghi et al., 2014) contains 10,103 per-pixel annotations for images of PASCAL VOC 2010 (Everingham et al.), where 4998 for training and 5105 for validation.
Revised: PASCAL Context: PASCAL Context (Mottaghi et al., 2014) contains 10,103 per-pixel annotations for images of PASCAL VOC 2010 (Everingham et al.), where 4998 for training and 5105 for validation.

Original: The full version (P-459) includes 459 classes and the short version includes 59 classes.
Revised: The full version (P-459) includes 459 classes and the short version includes 59 classes.

Original: This dataset serves as another benchmark testing our model’s open vocabulary segmentation abiltiy.
Revised: This dataset serves as another benchmark testing our model’s open vocabulary segmentation abiltiy.

Original: LVIS: LVIS (Gupta et al., 2019) contains 100,170 images for training and 19,809 images for validation.
Revised: LVIS: LVIS (Gupta et al., 2019) contains 100,170 images for training and 19,809 images for validation.

Original: It extends COCO (Lin et al., 2014) but contains 1,203 categories.
Revised: It extends COCO (Lin et al., 2014) but contains 1,203 categories.

Original: It is considered as one of the most challenging benchmark for instance segmentation because of its large vocabulary, long-tailed distribution, and fine-grained classification.
Revised: It is considered as one of the most challenging benchmark for instance segmentation because of its large vocabulary, long-tailed distribution, and fine-grained classification.

Original: We report our model’s performance of open vocabulary instance segmentation on the validation dataset.
Revised: We report our model’s performance of open vocabulary instance segmentation on the validation dataset.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Class-Agnostic Mask Proposal Network.
Revised: Class-Agnostic Mask Proposal Network.

Original: In our first stage, we train a class-agnostic mask proposal network using MaskRCNN (He et al., 2017) and Mask2Former (Cheng et al., 2022) on COCO training data.
Revised: In our first stage, we train a class-agnostic mask proposal network using MaskRCNN (He et al., 2017) and Mask2Former (Cheng et al., 2022) on COCO training data.

Original: The experiment setting we use for MaskRCNN is R50-FPN-1x.
Revised: The experiment setting we use for MaskRCNN is R50-FPN-1x.

Original: The backbone we use in Mask2Former is ResNet-50.
Revised: The backbone we use in Mask2Former is ResNet-50.

Original: All the training setting follows the default in their models.
Revised: All the training setting follows the default in their models.

Original: CLIP Baseline.
Revised: CLIP Baseline.

Original: We design our first baseline by directly using the class-agnostic mask proposal network from the first stage and the pretrained CLIP model.
Revised: We design our first baseline by directly using the class-agnostic mask proposal network from the first stage and the pretrained CLIP model.

Original: We mask the images according to the masks from the class-agnostic mask proposal network and send the masked images to the CLIP model to get classification results.
Revised: We mask the images according to the masks from the class-agnostic mask proposal network and send the masked images to the CLIP model to get classification results.

Original: The pretrained CLIP model we use is ViT-L/14@336px and the text inputs we use are simply the category names defined by each dataset.
Revised: The pretrained CLIP model we use is ViT-L/14@336px and the text inputs we use are simply the category names defined by each dataset.

Original: Those two settings keep the same with the following two methods for fair comparison.
Revised: Those two settings keep the same with the following two methods for fair comparison.

Original: MaskCLIP w/o RMA Baseline.
Revised: MaskCLIP w/o RMA Baseline.

Original: Our second
Revised: Our second

Original: baseline is based on the Mask Class Tokens which doesn’t use the Relative Mask Attention mechanism.
Revised: baseline is based on the Mask Class Tokens which doesn’t use the Relative Mask Attention mechanism.

Original: Instead of masking the images and sending the resulted images directly to the CLIP model for feature extraction, we use Mask Class Tokens to acquire the corresponding dense image features.
Revised: Instead of masking the images and sending the resulted images directly to the CLIP model for feature extraction, we use Mask Class Tokens to acquire the corresponding dense image features.

Original: The obtained image features will then be used for further open vocabulary classification.
Revised: The obtained image features will then be used for further open vocabulary classification.

Original: The two baselines above don’t need any training in the second stage and can be used to directly perform the open vocabulary tasks.
Revised: The two baselines above don’t need any training in the second stage and can be used to directly perform the open vocabulary tasks.

Original: We will demonstrate that the second baseline is better at feature extraction in both quantitative results and qualitative results under the open vocabulary setting and show the effectiveness and efficiency of the proposed Mask Class Tokens.
Revised: We will demonstrate that the second baseline is better at feature extraction in both quantitative results and qualitative results under the open vocabulary setting and show the effectiveness and efficiency of the proposed Mask Class Tokens.

Original: MaskCLIP.
Revised: MaskCLIP.

Original: In our MaskCLIP method, we still use the CLIP ViT-L/14@336px pretrained model as with the previous two.
Revised: In our MaskCLIP method, we still use the CLIP ViT-L/14@336px pretrained model as with the previous two.

Original: This model has 24 attention layers and we add Relative Mask Attention in four of them which is 6, 12, 18, 24.
Revised: This model has 24 attention layers and we add Relative Mask Attention in four of them which is 6, 12, 18, 24.

Original: We use AdamW (Loshchilov & Hutter, 2019) as our optimizer and the learning rate is set to 0.0001.
Revised: We use AdamW (Loshchilov & Hutter, 2019) as our optimizer and the learning rate is set to 0.0001.

Original: We train our model on COCO training data for 10k iterations with a batchsize of 8.
Revised: We train our model on COCO training data for 10k iterations with a batchsize of 8.

Original: The training takes around 3h on 8 Nvidia A5000 GPUs.
Revised: The training takes around 3h on 8 Nvidia A5000 GPUs.

Original: Loss Function.
Revised: Loss Function.

Original: The loss function is L = λ
Revised: The loss function is L = λ

Original: ce L ce + λ dice L dice + λ bce L bce , where L ce is the loss for classification, L dice and L bce are the losses for mask localization.
Revised: ce L ce + λ dice L dice + λ bce L bce , where L ce is the loss for classification, L dice and L bce are the losses for mask localization.

Original: In our experiments, We set λ ce = 2 , λ dice = 5 , λ bce = 5 .
Revised: In our experiments, We set λ ce = 2 , λ dice = 5 , λ bce = 5 .

Original: In next three parts, we evaluate our methods on open vocabulary panoptic, instance segmentation, and semantic segmentation tasks.
Revised: In next three parts, we evaluate our methods on open vocabulary panoptic, instance segmentation, and semantic segmentation tasks.

Original: The class-agnostic mask proposal networks we use in those methods are trained using Mask2Former other than noted.
Revised: The class-agnostic mask proposal networks we use in those methods are trained using Mask2Former other than noted.

Original: 4. O PEN -V OCABULARY P ANOPTIC S EGMENTATION
Revised: 4. O PEN -V OCABULARY P ANOPTIC S EGMENTATION

Original: First, we compare our MaskCLIP with the two baselines on ADE20K validation set under the open vocabulary panoptic segmentation setting.
Revised: First, we compare our MaskCLIP with the two baselines on ADE20K validation set under the open vocabulary panoptic segmentation setting.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: As can be seen from the table, the MaskCLIP w/o RMA baseline performs better on all the metrics in panoptic segmentation setting which demonstrates that our feature extraction method is better than the vanilla way in this setting.
Revised: As can be seen from the table, the MaskCLIP w/o RMA baseline performs better on all the metrics in panoptic segmentation setting which demonstrates that our feature extraction method is better than the vanilla way in this setting.

Original: It extracts the features without the need to changing the input and can simultaneously extract multiple mask area features easily.
Revised: It extracts the features without the need to changing the input and can simultaneously extract multiple mask area features easily.

Original: For 100 masks’ feature extraction in a single image, the CLIP baseline takes ~3s on a single 3090 GPU while the MaskCLIP w/o RMA baseline only takes ~0.6s which is ~4x faster.
Revised: For 100 masks’ feature extraction in a single image, the CLIP baseline takes ~3s on a single 3090 GPU while the MaskCLIP w/o RMA baseline only takes ~0.6s which is ~4x faster.

Original: Our MaskCLIP beats both baselines significantly as it utilizes the accurate mask information and refines the masks during the feature extraction process.
Revised: Our MaskCLIP beats both baselines significantly as it utilizes the accurate mask information and refines the masks during the feature extraction process.

Original: [Table]
Revised: [Table]

Original: In this part, we show two sets of images to demonstrate our model capability.
Revised: In this part, we show two sets of images to demonstrate our model capability.

Original: The first is the qualitative results on ADE20K.
Revised: The first is the qualitative results on ADE20K.

Original: We compare our method with the two baselines in Figure 3.
Revised: We compare our method with the two baselines in Figure 3.

Original: It can be seen that our method performs much better than the two baselines.
Revised: It can be seen that our method performs much better than the two baselines.

Original: The results from the first column show that due to the lack of global information, CLIP baseline failes to predict the floor.
Revised: The results from the first column show that due to the lack of global information, CLIP baseline failes to predict the floor.

Original: Instead it predicts skyscraper.
Revised: Instead it predicts skyscraper.

Original: This is an easy case but if only floor area is provided it does have some similarities with the wall of a skyscraper.
Revised: This is an easy case but if only floor area is provided it does have some similarities with the wall of a skyscraper.

Original: While the MaskCLIP w/o RMA baseline and MaskCLIP model can predict the floor correctly with the the global image context information.
Revised: While the MaskCLIP w/o RMA baseline and MaskCLIP model can predict the floor correctly with the the global image context information.

Original: I npu t I m a g e
Revised: I npu t I m a g e

Original: C L I P
Revised: C L I P

Original: B a s e li n e
Revised: B a s e li n e

Original: M a s k
Revised: M a s k

Original: C L I P w / o
Revised: C L I P w / o

Original: R M
Revised: R M

Original: A
Revised: A

Original: M a s k
Revised: M a s k

Original: C L I P
Revised: C L I P

Original: Figure
Revised: Figure

Original: The second set of images we’re presenting are in Figure 4.
Revised: The second set of images we’re presenting are in Figure 4.

Original: These figures show our capability of specifying any arbitrary classes in performing panoptic segmentation task.
Revised: These figures show our capability of specifying any arbitrary classes in performing panoptic segmentation task.

Original: The results show that though we train a new model based on the CLIP model without any distillation methods, we can still preserve the CLIP image features very well.
Revised: The results show that though we train a new model based on the CLIP model without any distillation methods, we can still preserve the CLIP image features very well.

Original: Our model doesn’t have a clear bias towards the base classes in the training set and could tell the difference very well that have no chance to learn in the
Revised: Our model doesn’t have a clear bias towards the base classes in the training set and could tell the difference very well that have no chance to learn in the

Original: COCO training: e.g toy vs real and filled vs empty.
Revised: COCO training: e.g toy vs real and filled vs empty.

Original: [Figure]
Revised: [Figure]

Original: 4. O PEN -V OCABULARY S EMANTIC S EGMENTATION /L ABELING
Revised: 4. O PEN -V OCABULARY S EMANTIC S EGMENTATION /L ABELING

Original: We also use our method to compare with open-vocabulary semantic segmentation as in Table 3.
Revised: We also use our method to compare with open-vocabulary semantic segmentation as in Table 3.

Original: The setting is similar, they all train on COCO panoptic training set and test on ADE20K validation set
Revised: The setting is similar, they all train on COCO panoptic training set and test on ADE20K validation set

Original: On the four datasets we test, MaskCLIP reaches the state-of-the-art results on three of them with only
Revised: On the four datasets we test, MaskCLIP reaches the state-of-the-art results on three of them with only

Original: P-59 slightly lower.
Revised: P-59 slightly lower.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: To compare with previous methods, we also provide a semantic segmentation comparison in Figure 5.
Revised: To compare with previous methods, we also provide a semantic segmentation comparison in Figure 5.

Original: Results on ALIGN++ and OpenSeg are directly from (Ghiasi et al., 2022) and we run the same image using our MaskCLIP model.
Revised: Results on ALIGN++ and OpenSeg are directly from (Ghiasi et al., 2022) and we run the same image using our MaskCLIP model.

Original: It can be seen that due to the open vocabulary setting, some similar classes may be mistakenly classified e.g. all three methods predict the house in this image while the ground truth is building.
Revised: It can be seen that due to the open vocabulary setting, some similar classes may be mistakenly classified e.g. all three methods predict the house in this image while the ground truth is building.

Original: [Figure]
Revised: [Figure]

Original: 4. O PEN -V OCABULARY I NSTANCE S EGMENTATION
Revised: 4. O PEN -V OCABULARY I NSTANCE S EGMENTATION

Original: Cross-Dataset Setting.
Revised: Cross-Dataset Setting.

Original: We present the results on open vocabulary instance segmentation in Table 4 under the cross-dataset setting.
Revised: We present the results on open vocabulary instance segmentation in Table 4 under the cross-dataset setting.

Original: Since instance segmentation can be regarded as “thing-only“ panoptic segmentation, we directly apply our model trained on coco panoptic dataset to the instance segmentation task.
Revised: Since instance segmentation can be regarded as “thing-only“ panoptic segmentation, we directly apply our model trained on coco panoptic dataset to the instance segmentation task.

Original: MaskCLIP with different class-agnostic mask proposal networks perform better than CLIP Baseline and MaskCLIP w/o RMA in general.
Revised: MaskCLIP with different class-agnostic mask proposal networks perform better than CLIP Baseline and MaskCLIP w/o RMA in general.

Original: [Table]
Revised: [Table]

Original: COCO Split Setting.
Revised: COCO Split Setting.

Original: Besides the cross-dataset setting
Revised: Besides the cross-dataset setting

Original: [Table]
Revised: [Table]

Original: we also follow the COCO Split Setting in
Revised: we also follow the COCO Split Setting in

Original: XPM(Huynh et al., 2022) to perform the instance segmentation in Table 5.
Revised: XPM(Huynh et al., 2022) to perform the instance segmentation in Table 5.

Original: On the generalized setting which is a more challenging setting
Revised: On the generalized setting which is a more challenging setting

Original: [Table]
Revised: [Table]

Original: we outperforms previous results in base, target and all categories.
Revised: we outperforms previous results in base, target and all categories.

Original: On the constrained setting, we also show competitive results in both base and target categories
Revised: On the constrained setting, we also show competitive results in both base and target categories.

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  A BLATION S TUDY
Revised:  A BLATION S TUDY

Original: Incorporating GT Masks.
Revised: Incorporating GT Masks.

Original: Since our model can decouple the mask proposal process and the classification process, we could also use the ground truth mask proposals which can be regarded as a “perfect” mask proposal network in our method.
Revised: Since our model can decouple the mask proposal process and the classification process, we could also use the ground truth mask proposals which can be regarded as a “perfect” mask proposal network in our method.

Original: In this way, we can eliminate the effects of quality of the mask proposals and inspect the method’s classification capabilities.
Revised: In this way, we can eliminate the effects of quality of the mask proposals and inspect the method’s classification capabilities.

Original: In Table 6.
Revised: In Table 6.

Original: We can see that the performance could gain a lot from the “perfect” mask proposals.
Revised: We can see that the performance could gain a lot from the “perfect” mask proposals.

Original: And our MaskCLIP method also outperforms OpenSeg in this setting.
Revised: And our MaskCLIP method also outperforms OpenSeg in this setting.

Original: [Table]
Revised: [Table]

Original: Mask Refinement.
Revised: Mask Refinement.

Original: In our Relative Mask Attention part, the attention layer will use the accurate mask information to learn a better attention matrix and the mask will also use the attention information to gradually refine itself.
Revised: In our Relative Mask Attention part, the attention layer will use the accurate mask information to learn a better attention matrix and the mask will also use the attention information to gradually refine itself.

Original: In this ablation study [Table] we only let the attention matrix learn from the mask without any mask refinement.
Revised: In this ablation study, we only let the attention matrix learn from the mask without any mask refinement.

Original: And we get the results in Table 7.
Revised: And we get the results in Table 7.

Original: Since the SQ reflects the segemtation quality, we care more about SQ here.
Revised: Since the SQ reflects the segemtation quality, we care more about SQ here.

Original: It can be seen that MaskCLIP performs slightly better than that without the mask refinement which demonstrates the effectivity of the mask refinement.
Revised: It can be seen that MaskCLIP performs slightly better than that without the mask refinement which demonstrates the effectivity of the mask refinement.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented a new algorithm, MaskCLIP, to tackle an emerging computer vision task, open-vocabulary panoptic segmentation.
Revised: In this paper, we have presented a new algorithm, MaskCLIP, to tackle an emerging computer vision task, open-vocabulary panoptic segmentation.

Original: MaskCLIP is a Transformer-based approach using mask queries with the ViT-based CLIP backbone to efficiently and effectively utilize pre-trained dense/local CLIP features.
Revised: MaskCLIP is a Transformer-based approach using mask queries with the ViT-based CLIP backbone to efficiently and effectively utilize pre-trained dense/local CLIP features.

Original: MaskCLIP consists of a Relative Mask Attention (RMA) module that is seamlessly integrated with a pre-trained CLIP.
Revised: MaskCLIP consists of a Relative Mask Attention (RMA) module that is seamlessly integrated with a pre-trained CLIP.

Original: MaskCLIP is distinct compared with existing approaches in open-vocabulary semantic segmentation/object detection by building an integrated encoder module for segmentation mask refinement and image feature extraction with a pre-trainedCLIP image model. Encouraging experimental results on open-vocabulary panoptic/semantic/instance segmentation have been obtained.
Revised: MaskCLIP is distinct compared with existing approaches in open-vocabulary semantic segmentation/object detection by building an integrated encoder module for segmentation mask refinement and image feature extraction with a pre-trained

Original: A CLIP B ASELINE D ETAILS
Revised: A CLIP B ASELINE D ETAILS

Original: Here we provide more details on our CLIP Baseline.
Revised: Here we provide more details on our CLIP Baseline.

Original: Given an RGB image
Revised: Given an RGB image

Original: I ∈ R H
Revised: I ∈ R H

Original: × W × 3 with height
Revised: × W × 3 with height

Original: H and width W and a list of category names with C classes, we precompute the text embedding of the category names as E ∈ R C × D .
Revised: H and width W and a list of category names with C classes, we precompute the text embedding of the category names as E ∈ R C × D .

Original: The mask proposal network f m outputs N masks M ∈ R N × H × W .
Revised: The mask proposal network f m outputs N masks M ∈ R N × H × W .

Original: For each mask: the cropped image region is the element-wise product between the binary mask
Revised: For each mask: the cropped image region is the element-wise product between the binary mask

Original: M i and the image I , i.e. R i ∈
Revised: M i and the image I , i.e. R i ∈

Original: R H
Revised: R H

Original: × W × 

Revised: × W ×

Original: ; the visual embedding V i ∈ R D of the cropped region is computed by the visual encoder where D is the hidden dimension; the final classification score
Revised: ; the visual embedding V i ∈ R D of the cropped region is computed by the visual encoder where D is the hidden dimension; the final classification score

Original: Y i ∈
Revised: Y i ∈

Original: R C is the softmax over the dot product between the visual embedding V i and the text embedding
Revised: R C is the softmax over the dot product between the visual embedding V i and the text embedding

Original: T .
Revised: T .

Original: A formal algorithm is described as 1 and a visualization of this is shown as 6.
Revised: A formal algorithm is described as 1 and a visualization of this is shown as 6.

Original: Algorithm 1 CLIP Baseline
Revised: Algorithm 1 CLIP Baseline

Original: Require:
Revised: Require:

Original: Mask proposal network f m , CLIP visual encoder f v , CLIP text encoder f t .
Revised: Mask proposal network f m , CLIP visual encoder f v , CLIP text encoder f t .

Original: Given an image I ∈
Revised: Given an image I ∈

Original: R H
Revised: R H

Original: × W × 3 and a list T containing C category names.
Revised: × W × 3 and a list T containing C category names.

Original: E = f t ( T ) .
Revised: E = f t ( T ) .

Original: M = f m ( I ) .
Revised: M = f m ( I ) .

Original: for t = 1 , 2 , . . .
Revised: for t = 1 , 2 , . . .

Original: , N do
Revised: , N do

Original: R i =
Revised: R i =

Original: M i ⊙ I .
Revised: M i ⊙ I .

Original: V i =
Revised: V i =

Original: f v ( R i ) .
Revised: f v ( R i ) .

Original: Y i =
Revised: Y i =

Original: softmax( E ⊗ V i ) .
Revised: softmax( E ⊗ V i ) .

Original: end for
Revised: end for

Original: Category Names:
Revised: Category Names:

Original: [Figure]
Revised: [Figure]

Original: B A BLATION ON USING R ELATIVE M ASK A TTENTION IN D IFFERENT L AYERS
Revised: B A BLATION ON USING R ELATIVE M ASK A TTENTION IN D IFFERENT L AYERS

Original: In this part we conduct an ablation study on using different layers for relative mask attention.
Revised: In this part we conduct an ablation study on using different layers for relative mask attention.

Original: Since our pretrained CLIP model is fixed during the whole training procedure, whether each layer would help the final results remains a question.
Revised: Since our pretrained CLIP model is fixed during the whole training procedure, whether each layer would help the final results remains a question.

Original: We use four different kinds of layers combination of the layers in this part and provide the results in Table 8.
Revised: We use four different kinds of layers combination of the layers in this part and provide the results in Table 8.

Original: We can see that the last layer is a key part of our results since the features are gradually learned through all the attention layer.
Revised: We can see that the last layer is a key part of our results since the features are gradually learned through all the attention layer.

Original: Though the last four layers’ features should the best, the performance wouldn’t be better if Relative Mask Attention is only used in the last four layers.
Revised: Though the last four layers’ features should the best, the performance wouldn’t be better if Relative Mask Attention is only used in the last four layers.

Original: This is also reasonable since the network should not have the accurate mask information too late.
Revised: This is also reasonable since the network should not have the accurate mask information too late.

Original: C M ORE V ISUALIZATION R ESULTS ON A RBITRARY C ATEGORIES
Revised: C M ORE V ISUALIZATION R ESULTS ON A RBITRARY C ATEGORIES

Original: In this part, we provide more visualization results on user-specified class discoveries in Figure 7.
Revised: In this part, we provide more visualization results on user-specified class discoveries in Figure 7.

Original: We select some very close text prompts such as “four-leg animal" and “two-leg animal"; “car”, “truck” and “SUV” and find that our method can still classify them.
Revised: We select some very close text prompts such as “four-leg animal" and “two-leg animal"; “car”, “truck” and “SUV” and find that our method can still classify them.

Original: We also show another result which is “person identification” in Figure 7 (c) which shows our model preserve the dense/local CLIP features rather well.
Revised: We also show another result which is “person identification” in Figure 7 (c) which shows our model preserve the dense/local CLIP features rather well.

Original: Different
Revised: Different

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: D Q UALITATIVE R ESULTS ON ADE-20K I NSTANCE S EGMENTATION
Revised: D Q UALITATIVE R ESULTS ON ADE-20K I NSTANCE S EGMENTATION

Original: We also show qualitative results on ADE20K comparing the CLIP Baseline, MaskCLIP w/o RMA and
Revised: We also show qualitative results on ADE20K comparing the CLIP Baseline, MaskCLIP w/o RMA and

Original: MaskCLIP.
Revised: MaskCLIP.

Original: As shown in Figure 8, MaskCLIP is much better than the CLIP Baseline and MaskCLIP w/o RMA.
Revised: As shown in Figure 8, MaskCLIP is much better than the CLIP Baseline and MaskCLIP w/o RMA.

Original: Since the mask proposal network is class-agnostic, some masks that are not objects would actually be predicted in which case the classification later will be very important as it may be classified as some object classes in the dataset.
Revised: Since the mask proposal network is class-agnostic, some masks that are not objects would actually be predicted in which case the classification later will be very important as it may be classified as some object classes in the dataset.

Original: The visualization results of MaskCLIP contains fewer non-object masks and are more accurate in class prediction.
Revised: The visualization results of MaskCLIP contains fewer non-object masks and are more accurate in class prediction.

Original: [Figure]
Revised: [Figure]

Original: E COCO V ALIDATION R ESULTS
Revised: E COCO V ALIDATION R ESULTS

Original: In this part, we provide the COCO validation results.
Revised: In this part, we provide the COCO validation results.

Original: The model is trained on COCO panotpic training data and evaluated on COCO validation data.
Revised: The model is trained on COCO panotpic training data and evaluated on COCO validation data.

Original: This results provide more comparison on model’s
Revised: This results provide more comparison on model’s

Original: [Table]
Revised: [Table]

Original: ALIGNJia et al.
Revised: ALIGNJia et al.

Original: [Table]
Revised: [Table]

Original: ALIGN w/ proposalJia et al.
Revised: ALIGN w/ proposalJia et al.

Original: [Table]
Revised: [Table]

Original: LSeg+Jia et al.
Revised: LSeg+Jia et al.

Original: [Table]
Revised: [Table]

Original: OpenSegGhiasi et al.
Revised: OpenSegGhiasi et al.

Original: [Table]
Revised: [Table]

Original: MaskCLIP w/o RMA
Revised: MaskCLIP w/o RMA

Original: [Table]
Revised: [Table]

Original: MaskCLIP
Revised: MaskCLIP

Original: [Table]
Revised: [Table]

Original: F E FFICIENCY A NALYSIS .
Revised: F E FFICIENCY A NALYSIS .

Original: We also provide the FLOPs for differnt models in Table 10.
Revised: We also provide the FLOPs for differnt models in Table 10.

Original: We use an input resolution of 640x here.
Revised: We use an input resolution of 640x here.

Original: The CLIP visual encoder we use is ViT-L/14@336px.
Revised: The CLIP visual encoder we use is ViT-L/14@336px.

Original: As we can see, our MaskCLIP w/o RMA and MaskCLIP’s FLOPs are much lower than the CLIP Baseline which needs to send each image to the CLIP Visual Encoder.
Revised: As we can see, our MaskCLIP w/o RMA and MaskCLIP’s FLOPs are much lower than the CLIP Baseline which needs to send each image to the CLIP Visual Encoder.

Original: [Table]
Revised: [Table]