Original: V IEW C O : D ISCOVERING T EXT -S UPERVISED S EGMENTATION M ASKS VIA M ULTI -V IEW S EMANTIC C ONSISTENCY
Revised: V IEW C O : D ISCOVERING T EXT -S UPERVISED S EGMENTATION M ASKS VIA M ULTI -V IEW S EMANTIC C ONSISTENCY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation.
Revised: Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation.

Original: However, existing works focus on pixel grouping and cross-modal semantic alignment, while ignoring the correspondence among multiple augmented views of the same image.
Revised: However, existing works focus on pixel grouping and cross-modal semantic alignment, while ignoring the correspondence among multiple augmented views of the same image.

Original: To overcome such limitation, we propose multi- View Co nsistent learning (ViewCo) for text-supervised semantic segmentation.
Revised: To overcome such limitation, we propose multi- View Co nsistent learning (ViewCo) for text-supervised semantic segmentation.

Original: Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image.
Revised: Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image.

Original: Additionally, we propose cross-view segmentation consistency modeling to address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders.
Revised: Additionally, we propose cross-view segmentation consistency modeling to address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders.

Original: The text-to-views consistency benefits the dense assignment of the visual features by encouraging different crops to align with the same text, while the cross-view segmentation consistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks.
Revised: The text-to-views consistency benefits the dense assignment of the visual features by encouraging different crops to align with the same text, while the cross-view segmentation consistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks.

Original: Trained with large-scale image-text data, our model can directly segment objects of arbitrary categories in a zero-shot manner.
Revised: Trained with large-scale image-text data, our model can directly segment objects of arbitrary categories in a zero-shot manner.

Original: Extensive experiments show that ViewCo outperforms stateof-the-art methods on average by up to 2.9%, 1.6%, and 2.4% mIoU on PASCAL
Revised: Extensive experiments show that ViewCo outperforms stateof-the-art methods on average by up to 2.9%, 1.6%, and 2.4% mIoU on PASCAL

Original: VOC2012, PASCAL Context, and COCO, respectively.
Revised: VOC2012, PASCAL Context, and COCO, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, vision-language contrastive learning (Radford et al.
Revised: Recently, vision-language contrastive learning (Radford et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021a)) has attracted a lot of attention because it can obtain more generalized feature representation.
Revised: (2021a)) has attracted a lot of attention because it can obtain more generalized feature representation.

Original: And at the same time, it can also make use of abundant image-text pairs to avoid labor-intensive annotation costs.
Revised: And at the same time, it can also make use of abundant image-text pairs to avoid labor-intensive annotation costs.

Original: Vision-language pre-training (VLP) models have exhibited impressive potential in various visual (Xu et al.
Revised: Vision-language pre-training (VLP) models have exhibited [Figure] view text pull together positive pairs push away negative pairs distance between different objects

Revised: [Figure]

Revised: text view ？ impressive potential in various visual (Xu et al.

Original: ; Mu et al.
Revised: ; Mu et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: and multimodal (Wang et al.
Revised: and multimodal (Wang et al.

Original: (2021b);
Revised: (2021b)

Original:  Kim et al.
Revised: ; Kim et al.

Original: (2021)) tasks, including text-supervised semantic segmentation (Xu et al.
Revised: (2021)) tasks, including textsupervised semantic segmentation (Xu et al.

Original: ; Ghiasi et al.
Revised: ; Ghiasi et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: (2021); Zabari & Hoshen (2021)
Revised: (2021); Zabari & Hoshen (2021)

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2021a), which uses text instead of traditional dense labels for supervision to achieve zero-shot semantic segmentation.
Revised: (2021a), which uses text instead of traditional dense labels for supervision to achieve zero-shot semantic segmentation.

Original: It provides a feasible solution for learning segmentation masks without mask annotation.
Revised: It provides a feasible solution for learning segmentation masks without mask annotation.

Original: However, existing works with CLIP-based (Radford et al.
Revised: However, existing works with CLIP-based (Radford et al.

Original: (2021)) segmentation (Xu et al.
Revised: (2021)) segmentation (Xu et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2021a)) mainly focus on pixel grouping or cross-modal semantic alignment.
Revised: (2021a)) mainly focus on pixel grouping or cross-modal semantic alignment.

Original: They have the following two obvious limitations: (i) the excessive strictness of image-text correspondence; and (ii) the ambiguity of text description.
Revised: They have the following two obvious limitations: (i) the excessive strictness of image-text correspondence; and (ii) the ambiguity of text description.

Original: First , in vanilla vision-language contrastive learning, each image-text  pair is regarded as a unique positive pair, while all the other combinations are regarded as negative ones.
Revised: First , in vanilla vision-language contrastive learning, each image-text [Figure] pair is regarded as a unique positive pair, while all the other combinations are regarded as negative ones.

Original: This image-text correspondence is actually too rigorous.
Revised: This image-text correspondence is actually too rigorous.

Original: In fact, one textual description may correspond to different images.
Revised: In fact, one textual description may correspond to different images.

Original: The excessive strictness is not conducive to the model learning high-level cross-modal semantic correspondences.
Revised: The excessive strictness is not conducive to the model learning highlevel cross-modal semantic correspondences.

Original: Therefore, more relaxed vision-language contrastive learning needs to be considered.
Revised: Therefore, more relaxed vision-language contrastive learning needs to be considered.

Original: Second , the ambiguity of textual descriptions is also a key challenge.
Revised: Second , the ambiguity of textual descriptions is also a key challenge.

Original: Compared with the traditional semantic segmentation pipeline that uses dense annotations as supervision information (Touvron et al.
Revised: Compared with the traditional semantic segmentation pipeline that uses dense annotations as supervision information (Touvron et al.

Original: ; Ren et al.
Revised: ; Ren et al.

Original: the CLIP-based segmentation methods (Xu et al.
Revised: the CLIP-based segmentation methods (Xu et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2021a)) use text as supervision, which is easier to access but more noisy and ambiguous.
Revised: (2021a)) use text as supervision, which is easier to access but more noisy and ambiguous.

Original: This is mainly because compared with traditional segmentation annotations, text descriptions are often more abstract and do not contain location information.
Revised: This is mainly because compared with traditional segmentation annotations, text descriptions are often more abstract and do not contain location information.

Original: Moreover, the background in the image is usually ignored in the description.
Revised: Moreover, the background in the image is usually ignored in the description.

Original: In some cases, the objects in the image do not even exist in the text description (see Figure 1).
Revised: In some cases, the objects in the image do not even exist in the text description (see Figure 1).

Original: Such ambiguity is common in the textual supervision in vision-language pre-training.
Revised: Such ambiguity is common in the textual supervision in visionlanguage pre-training.

Original: In the semantic segmentation task, the ambiguity of textual supervision makes the segmented object-label correspondence very fragile.
Revised: In the semantic segmentation task, the ambiguity of textual supervision makes the segmented object-label correspondence very fragile.

Original: Therefore, Fully mining the information carried by the dataset itself may need to be considered.
Revised: Therefore, Fully mining the information carried by the dataset itself may need to be considered.

Original: On the other hand, visual self-supervision (Caron et al.
Revised: On the other hand, visual self-supervision (Caron et al.

Original:  He et al.
Revised: ; He et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020a)
Revised: (2020a);

Original: ; Zhou et al.
Revised: Zhou et al.

Original: (2021b)) has been widely used for visual pre-training.
Revised: (2021b)) has been widely used for visual pre-training.

Original: It includes two categories: reconstructing masked images (He et al.
Revised: It includes two categories: reconstructing masked images (He et al.

Original:  Zhou et al.
Revised: ; Zhou et al.

Original: (2021b))
Revised: (2021b))

Original: and multicrop image contrast (Caron et al.
Revised: and multicrop image contrast (Caron et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020a)).
Revised: (2020a)).

Original: For example, SLIP (Mu et al.
Revised: For example, SLIP (Mu et al.

Original: (2021)) introduces contrastive learning of multicrop visual consistency for VLP.
Revised: (2021)) introduces contrastive learning of multicrop visual consistency for VLP.

Original: MaskCLIP (Dong et al.
Revised: MaskCLIP (Dong et al.

Original: (2022)) introduces a visual self-supervised task of reconstructing masked images.
Revised: (2022)) introduces a visual self-supervised task of reconstructing masked images.

Original: They utilize visual self-supervision to provide more useful information for VLP models.
Revised: They utilize visual self-supervision to provide more useful information for VLP models.

Original: However, the semantic consistency of multiple views of an image in segmentation and cross-modal contrast have not received enough attention and research.
Revised: However, the semantic consistency of multiple views of an image in segmentation and cross-modal contrast have not received enough attention and research.

Original: Based on the above observations, in this paper, we explore the impact of multi-view semantic consistency on the task of text-supervised semantic segmentation through visual self-supervision.
Revised: Based on the above observations, in this paper, we explore the impact of multi-view semantic consistency on the task of text-supervised semantic segmentation through visual self-supervision.

Original: To this end, we propose multi- View Co nsistency learning (ViewCo), which aims at discovering textsupervised segmentation masks via multi-view semantic consistency.
Revised: To this end, we propose multi- View Co nsistency learning (ViewCo), which aims at discovering textsupervised segmentation masks via multi-view semantic consistency.

Original: Specifically, we propose textto-views consistency modeling to alleviate the excessive strictness of image-text correspondence in vanilla vision-language contrastive learning.
Revised: Specifically, we propose textto-views consistency modeling to alleviate the excessive strictness of image-text correspondence in vanilla vision-language contrastive learning.

Original: It enables the model to benefit from the dense assignment of visual features by encouraging different crops to align with the same text.
Revised: It enables the model to benefit from the dense assignment of visual features by encouraging different crops to align with the same text.

Original: This relaxed one-to-many contrast mechanism also facilitates the learning of multi-view consistent semantics, enabling the model to acquire high-level cross-modal alignment capabilities.
Revised: This relaxed one-to-many contrast mechanism also facilitates the learning of multi-view consistent semantics, enabling the model to acquire high-level cross-modal alignment capabilities.

Original: Moreover, as shown in
Revised: Moreover, as shown in

Original: Figure 1, to alleviate the ambiguity issue of textual supervision, we propose cross-view segmentation consistency modeling .
Revised: Figure 1, to alleviate the ambiguity issue of textual supervision, we propose cross-view segmentation consistency modeling .

Original: It overcomes the limitation imposed by textual ambiguity by providing additional self-supervision to vision-language contrastive learning via cross-view segmentation consistency.
Revised: It overcomes the limitation imposed by textual ambiguity by providing additional self-supervision to vision-language contrastive learning via cross-view segmentation consistency.

Original: ViewCo uses the proposed text-to-views consistency modeling for vision-language crossmodal contrastive learning and additionally enables cross-view segmentation consistency modeling by contrasting the segment features of Siamese visual encoders.
Revised: ViewCo uses the proposed text-to-views consistency modeling for vision-language crossmodal contrastive learning and additionally enables cross-view segmentation consistency modeling by contrasting the segment features of Siamese visual encoders.

Original: As shown in Figure 2, with the help of the two consistency modeling schemes, ViewCo establishes a solid semantic correspondence in different views, and the semantics in different views maintain a good consistency.
Revised: As shown in Figure 2, with the help of the two consistency modeling schemes, ViewCo establishes a solid semantic correspondence in different views, and the semantics in different views maintain a good consistency.

Original: The semantic consistency of GroupViT in different views is difficult to guarantee.
Revised: The semantic consistency of GroupViT in different views is difficult to guarantee.

Original: Overall, ViewCo’s design is simple and effective.
Revised: Overall, ViewCo’s design is simple and effective.

Original: We train it on large-scale image-text pair datasets
Revised: We train it on large-scale image-text pair datasets

Original: CC12M (Changpinyo et al.
Revised: CC12M (Changpinyo et al.

Original: and YFCC (Thomee et al.
Revised: and YFCC (Thomee et al.

Original: In the inference stage, we use the similarity scores between the segmentation embeddings generated by the teacher network and  semantic 
Revised: In the inference stage, we use the similarity scores between the segmentation embeddings generated by the teacher network and the label prompts to assign labels to the image masks for zero-shot semantic segmentation.

Original: Compared with the state-of-the-art methods, ViewCo achieves an average improvement of 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.
Revised: Compared with the state-of-the-art methods, ViewCo achieves an average improvement of 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose a novel one-to-many text-to-views consistency modeling that improves the model’s ability of high-level cross-modal semantic alignment by encouraging different crops of an image to align with the same text.
Revised: • We propose a novel one-to-many text-to-views consistency modeling that improves the model’s ability of high-level cross-modal semantic alignment by encouraging different crops of an image to align with the same text.

Original: • To alleviate the problem of supervision failure that may arise from text ambiguity, we propose cross-view segmentation consistency modeling to provide additional self-supervision for the vision branch and encourage the model to generate consistent segmentation masks for different views.
Revised: • To alleviate the problem of supervision failure that may arise from text ambiguity, we propose cross-view segmentation consistency modeling to provide additional self-supervision for the vision branch and encourage the model to generate consistent segmentation masks for different views.

Original: • ViewCo consistently outperforms the state-of-the-art methods on PASCAL VOC2012, PASCAL Context, and MS-COCO when pre-trained on CC12M or CC12M+YFCC.
Revised: • ViewCo consistently outperforms the state-of-the-art methods on PASCAL VOC2012, PASCAL Context, and MS-COCO when pre-trained on CC12M or CC12M+YFCC.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Vision-Language Pretraining.
Revised: Vision-Language Pretraining.

Original: In recent years, vision-language pre-training models (Chen et al.
Revised: In recent years, vision-language pre-training models (Chen et al.

Original: (2020b)
Revised: (2020b)

Original: ; Desai & Johnson (2021);
Revised: ; Desai & Johnson (2021);

Original: Li et al.
Revised: Li et al.

Original: (2020a; 2021a; 2020b)) have developed rapidly with the help of large-scale image-text pair data available on the Internet.
Revised: (2020a; 2021a; 2020b)) have developed rapidly with the help of large-scale image-text pair data available on the Internet.

Original: Recently, VLP models such as
Revised: Recently, VLP models such as

Original: CLIP (Radford et al.
Revised: CLIP (Radford et al.

Original: (2021)), ALIGN (Li et al.
Revised: (2021)), ALIGN (Li et al.

Original: (2021a)), and SLIP (Mu et al.
Revised: (2021a)), and SLIP (Mu et al.

Original: (2021)) have made great progress in visual representation learning by using contrastive learning.
Revised: (2021)) have made great progress in visual representation learning by using contrastive learning.

Original: And they have been successfully transferred to various downstream tasks, such as visual question answering (Antol et al.
Revised: And they have been successfully transferred to various downstream tasks, such as visual question answering (Antol et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: and visual reasoning (Zellers et al.
Revised: and visual reasoning (Zellers et al.

Original: In particular, CLIP (Radford et al.
Revised: In particular, CLIP (Radford et al.

Original: (2021)) uses the image-text matching relationship for contrastive learning, and the learned model can be directly transferred to ImageNet classification (Deng et al.
Revised: (2021)) uses the image-text matching relationship for contrastive learning, and the learned model can be directly transferred to ImageNet classification (Deng et al.

Original: in a zero-shot manner without any fine-tuning.
Revised: in a zero-shot manner without any fine-tuning.

Original: This success is also found on zero-shot semantic segmentation (Xu et al.
Revised: This success is also found on zero-shot semantic segmentation (Xu et al.

Original: However, the one-to-one contrastive learning mechanism between image and text in the vanilla VLP pipeline is too strict, which is not conducive to the model learning high-level crossmodal semantic alignment.
Revised: However, the one-to-one contrastive learning mechanism between image and text in the vanilla VLP pipeline is too strict, which is not conducive to the model learning high-level crossmodal semantic alignment.

Original: Based on the above observations, this paper proposes one-to-many textto-views consistency modeling.
Revised: Based on the above observations, this paper proposes one-to-many textto-views consistency modeling.

Original: It relaxes the original one-to-one correspondence by encouraging different crops of an image to match the same text, allowing the model to benefit from the dense assignment of the visual features.
Revised: It relaxes the original one-to-one correspondence by encouraging different crops of an image to match the same text, allowing the model to benefit from the dense assignment of the visual features.

Original: Visual Self-Supervision.
Revised: Visual Self-Supervision.

Original: This framework relies on the information carried by the image itself for self-supervision without any additional annotation information.
Revised: This framework relies on the information carried by the image itself for self-supervision without any additional annotation information.

Original: Visual self-supervision is mainly divided into generative (He et al.
Revised: Visual self-supervision is mainly divided into generative (He et al.

Original: ; Bao et al.
Revised: ; Bao et al.

Original: and contrastive (He et al.
Revised: and contrastive (He et al.

Original: (2020a)
Revised: (2020a)

Original: ; Caron et al.
Revised: ; Caron et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020a)).
Revised: (2020a)).

Original: A generative model allows the model to learn the feature representation of the image by reconstructing the masked image.
Revised: A generative model allows the model to learn the feature representation of the image by reconstructing the masked image.

Original: Contrastive models focus more on learning-centric global representations.
Revised: Contrastive models focus more on learning-centric global representations.

Original: Since semantic segmentation requires dense prediction of images, generative models may not help much because they destroy the original structure and information of images.
Revised: Since semantic segmentation requires dense prediction of images, generative models may not help much because [Figure] view text pull together positive pairs push away negative pairs distance between different objects [Figure]

Original:  On the other hand, the contrastive visual selfsupervised model can provide the required multi-view features for ViewCo’s text-to-views consistency modeling.
Revised: [Equation] [Figure] they destroy the original structure and information of images. On the other hand, the contrastive visual selfsupervised model can provide the required multi-view features for ViewCo’s text-to-views consistency modeling.

Original: Moreover, this visual contrastive learning can provide additional visual self-supervision information for the
Revised: Moreover, this visual contrastive learning can provide additional visual self-supervision information for the

Original: VLP model to alleviate the risk of supervision failure caused by text ambiguity.
Revised: VLP model to alleviate the risk of supervision failure caused by text ambiguity.

Original: Therefore, this paper focuses on the help of visual contrastive learning for semantic segmentation consistency.
Revised: Therefore, this paper focuses on the help of visual contrastive learning for semantic segmentation consistency.

Original: Consistent Semantics.
Revised: Consistent Semantics.

Original: Capturing consistent semantics is one of the main challenges shared by many tasks such as cross-modality and visual understanding.
Revised: Capturing consistent semantics is one of the main challenges shared by many tasks such as cross-modality and visual understanding.

Original: Visionlanguage contrastive learning (Radford et al.
Revised: Visionlanguage contrastive learning (Radford et al.

Original: (2021)) is essential to encode different modal data into the same feature space and enforces the features sharing the same semantics to get closer, and the features with different semantics to be pushed away.
Revised: (2021)) is essential to encode different modal data into the same feature space and enforces the features sharing the same semantics to get closer, and the features with different semantics to be pushed away.

Original: Similarly, multicrop image-level semantic consistency is also the core idea of visual self-supervised contrastive learning (Caron et al.
Revised: Similarly, multicrop image-level semantic consistency is 𝑓 𝑠 EMA MLP 𝑧 𝑇 𝑧 𝑡 1 𝑧 𝑡𝑓 𝑡 ℒ [𝑆𝑒𝑔] ℒ [𝑇𝑒𝑥𝑡−𝑉𝑖𝑒𝑤𝑠] 𝑢 𝑣

Revised: A womanusing her tablet while it's snowing.

Revised: A photoof a woman.

Revised: A photoof a tablet.

Revised: Extract Nouns & Prompts

Revised: MLP

Revised: [Figure]

Revised: Segmantation

Revised: [Figure]

Revised: 𝐸

Revised: [Figure]

Revised: 𝑍 𝐼 𝑢 𝑍 𝐼 𝑣 also the core idea of visual self-supervised contrastive learning (Caron et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020a)
Revised: (2020a)

Original: ; He et al.
Revised: ; He et al.

Original: (2020a)).
Revised: (2020a)).

Original: For example, DenseCL (Wang et al.
Revised: For example, DenseCL (Wang et al.

Original: (2021a)) performs pixel-level dense contrastive learning on dense output vectors from multiple views, which is not helpful to the learning of high-level global semantic information.
Revised: (2021a)) performs pixel-level dense contrastive learning on dense output vectors from multiple views, which is not helpful to the learning of high-level global semantic information.

Original: Further, GroupViT (Xu et al.
Revised: Further, GroupViT (Xu et al.

Original: (2022)) uses text as supervision and achieves pixel grouping by capturing the contextual consistent semantics of images.
Revised: (2022)) uses text as supervision and achieves pixel grouping by capturing the contextual consistent semantics of images.

Original: However, in the text-supervised semantic segmentation task, the ambiguous properties of text relative to dense annotations result in that the semantic consistency of images sharing the same semantics cannot be sufficiently guaranteed in the embedding space.
Revised: However, in the text-supervised semantic segmentation task, the ambiguous properties of text relative to dense annotations result in that the semantic consistency of images sharing the same semantics cannot be sufficiently guaranteed in the embedding space.

Original: Furthermore, the strict one-toone correspondence between image and text in the vanilla VLP model is also not conducive to the true alignment of high-level cross-modal semantics.
Revised: Furthermore, the strict one-toone correspondence between image and text in the vanilla VLP model is also not conducive to the true alignment of high-level cross-modal semantics.

Original: Figure 3 (left) illustrates the above observation: although one of the views of an image ( e.g. , the solid circle) is already close to the corresponding text embedding, other views ( e.g. , the dashed circles) may still be far away.
Revised: Figure 3 (left) illustrates the above observation: although one of the views of an image ( e.g. , the solid circle) is already close to the corresponding text embedding, other views ( e.g. , the dashed circles) may still be far away.

Original: Previous VLP methods generally only focus on the alignment of a single view with text.
Revised: Previous VLP methods generally only focus on the alignment of a single view with text.

Original: In contrast, as shown in Figure 3 (right), ViewCo focuses on text-to-views consistency modeling, doing one-to-many matching in cross-modal contrastive learning.
Revised: In contrast, as shown in Figure 3 (right), ViewCo focuses on text-to-views consistency modeling, doing one-to-many matching in cross-modal contrastive learning.

Original:  M ULTI -V IEW C ONSISTENT L EARNING
Revised:  M ULTI -V IEW C ONSISTENT L EARNING

Original: As shown in Figure 4, our ViewCo is mainly composed of a cross-view segmentation consistency module and a text-to-views consistency module.
Revised: As shown in Figure 4, our ViewCo is mainly composed of a cross-view segmentation consistency module and a text-to-views consistency module.

Original: We describe these two modules in Sections 3.1 and 3.2, respectively, and summarize the final loss function in Section 3.3.
Revised: We describe these two modules in Sections 3.1 and 3.2, respectively, and summarize the final loss function in Section 3.3.

Original: 3. CROSS - VIEW SEGMENTATION CONSISTENCY MODULE
Revised: 3. CROSS - VIEW SEGMENTATION CONSISTENCY MODULE

Original: As shown in Figure 4 (left), given a batch of image-text pairs { ( x Ii , x Ti ) } 
Revised: As shown in Figure 4 (left), given a batch of image-text pairs { ( x I i , x T i ) } B i

Original: Bi =1 , two random augmentations are performed on the input image x Ii , generating two warped views u and v .
Revised: =1 , two random augmentations are performed on the input image x Ii , generating two warped views u and v .

Original: We use
Revised: We use

Original: GroupViT (Xu et al.
Revised: GroupViT (Xu et al.

Original: as the bottom-up segmentation backbone of ViewCo, where each view is segmented into K segment tokens.
Revised: as the bottom-up segmentation backbone of ViewCo, where each view is segmented into K segment tokens.

Original: For each of the views (e.g., u ), this process is expressed as:
Revised: For each of the views (e.g., u ), this process is expressed as:

Original: Z u s Seg =
Revised: Z u s Seg =

Original: { Z u s seg k , k = 1 , ..., K } = f s ( u ) ∈
Revised: { Z u s seg k , k = 1 , ..., K } = f s ( u ) ∈

Original: R K × d , where Z u s seg k ∈ R
Revised: R K × d , where Z u s seg k ∈ R

Original: d is the k -th segment feature from f s , and d is the dimensionality of the segment feature.
Revised: d is the k -th segment feature from f s , and d is the dimensionality of the segment feature.

Original: Similarly, we have Z v s seg k and the segment features Z u t seg k and Z v t seg k from the teacher network f t .
Revised: Similarly, we have Z v s seg k and the segment features Z u t seg k and Z v t seg k from the teacher network f t .

Original: We update the parameters of f t using the exponential moving average (EMA)
Revised: We update the parameters of f t using the exponential moving average (EMA)

Original: He et al.
Revised: He et al.

Original: (2020b) of the parameters of f s .
Revised: (2020b) of the parameters of f s .

Original: For example, let θ
Revised: For example, let θ

Original: i
Revised: i

Original: and θ i be the parameters of f s and f t at training step i , respectively, and then θ i is updated as: θ i
Revised: and θ i be the parameters of f s and f t at training step i , respectively, and then θ i is updated as: θ i

Original: =
Revised: =

Original: αθ
Revised: αθ

Original: i − 1 + (1 − α )
Revised: i − 1 + (1 − α )

Original: θ i , where α is a hyper-parameter for smoothing the update.
Revised: θ i , where α is a hyper-parameter for smoothing the update.

Original: In addition, the standard contrastive loss function, called InfoNCE (Oord et al.
Revised: In addition, the standard contrastive loss function, called InfoNCE (Oord et al.

Original: (2018)), is considered in this paper, for an encoded query q and a set of encoded samples k = { k 0 , k 1 , k 2 , ... } N that are the keys of a[Figure] dictionary, we have:
Revised: (2018)), is considered in this paper, for an encoded query q and a set of encoded samples k = { k 0 , k 1 , k 2 , ... } N that are the keys of a dictionary, we have:

Original: [Equation]
Revised: [Equation]

Original: where τ is a learnable temperature parameter.
Revised: where τ is a learnable temperature parameter.

Original: And q and k + are positive pairs, and the other ( N − 1) pairs are negative.
Revised: And q and k + are positive pairs, and the other ( N − 1) pairs are negative.

Revised: 𝑎 ℒ [𝑆𝑒𝑔] 𝑏 ℒ [𝐼𝑚𝑔−𝑇𝑒𝑥𝑡] ⋮ text 𝑍 1𝑇 𝑍

Revised: 2 𝑇 𝑍 𝑛𝑇 ⋮ 𝑍 1𝐼 𝑢 𝑍 2𝐼

Original: Intuitively, the segment features obtained from different crops of the same image should be roughly the same, i.e. , cross-view segmentation consistency.
Revised: 𝑢 𝑍 𝑛𝐼 𝑢 ⋮ 𝑍 1 𝐼 𝑣 𝑍 2𝐼 𝑣 𝑍 𝑛𝐼 𝑣 ⋮ 𝑣 𝑢 𝑍 Seg 1 𝐼 1 𝑍 Seg 2 𝐼𝑍 Seg 𝐾 𝐼𝑍 Seg 1 𝐼𝑍 Seg 2 𝐼𝑍 Seg 𝐾 𝐼student 𝑣/𝑢 ⋮ ⋮ ⋮ 𝑍 Seg 1 𝐼 𝑛 𝑍 Seg 2 𝐼 𝑛 𝑍 Seg 𝐾 𝐼 𝑛 𝑍 Seg 1 𝐼 𝑛 𝑍 Seg 2 𝑛 𝑍 Seg 𝐾 𝐼 𝑛 ⋮ [Figure] pairs negative pairs 𝑢/𝑣 teacher views (a) L [ Seg ] 𝑎 ℒ

Revised: [𝑆𝑒𝑔] 𝑏 ℒ [𝐼𝑚𝑔−𝑇𝑒𝑥𝑡] ⋮ text 𝑍 1𝑇 𝑍

Revised: 2 𝑇 𝑍 𝑛𝑇 ⋮ 𝑍 1𝐼 𝑢 𝑍 2𝐼

Revised: 𝑢 𝑍 𝑛𝐼 𝑢 ⋮ 𝑍 1 𝐼 𝑣 𝑍 2𝐼 𝑣 𝑍 𝑛𝐼 𝑣

Revised: ⋮ 𝑣 𝑢 𝑍 Seg 1 𝐼 1 𝑍 Seg 2 𝐼𝑍 Seg 𝐾 𝐼𝑍 Seg 1 𝐼𝑍 Seg 2 𝐼𝑍 Seg 𝐾 𝐼student 𝑣/𝑢 ⋮ ⋮ ⋮ 𝑍 Seg 1 𝐼 𝑛 𝑍 Seg 2 𝐼 𝑛 𝑍 Seg 𝐾 𝐼 𝑛 𝑍 Seg 1 𝐼 𝑛 𝑍 Seg 2 𝑛 𝑍 Seg 𝐾 𝐼 𝑛 ⋮

Revised: [Figure]

Revised: Intuitively, the segment features obtained from different crops of the same image should be roughly the same, i.e. , cross-view segmentation consistency.

Original: To this end, for the semantic segmentation task, we replace the image-level contrastive learning in previous methods (Caron et al.
Revised: To this end, for the semantic segmentation task, we replace the image-level contrastive learning in previous methods (Caron et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2021b)) with cross-view segmentation consistency learning within images.
Revised: (2021b)) with cross-view segmentation consistency learning within images.

Original: Therefore, we define the minimization training objective of the cross-view segmentation consistency module in ViewCo as:
Revised: Therefore, we define the minimization training objective of the cross-view segmentation consistency module in ViewCo as:

Original: [Equation]
Revised: [Equation]

Original: It is a bi-directional contrast loss between the segment features from the teacher f t and the student f s .
Revised: It is a bi-directional contrast loss between the segment features from the teacher f t and the student f s .

Original: L t → s [ Seg ] considers two pairs of views ( i.e. , ( u t , v s ) and ( v t , u s ) ) outputted by f t and f s .
Revised: L t → s [ Seg ] considers two pairs of views ( i.e. , ( u t , v s ) and ( v t , u s ) ) outputted by f t and f s .

Original: The segment features of ( u t , v s ) from the same image are multiplied ( Z u t seg · Z v s seg T ∈ R K × K ) after l 2 normalization.
Revised: The segment features of ( u t , v s ) from the same image are multiplied ( Z u t seg · Z v s seg T ∈ R K × K ) after l 2 normalization.

Original: In the image branch of ViewCo, we use the EMA policy for parameter updates, so the learnable grouping tokens on the corresponding position IDs of different views of the same image are highly correlated, and they have the same semantics.
Revised: In the image branch of ViewCo, we use the EMA policy for parameter updates, so the learnable grouping tokens on the corresponding position IDs of different views of the same image are highly correlated, and they have the same semantics.

Original: Therefore, the semantic pairs { ( Z u t seg i , Z v s seg i ) , i = 1 , ..., K } on the diagonal are regarded as positive, and the other K ( K − 1) pairs { ( Z u t seg
Revised: Therefore, the semantic pairs { ( Z u t seg i , Z v s seg i ) , i = 1 , ..., K } on the diagonal are regarded as positive, and the other K ( K − 1) pairs { ( Z u t seg

Original: i , Z v s seg j ) , i, j = 1 , ..., K, i ̸
Revised: i , Z v s seg j ) , i, j = 1 , ..., K, i ̸

Original: = j } are regarded as negative.
Revised: = j } are regarded as negative.

Original: Therefore, the contrastive loss L t → s [ Seg ] of the teacher-tostudent segment features is defined as L t → s [ Seg ] =
Revised: Therefore, the contrastive loss L t → s [ Seg ] of the teacher-tostudent segment features is defined as L t → s [ Seg ] =

Original: L u t → v s + L v t → u s , more specifically:
Revised: L u t → v s + L v t → u s , more specifically:

Original: [Equation]
Revised: [Equation]

Original: Similarly, the contrastive loss L s → t [ Seg ] of the student-to-teacher segment features is defined as L s → t [ Seg ] =
Revised: Similarly, the contrastive loss L s → t [ Seg ] of the student-to-teacher segment features is defined as L s → t [ Seg ] =

Original: L u s → v t + L v s → u t , more specifically:
Revised: L u s → v t + L v s → u t , more specifically:

Original: [Equation]
Revised: [Equation]

Original: Figure 5a shows the positive and negative pairs for cross-view segmentation consistency learning in the vision branch.
Revised: Figure 5a shows the positive and negative pairs for cross-view segmentation consistency learning in the vision branch.

Original: 3. TEXT - TO - VIEWS CONSISTENCY MODULE
Revised: 3. TEXT - TO - VIEWS CONSISTENCY MODULE

Original: Previous methods (Radford et al. (2021); Xu et al.
Revised: Previous methods (Radford et al. (2021); Xu et al.

Original: build visual-linguistic semantic correspondences by performing a contrastive loss on image-text pairs.
Revised: build visual-linguistic semantic correspondences by performing a contrastive loss on image-text pairs.

Original: In this paper, we consider the contrastive learning between multiple views and text, using one-to-many text-to-views consistency modeling instead of one-to-one text-to-image contrastive learning.
Revised: In this paper, we consider the contrastive learning between multiple views and text, using one-to-many text-to-views consistency modeling instead of one-to-one text-to-image contrastive learning.

Original: The model learns to capture 
Revised: The model learns to capture intra-modal and inter-modal semantic consistency through the alignment of multi-view images and text.

Original: Specifically, for a given image-text pair ( x Ii , x Ti ) , by performing two different augmentations to the input image, we have a triplet ( u i , v i , x Ti ) .
Revised: Specifically, for a given image-text pair ( x Ii , x Ti ) , by performing two different augmentations to the input image, we have a triplet ( u i , v i , x Ti ) .

Original: As shown in Figure 4 (right), in the training phase, we take the output ( Z ui , Z vi ) of the view pair ( u i , v i ) through the student network f s and the output Z Ti of the text encoder E T to calculate the contrastive loss respectively.
Revised: As shown in Figure 4 (right), in the training phase, we take the output ( Z ui , Z vi ) of the view pair ( u i , v i ) through the student network f s and the output Z Ti of the text encoder E T to calculate the contrastive loss respectively.

Original: The visual embeddings ( Z ui , Z vi ) and text embedding Z Ti are mapped to the same feature space through two
Revised: The visual embeddings ( Z ui , Z vi ) and text embedding Z Ti are mapped to the same feature space through two

Original: MLPs, respectively, before performing the final l 2 regularization.
Revised: MLPs, respectively, before performing the final l 2 regularization.

Original: This procedure is represented as:
Revised: This procedure is represented as:

Original: [Equation]

Original: I

Original: [Equation]
Revised: [Equation]

Original: The multi-view feature Z Ii =
Revised: The multi-view feature Z Ii =

Original: { Z I u i ,
Revised: { Z I u i ,

Original: Z I v i } and text embedding Z Ti constitute positive pairs, and the other 2 B ( B − 1) pairs are negative pairs.
Revised: Z I v i } and text embedding Z Ti constitute positive pairs, and the other 2 B ( B − 1) pairs are negative pairs.

Original: The contrastive loss of text-to-views consistency modeling is defined as follows:
Revised: The contrastive loss of text-to-views consistency modeling is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where the contrastive loss of views I { u,v } -to-text is defined as:
Revised: where the contrastive loss of views I { u,v } -to-text is defined as:

Original: [Equation]
Revised: [Equation]

Original: and the contrastive loss of text-to-views I { u,v } is defined as:
Revised: and the contrastive loss of text-to-views I { u,v } is defined as:

Original: [Equation]
Revised: [Equation]

Original: Additionally, in order to further enhance the association between multi-view semantics and text semantics, we also compute the multi-label image-text contrastive loss (Xu et al.
Revised: Additionally, in order to further enhance the association between multi-view semantics and text semantics, we also compute the multi-label image-text contrastive loss (Xu et al.

Original: of multiview and “prompted text” pairs { ( Z I u i , { Z t
Revised: of multiview and “prompted text” pairs { ( Z I u i , { Z t

Original: m i } Mm =1 )
Revised: m i } Mm =1 )

Original: Bi
Revised: Bi

Original: =1 , ( Z I v i , { Z t m i } Mm =1 )
Revised: =1 , ( Z I v i , { Z t m i } Mm =1 )

Original: Bi =1 } , where { Z t m i } Mm
Revised: Bi =1 } , where { Z t m i } Mm

Original: = are the embeddings of the additional M text prompts { T mi } Mm =1 generated by the i -th text x
Revised: = are the embeddings of the additional M text prompts { T mi } Mm =1 generated by the i -th text x

Original: Ti according to the “prompt engineering” mechanism (Radford et al.
Revised: Ti according to the “prompt engineering” mechanism (Radford et al.

Original: ( Z I u i , { Z t
Revised: ( Z I u i , { Z t

Original: m i } Mm =1 ) , i.e. , the embedding of the i -th image view u and the generated M text embeddings {
Revised: m i } Mm =1 ) , i.e. , the embedding of the i -th image view u and the generated M text embeddings {

Original: Z t m i } Mm =1 are positive pairs, and the other combinations are negative pairs.
Revised: Z t m i } Mm =1 are positive pairs, and the other combinations are negative pairs.

Original: Therefore, similar to Eq.(5), the multi-label contrastive loss of multi-view I { u,v } and multi-prompt { T m } Mm =1 is defined as:
Revised: Therefore, similar to Eq.(5), the multi-label contrastive loss of multi-view I { u,v } and multi-prompt { T m } Mm =1 is defined as:

Original: [Equation]
Revised: [Equation]

Original: First, the views-to-prompts loss is the average of the losses of the two views.
Revised: First, the views-to-prompts loss is the average of the losses of the two views.

Original: Considering a single view, e.g. u , the contrastive loss of u to all the prompts is defined as:
Revised: Considering a single view, e.g. u , the contrastive loss of u to all the prompts is defined as:

Original: [Equation]
Revised: [Equation]

Original: Second, the contrastive loss of multi-prompt-to-views is defined as:
Revised: Second, the contrastive loss of multi-prompt-to-views is defined as:

Original: [Equation]
Revised: [Equation]

Original:  In particular, a similar work to our text-to-views consistency module is DeCLIP (Li et al.
Revised: [Equation] In particular, a similar work to our text-to-views consistency module is DeCLIP (Li et al.

Original: (2021b)).
Revised: (2021b)).

Original: It believes that the text description may only be a small part of the image, so in addition to the global view in CLIP (Radford et al.
Revised: It believes that the text description may only be a small part of the image, so in addition to the global view in CLIP (Radford et al.

Original: , DeCLIP also adds a local view for image self-supervision, which may cause information leakage.
Revised: , DeCLIP also adds a local view for image self-supervision, which may cause information leakage.

Original: In addition, DeCLIP uses EDA (Wei & Zou (2019)) as a text augmentation strategy.
Revised: In addition, DeCLIP uses EDA (Wei & Zou (2019)) as a text augmentation strategy.

Original: The augmented text still contains multiple semantics, which is not helpful to the alignment of local semantics in segmentation tasks.
Revised: The augmented text still contains multiple semantics, which is not helpful to the alignment of local semantics in segmentation tasks.

Original: In contrast, ViewCo uses self-supervision of two local views to ensure the difficulty of the task, while using a “prompt engineering” mechanism to obtain an augmented text with a single semantic.
Revised: In contrast, ViewCo uses self-supervision of two local views to ensure the difficulty of the task, while using a “prompt engineering” mechanism to obtain an augmented text with a single semantic.

Original: Combining one-to-many alignment can help
Revised: Combining one-to-many alignment can help

Original: ViewCo to better mine consistent segmentation semantics in images.
Revised: ViewCo to better mine consistent segmentation semantics in images.

Original: 3. O VERALL L OSS F UNCTION
Revised: 3. O VERALL L OSS F UNCTION

Original: Finally, the total loss of ViewCo is the sum of the cross-view segmentation consistency contrastive loss and the two cross-modal contrastive losses:
Revised: Finally, the total loss of ViewCo is the sum of the cross-view segmentation consistency contrastive loss and the two cross-modal contrastive losses:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: Architecture.
Revised: Architecture.

Original: In the cross-view segmentation consistency module, f t and f s have the same network structure.
Revised: In the cross-view segmentation consistency module, f t and f s have the same network structure.

Original: The parameters of f t are updated using the exponential moving average of the parameters  of f s .
Revised: The parameters of f t are updated using the exponential moving average of the parameters [Table] of f s .

Original: We use GroupViT
Revised: We use GroupViT

Original: (Xu et al.
Revised: (Xu et al.

Original: with two stages as the backbone for semantic feature extraction of ViewCo’s visual branch.
Revised: with two stages as the backbone for semantic feature extraction of ViewCo’s visual branch.

Original: It is built on ViT-S (Dosovitskiy et al.
Revised: It is built on ViT-S (Dosovitskiy et al.

Original: (2020); Touvron et al.
Revised: (2020); Touvron et al.

Original: with 12 Transformer layers.
Revised: with 12 Transformer layers.

Original: The input image size is 224 × 224 , the patch size is 16 × 16 , and the hidden dimensionality is 384.
Revised: The input image size is 224 × 224 , the patch size is 16 × 16 , and the hidden dimensionality is 384.

Original: The 2-stage GroupViT finally outputs 8 segment tokens, i.e.,
Revised: The 2-stage GroupViT finally outputs 8 segment tokens, i.e.,

Original: K = 8 ).
Revised: K = 8 ).

Original: Following Radford et al.
Revised: Following Radford et al.

Original: (2021), ViewCo’s text encoder E T consists of 12 Transformer layers with a hidden feature dimensionality of 256.
Revised: (2021), ViewCo’s text encoder E T consists of 12 Transformer layers with a hidden feature dimensionality of 256.

Original: Training and Inference.
Revised: Training and Inference.

Original: In the training phase, we use CC12M (Changpinyo et al.
Revised: In the training phase, we use CC12M (Changpinyo et al.

Original: and the filtered YFCC (Thomee et al.
Revised: and the filtered YFCC (Thomee et al.

Original: as training datasets, which contain 12M and 14M image-text pairs, respectively.
Revised: as training datasets, which contain 12M and 14M image-text pairs, respectively.

Original: See A.1 of the supplementary material for more training details.
Revised: See A.1 of the supplementary material for more training details.

Original: In the inference phase, following (Xu et al.
Revised: In the inference phase, following (Xu et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: , the image is segmented by associating the image patches with the K segment tokens outputted by the teacher network f t .
Revised: , the image is segmented by associating the image patches with the K segment tokens outputted by the teacher network f t .

Original: The semantics in the images are further classified by computing the similarity of the K visual-semantic embeddings to the text embeddings generated from the class labels of the test dataset.
Revised: The semantics in the images are further classified by computing the similarity of the K visual-semantic embeddings to the text embeddings generated from the class labels of the test dataset.

Original: Zero-Shot Transfer to Semantic Segmentation.
Revised: Zero-Shot Transfer to Semantic Segmentation.

Original: We evaluate ViewCo on the task of zero-shot transfer to semantic segmentation on the validation sets of PASCAL VOC 2012 (Everingham et al.
Revised: We evaluate ViewCo on the task of zero-shot transfer to semantic segmentation on the validation sets of PASCAL VOC 2012 (Everingham et al.

Original: (2010)), PASCAL Context (Mottaghi et al.
Revised: (2010)), PASCAL Context (Mottaghi et al.

Original: and COCO Stuff (Lin et al.
Revised: and COCO Stuff (Lin et al.

Original: datasets.
Revised: datasets.

Original: The three datasets contain 20, 59, and 80 foreground classes and an additional background class, respectively.
Revised: The three datasets contain 20, 59, and 80 foreground classes and an additional background class, respectively.

Original: During inference, following GroupViT (Xu et al.
Revised: During inference, following GroupViT (Xu et al.

Original: (2022)), ViewCo predicts only the foreground classes by thresholding the softmax-normalized-similarity between the embedding of the outputted image segments and the text segmentation labels.
Revised: (2022)), ViewCo predicts only the foreground classes by thresholding the softmax-normalized-similarity between the embedding of the outputted image segments and the text segmentation labels.

Original: The thresholds on PASCAL VOC 2012, PASCAL Context, and COCO are set to 0.95, 0.35, and 0.95, respectively.
Revised: The thresholds on PASCAL VOC 2012, PASCAL Context, and COCO are set to 0.95, 0.35, and 0.95, respectively.

Original: We resize each input image to have a shorter side of 448.
Revised: We resize each input image to have a shorter side of 448.

Original: 4. C OMPARISONS WITH R ECENT M ETHODS
Revised: 4. C OMPARISONS WITH R ECENT M ETHODS

Original: We first compare the performance of ViewCo with some ViT-S-based zero-shot baselines.
Revised: We first compare the performance of ViewCo with some ViT-S-based zero-shot baselines.

Original: Then, to further evaluate the performance of ViewCo on the zero-shot semantic segmentation task, we compare ViewCo with some fully supervised transfer and CLIP-based models.
Revised: Then, to further evaluate the performance of ViewCo on the zero-shot semantic segmentation task, we compare ViewCo with some fully supervised transfer and CLIP-based models.

Original: [Table]
Revised: [Table]

Original: Comparison with Zero-Shot Baselines.
Revised: Comparison with Zero-Shot Baselines.

Original: Table 1 shows the performance comparison of ViewCo and zero-shot baselines on PASCAL VOC 2012.
Revised: Table 1 shows the performance comparison of ViewCo and zero-shot baselines on PASCAL VOC 2012.

Original: Among them, the four ViT-based baselines train vision and text encoders through the image-text contrastive loss defined in CLIP (Radford et al.
Revised: Among them, the four ViT-based baselines train vision and text encoders through the image-text contrastive loss defined in CLIP (Radford et al.

Original: They adopt four different pixel grouping methods: pixel-wise, Kmeans, Mean-shift (Comaniciu & Meer (2002)), and Spectral clustering (Shi & Malik (1997)) respectively.
Revised: They adopt four different pixel grouping methods: pixel-wise, K-means, Mean-shift (Comaniciu & Meer (2002)), and Spectral clustering (Shi & Malik (1997)) respectively.

Original: And GroupViT (Xu et al.
Revised: And GroupViT (Xu et al.

Original: (2022)) uses the bottom-up patch grouping mechanism.
Revised: (2022)) uses the bottom-up patch grouping mechanism.

Original: As shown in Table 1, ViewCo significantly outperforms the CLIP-trained ViT and GroupViT (52.4% vs. 51.2%) baselines.
Revised: As shown in Table 1, ViewCo significantly outperforms the CLIP-trained ViT and GroupViT (52.4% vs. 51.2%) baselines.

Original: It is worth noting that ViewCo and GroupViT adopt the same segmentation backbone, indicating that ViewCo can effectively improve the model’s ability of segmentation and cross-modal semantic alignment with the help of the two consistent semantic modelings.
Revised: It is worth noting that ViewCo and GroupViT adopt the same segmentation backbone, indicating that ViewCo can effectively improve the model’s ability of segmentation and cross-modal semantic alignment with the help of the two consistent semantic modelings.

Original: Comparison with Other SoTA Methods.
Revised: Comparison with Other SoTA Methods.

Original: These methods include one fully supervised baseline
Revised: These methods include one fully supervised baseline

Original: DeiT (Touvron et al.
Revised: DeiT (Touvron et al.

Original: (2021)), two visual self-supervised baselines DINO (Caron et al.
Revised: (2021)), two visual self-supervised baselines DINO (Caron et al.

Original: and
Revised: and

Original: MoCo (He et al. (2020a)), two vision-language contrastive learning baselines CLIP (Radford et al.
Revised: MoCo (He et al. (2020a)), two vision-language contrastive learning baselines CLIP (Radford et al.

Original: and GroupViT (Xu et al.
Revised: and GroupViT (Xu et al.

Original: , two vision-language contrast and visual self-supervised learning combined baselines SLIP (Mu et al.
Revised: , two vision-language contrast and visual self-supervised learning combined baselines SLIP (Mu et al.

Original: and CLIP-MAE (Dong et al.
Revised: and CLIP-MAE (Dong et al.

Original: , and one vision-language contrast and self-distillation combined baseline MaskCLIP (Dong et al.
Revised: , and one vision-language contrast and self-distillation combined baseline MaskCLIP (Dong et al.

Original: Table 2 shows the mIoU performance comparison between ViewCo and the SoTA methods on PASCAL VOC 2012, PASCAL Context, and COCO validation sets.
Revised: Table 2 shows the mIoU performance comparison between ViewCo and the SoTA methods on PASCAL VOC 2012, PASCAL Context, and COCO validation sets.

Original: ViewCo significantly outperforms them on all three datasets.
Revised: ViewCo significantly outperforms them on all three datasets.

Original: Compared to GroupViT, when pre-trained on CC12M, ViewCo achieves[Table] a 4.6% mIoU improvement on PASCAL VOC.
Revised: Compared to GroupViT, when pre-trained on CC12M, ViewCo achieves a 4.6% mIoU improvement on PASCAL VOC.

Original: Similarly, when pre-trained on CC12M+YFCC,
Revised: Similarly, when pre-trained on CC12M+YFCC,

Original: ViewCo achieves a 2.6% mIoU improvement on COCO compared to GroupViT. Similar to ViewCo,
Revised: ViewCo achieves a 2.6% mIoU improvement on COCO compared to GroupViT. Similar to ViewCo,

Original: SLIP, MaskCLIP, and CLIP-MAE all use additional supervision information in the vision branch of the VLP models.
Revised: SLIP, MaskCLIP, and CLIP-MAE all use additional supervision information in the vision branch of the VLP models.

Original: Compared with them, ViewCo still has clear advantages in PASCAL Context and
Revised: Compared with them, ViewCo still has clear advantages in PASCAL Context and

Original: COCO.
Revised: COCO.

Original: In addition, ViewCo obtains segmentation performance close to the fully supervised DeiT on PASCAL VOC, which again demonstrates the effectiveness of ViewCo for zero-shot semantic segmentation.
Revised: In addition, ViewCo obtains segmentation performance close to the fully supervised DeiT on PASCAL VOC, which again demonstrates the effectiveness of ViewCo for zero-shot semantic segmentation.

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: In this section, for the convenience of comparison, we use CC12M as the pre-training dataset by default for the ablation of ViewCo components, qualitative analysis, and image classification performance comparison.
Revised: In this section, for the convenience of comparison, we use CC12M as the pre-training dataset by default for the ablation of ViewCo components, qualitative analysis, and image classification performance comparison.

Original: GroupViT (Xu et al.
Revised: GroupViT (Xu et al.

Original: (2022)) is used as the baseline for ViewCo.
Revised: (2022)) is used as the baseline for ViewCo.

Original: [Table]
Revised: [Table]

Original: visual module

Original: GroupViT+ represents modifying the corresponding component in GroupViT.

Original: Image-Level Contrast vs. Semantic-Level Contrast.
Revised: Image-Level Contrast vs. Semantic-Level Contrast.

Original: To ablate the role of the cross-view segmentation consistency module in the vision branch, we add an imagelevel contrastive module to GroupViT in the visual branch, where we first calculate the average of the K segment tokens outputted by the teacher and student networks, and then perform contrastive learning.
Revised: To ablate the role of the cross-view segmentation consistency module in the vision branch, we add an image-level contrastive module to GroupViT in the visual branch, where we first calculate the average of the K segment tokens outputted by the teacher and student networks, and then perform contrastive learning.

Original: For ViewCo, we remove the text-to-views consistency module and directly average pool the multi-view features outputted by the student network.
Revised: For ViewCo, we remove the text-to-views consistency module and directly average pool the multi-view features outputted by the student network.

Original: To be consistent with GroupViT, we use the pooled visual features for contrastive learning with text embeddings.
Revised: To be consistent with GroupViT, we use the pooled visual features for contrastive learning with text embeddings.

Original: As shown in Table 3, adding a visual self-supervised module for vision-language contrastive learning can improve the performance of the model on semantic segmentation by improving the quality of visual feature learning.
Revised: As shown in Table 3, adding a visual self-supervised module for vision-language contrastive learning can improve the performance of the model on semantic segmentation by improving the quality of visual feature learning.

Original: Furthermore, the improved performance ( i.e. , 19.1 vs. 18.6) of semanticlevel learning relative to image-level contrastive learning suggests that the cross-view segmentation consistency module can further improve the performance by capturing the consistency of cross-view semantic segmentation. 
Revised: Furthermore, the improved performance ( i.e. , 19.1 vs. 18.6) of semantic-level learning relative to image-level contrastive learning suggests that the cross-view segmentation consistency module can further improve the performance by capturing the consistency of cross-view semantic segmentation. [Table]

Original: Vision-Language Contrast: Text-to-Image vs. Text-to-Views.
Revised: Vision-Language Contrast: Text-to-Image vs. Text-to-Views.

Original: We further ablate the text-toviews consistency module in ViewCo.
Revised: We further ablate the text-to-views consistency module in ViewCo.

Original: In single-view vision-language contrastive learning, we use the average embedding of multi-view features outputted by the student network and the text embedding for contrastive learning during training.
Revised: In single-view vision-language contrastive learning, we use the average embedding of multi-view features outputted by the student network and the text embedding for contrastive learning during training.

Original: As shown in Table 4, text-to-views consistency modeling significantly improves the performances of the models compared to singleview text-to-image ( i.e. , 1.1% and 1.5%).
Revised: As shown in Table 4, text-to-views consistency modeling significantly improves the performances of the models compared to single-view text-to-image ( i.e. , 1.1% and 1.5%).

Original: This indicates that text-to-views consistency modeling has better high-level semantic alignment capabilities than text-to-image single-view modeling.[Figure] [Table] [Figure]
Revised: This indicates that text-toviews consistency modeling has better high-level semantic alignment capabilities than text-to-image single-view modeling.

Original: [Table]

Original: [Figure]

Original: denote the number of image views used in vision-language contrastive learning

Original: This is exactly what previous methods of single-view vision-language contrastive learning do not have.
Revised: This is exactly what previous methods of single-view vision-language contrastive learning do not have.

Original: Qualitative Analysis.
Revised: Qualitative Analysis.

Original: Figure 2 shows some visualization results of multi-view semantic segmentation consistency for ViewCo and GroupViT.
Revised: Figure 2 shows some visualization results of multi-view semantic segmentation consistency for ViewCo and GroupViT.

Original: As shown in Figure 2(a), in GroupViT, the semantic segmentations of different views from the same image are inconsistent. 
Revised: As shown in Figure 2(a), in GroupViT, the semantic segmentations of different views from the same image are inconsistent. For exam- [Figure]

Revised: V i T

Revised: V i e w C o

Original: For example, in image x , “umbrella” is misclassified as “cow”, and in view u , “umbrella” is misclassified as “horse”.
Revised: [Figure] r [Figure] ple, in image x , “umbrella” is misclassified as “cow”, and in view u , “umbrella” is misclassified as “horse”.

Original: There is also the problem of inconsistent semantic segmentations between views u and v .
Revised: There is also the problem of inconsistent semantic segmentations between views u and v .

Original: As shown in Figure 2(b), the semantic segmentation in different views in ViewCo is completely consistent.
Revised: As shown in Figure 2(b), the semantic segmentation in different views in ViewCo is completely consistent.

Original: This shows that our cross-view segmentation consistency modeling and text-to-views consistency modeling in ViewCo are effective.
Revised: This shows that our cross-view segmentation consistency modeling and text-to-views consistency modeling in ViewCo are effective.

Original:  To evaluate ViewCo’s ability to perform semantic segmentation through semantic understanding in rare scenes, we show the more visual comparison in Figure 6.
Revised: [Figure] To evaluate ViewCo’s ability to perform semantic segmentation through semantic understanding in rare scenes, we show the more visual comparison in Figure 6.

Original: The images of rare scenes are selected from the Internet.
Revised: The images of rare scenes are selected from the Internet.

Original: In Figure 6(a), we use the class labels of the PASCAL VOC 2012 dataset as the label set for the images.
Revised: In Figure 6(a), we use the class labels of the PASCAL VOC 2012 dataset as the label set for the images.

Original: ViewCo’s segmentation and prediction results in rare scenes are significantly better than GroupViT’s.
Revised: ViewCo’s segmentation and prediction results in rare scenes are significantly better than GroupViT’s.

Original: This indicates that ViewCo can better understand high-level semantics in images through consistent semantic learning.
Revised: This indicates that ViewCo can better understand high-level semantics in images through consistent semantic learning.

Original: In Figure 6(b), we only focus on the model’s ability to segment images in rare scenes.
Revised: In Figure 6(b), we only focus on the model’s ability to segment images in rare scenes.

Original: Compared to GroupViT, ViewCo handles the details of image segmentation much better.
Revised: Compared to GroupViT, ViewCo handles the details of image segmentation much better.

Original: More visual comparison results are shown in Figureof A.2 of the supplementary material.
Revised: More visual comparison results are shown in Figure 7 of A.2 of the supplementary material.

Original: In addition, we also evaluate the segmentation consistency of ViewCoon different views, andthe visual comparison with GroupViT is shown in Figure 8 of A.2.
Revised: In addition, we also visually compare the segmentation consistency of ViewCo and GroupViT on different views in A.3. Finally, we present an analysis of ViewCo’s cross-view segmentation consistency in

Original: [Table]
Revised: A.4.

Original: Image Classification.
Revised: Image Classification.

Original: We also evaluate the classification performance of ViewCo.
Revised: We also evaluate the classification performance of ViewCo.

Original: As shown inTable 5, ViewCo significantly outperforms
Revised: As shown in

Original:  ViT ( i.e. , 46.3% vs.  and GroupViT ( i.e. , 46.3% vs. 42.9%), showing that ViewCo achieves better cross-modal semantic alignment through text-to-views consistency modeling.
Revised: Table 5, ViewCo significantly outperforms ViT ( i.e. , 46.3% vs. 42.4%) and GroupViT ( i.e. , 46.3% vs. 42.9%), showing that ViewCo achieves better cross-modal semantic alignment through text-toviews consistency modeling.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a novel and simple multi-view consistency learning (ViewCo) for text-supervised semantic segmentation.
Revised: We propose a novel and simple multi-view consistency learning (ViewCo) for text-supervised semantic segmentation.

Original: To deal with the problems of excessively strict image-text correspondence and ambiguous text supervision in the VLP model, ViewCo models the text-to-views consistency and cross-view segmentation consistency.
Revised: To deal with the problems of excessively strict image-text correspondence and ambiguous text supervision in the VLP model, ViewCo models the text-to-views consistency and cross-view segmentation consistency.

Original: ViewCo can generate consistent segmentations and better capture high-level cross-modal semantic alignment.
Revised: ViewCo can generate consistent segmentations and better capture high-level cross-modal semantic alignment.

Original: We expect that this exploration of multi-view consistent learning is also applicable to other VLP tasks.  
Revised: We expect that this exploration of multi-view consistent learning is also applicable to other VLP tasks.  A CKNOWLEDGMENT

Revised: This work was supported in part by National Key R&D Program of China under Grant

Revised: No.2020AAA0109700,

Revised: National Natural Science Foundation of China (NSFC) under

Revised: Grant No.61976233, Guangdong Outstanding Youth Fund (Grant No.2021B1515020061),

Revised: Shenzhen

Revised: Fundamental

Revised: Research

Revised: Program (Project

Revised: No.RCYX20200714114642083,

Revised: No.JCYJ20190807154211365).

Revised: We thank MindSpore and CAAI-Huawei MindSpore Open

Revised: Fund for the partial support of this work, which is a new deep learning computing framwork 2 .