Original: CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Revised: CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Referring image segmentation aims at localizing all pixels of the visual objectsdescribed by a natural language sentence.
Revised: Referring image segmentation aims at localizing all pixels of the visual objectsdescribed by a natural language sentence.

Original: Previous works learn to straightforwardlyalign the sentence embedding and pixel-level embedding for highlighting thereferred objects, but ignore the semantic consistency of pixels within the sameobject, leading to incomplete masks and localization errors in predictions.
Revised: Previous works learn to straightforwardlyalign the sentence embedding and pixel-level embedding for highlighting thereferred objects, but ignore the semantic consistency of pixels within the sameobject, leading to incomplete masks and localization errors in predictions.

Original: Totackle this problem, we propose CoupAlign , a simple yet effective multi-levelvisual-semantic alignment method, to couple sentence-mask alignment with wordpixel alignment to enforce object mask constraint for achieving more accuratelocalization and segmentation.
Revised: Totackle this problem, we propose CoupAlign , a simple yet effective multi-levelvisual-semantic alignment method, to couple sentence-mask alignment with wordpixel alignment to enforce object mask constraint for achieving more accuratelocalization and segmentation.

Original: Specifically, the Word-Pixel Alignment (WPA)module performs early fusion of linguistic and pixel-level features in intermediatelayers of the vision and language encoders.
Revised: Specifically, the Word-Pixel Alignment (WPA)module performs early fusion of linguistic and pixel-level features in intermediatelayers of the vision and language encoders.

Original: Based on the word-pixel alignedembedding, a set of mask proposals are generated to hypothesize possible objects.
Revised: Based on the word-pixel alignedembedding, a set of mask proposals are generated to hypothesize possible objects.

Original: Then in the Sentence-Mask Alignment (SMA) module, the masks are weightedby the sentence embedding to localize the referred object, and finally projectedback to aggregate the pixels for the target.
Revised: Then in the Sentence-Mask Alignment (SMA) module, the masks are weightedby the sentence embedding to localize the referred object, and finally projectedback to aggregate the pixels for the target.

Original: To further enhance the learning of thetwo alignment modules, an auxiliary loss is designed to contrast the foregroundand background pixels.
Revised: To further enhance the learning of thetwo alignment modules, an auxiliary loss is designed to contrast the foregroundand background pixels.

Original: By hierarchically aligning pixels and masks with linguisticfeatures, our CoupAlign captures the pixel coherence at both visual and semanticlevels, thus generating more accurate predictions.
Revised: By hierarchically aligning pixels and masks with linguisticfeatures, our CoupAlign captures the pixel coherence at both visual and semanticlevels, thus generating more accurate predictions.

Original: Extensive experiments on populardatasets (e.g., RefCOCO and G-Ref) show that our method achieves consistentimprovements over state-of-the-art methods, e.g., about 2% oIoU increase on thevalidation and testing set of RefCOCO.
Revised: Extensive experiments on populardatasets (e.g., RefCOCO and G-Ref) show that our method achieves consistentimprovements over state-of-the-art methods, e.g., about 2% oIoU increase on thevalidation and testing set of RefCOCO.

Original: Especially, CoupAlign has remarkableability in distinguishing the target from multiple objects of the same class.
Revised: Especially, CoupAlign has remarkableability in distinguishing the target from multiple objects of the same class.

Original:  Introduction
Revised:  Introduction

Original: Taking an image and a natural language sentence as input a referring image segmentation (RIS)model is required to predict a mask for the object described by the sentence.
Revised: Taking an image and a natural language sentence as input a referring image segmentation (RIS)model is required to predict a mask for the object described by the sentence.

Original: Beyond semanticsegmentation which assigns each pixel in an image with a label from a fixed word set, referring imagesegmentation needs to localize the object pixels indicated by the language expression, which is ofarbitrary length and involves an open-world vocabulary, such as object names, attributes, positions,etc.
Revised: Beyond semanticsegmentation which assigns each pixel in an image with a label from a fixed word set, referring imagesegmentation needs to localize the object pixels indicated by the language expression, which is ofarbitrary length and involves an open-world vocabulary, such as object names, attributes, positions,etc.

Original: Therefore, this task is the foundation of many applications in the real world, such as imageediting [24, 33], autonomous driving [3, 37], and language-based human-robot interaction [34, 1].
Revised: Therefore, this task is the foundation of many applications in the real world, such as imageediting [25, 34], autonomous driving [3, 38], and language-based human-robot interaction [35, 1].

Original: The main challenge of RIS is how to align the given language expression with visual pixels forhighlighting the target.
Revised: The main challenge of RIS is how to align the given language expression with visual pixels forhighlighting the target.

Original: Early works [14, 6] typically follow the paradigm that first extracts visual andlanguage features from the corresponding encoder and then fuses them for language-pixel alignment.
Revised: Early works [15, 6] typically follow the paradigm that first extracts visual andlanguage features from the corresponding encoder and then fuses them for language-pixel alignment.

Original: Based on the aligned feature, a decoder is employed to predict a mask.
Revised: Based on the aligned feature, a decoder is employed to predict a mask.

Original: Recently, Transformer-basedvision-language models [18, 22, 20] have shown great potential in aligning visual and language
Revised: Recently, Transformer-basedvision-language models [19, 23, 21] have shown great potential in aligning visual and language

Original: [Figure]
Revised: [Figure]

Original: features from fine to coarse levels, e.g., pixel, concept, region and image from visual view, ande.g. , word, phrase, sentence, paragraph from language perspective.
Revised: features from fine to coarse levels, e.g., pixel, concept, region and image from visual view, and e.g.,word, phrase, sentence, paragraph from language perspective.

Original: Inspired by this, extensiveworks [15, 6, 40, 9, 44] have been investigated to take the advantage of the remarkable cross-modalalignment ability of Transformer architectures for aligning language with pixels more effectively.
Revised: Inspired by this, extensive works[16, 6, 41, 10, 45] have been investigated to take the advantage of the remarkable cross-modalalignment ability of Transformer architectures for aligning language with pixels more effectively.

Original: Some works [6, 40] seek to improve the cross-modal alignment of the encoded vision and languagefeatures, as shown in Fig. 1(a).
Revised: Some works [6, 41] seek to improve the cross-modal alignment of the encoded vision and languagefeatures, as shown in Fig. 1(a).

Original: Their cross-modal alignment mainly occurs at the sentence-pixellevel.
Revised: Their cross-modal alignment mainly occurs at the sentence-pixellevel.

Original: Furthermore, other works [9, 44] fuse the language information into the pixel embeddings atearly stages of the image encoder, as shown in Fig. 1(b).
Revised: Furthermore, other works [10, 45] fuse the language information into the pixel embeddings atearly stages of the image encoder, as shown in Fig. 1(b).

Original: The alignment happens more earlier thanbefore, starting from the word-pixel level which is more fine-grained.
Revised: The alignment happens more earlier thanbefore, starting from the word-pixel level which is more fine-grained.

Original: Though effective, in Fig. 1(a) and (b), we can also observe many undesirable pixel fragments andlocalization errors in their failure cases.
Revised: Though effective, in Fig. 1(a) and (b), we can also observe many undesirable pixel fragments andlocalization errors in their failure cases.

Original: The reason lies behind these issues may be that the pixel-levelalignment (e.g., word-pixel and sentence-pixel level) connects each pixel embedding to languageindependently, and ignores the visual and semantic consistency among pixels in an object.
Revised: The reason lies behind these issues may be that the pixel-levelalignment (e.g., word-pixel and sentence-pixel level) connects each pixel embedding to languageindependently, and ignores the visual and semantic consistency among pixels in an object.

Original: Therefore,the predicted masks may miss some target pixels or spill over to other objects.
Revised: Therefore,the predicted masks may miss some target pixels or spill over to other objects.

Original: To deal with thisproblem, we propose to regard the pixels of an object as a whole and explicitly introduce object maskconstraints for the cross-modal alignment, at both the fine-grained and the coarse-grained levels.
Revised: To deal with thisproblem, we propose to regard the pixels of an object as a whole and explicitly introduce object maskconstraints for the cross-modal alignment, at both the fine-grained and the coarse-grained levels.

Original: Specifically, in this paper, we propose CoupAlign that couples sentence-mask alignment with wordpixel alignment to enforce the object mask constraint for improving the localization and segmentationperformance of RIS.
Revised: Specifically, in this paper, we propose CoupAlign that couples sentence-mask alignment with wordpixel alignment to enforce the object mask constraint for improving the localization and segmentationperformance of RIS.

Original: The Word-Pixel Alignment (WPA) module enables cross-modal interactionat the intermediate layers of the vision and language encoders.
Revised: The Word-Pixel Alignment (WPA) module enables cross-modal interactionat the intermediate layers of the vision and language encoders.

Original: Then a set of mask proposals arecalculated based on the word-pixel aligned features to indicate the possible objects in an image.
Revised: Then a set of mask proposals arecalculated based on the word-pixel aligned features to indicate the possible objects in an image.

Original: The
Revised: The

Original: Sentence-Mask Alignment (SMA) module learns to weight the masks using the sentence embeddingto localize the referred object, and the weighted masks are projected back to provide mask signals foreffectively aggregating the pixels of the target.
Revised: Sentence-Mask Alignment (SMA) module learns to weight the masks using the sentence embeddingto localize the referred object, and the weighted masks are projected back to provide mask signals foreffectively aggregating the pixels of the target.

Original: To further enhance the word-pixel and sentence-maskalignments, we design an auxiliary loss that forces the pixels of the target to get close and the pixelsnot belonging to the target to get away.
Revised: To further enhance the word-pixel and sentence-maskalignments, we design an auxiliary loss that forces the pixels of the target to get close and the pixelsnot belonging to the target to get away.

Original: Our CoupAlign captures the pixel coherence at both visualand semantic levels, and hierarchically aligns the pixels and masks with language information.
Revised: Our CoupAlign captures the pixel coherence at both visualand semantic levels, and hierarchically aligns the pixels and masks with language information.

Original: Bydoing this, CoupAlign generates more accurate mask predictions for RIS.
Revised: Bydoing this, CoupAlign generates more accurate mask predictions for RIS.

Original: To validate the effectiveness of our method, we conduct extensive experiments on popular datasets(RefCOCO [16] and G-ref [30]).
Revised: To validate the effectiveness of our method, we conduct extensive experiments on popular datasets(RefCOCO [17] and G-ref [31]).

Original: It is shown that CoupAlign outperforms state-of-the-art RIS methodsby significant margins, e.g, about 2% oIoU increase on the validation and testing sets of RefCOCO.
Revised: It is shown that CoupAlign outperforms state-of-the-art RIS methodsby significant margins, e.g, about 2% oIoU increase on the validation and testing sets of RefCOCO.

Original: Besides, CoupAlign shows remarkable ability in distinguishing the target from multiple (even greaterthan twenty) objects of the same class, for example, localizing one person in a team photo using alanguage query like “middle row second kid from right”.
Revised: Besides, CoupAlign shows remarkable ability in distinguishing the target from multiple (even greaterthan twenty) objects of the same class, for example, localizing one person in a team photo using alanguage query like “middle row second kid from right”.

Original: The main contributions of this paper are summarized as follows:
Revised: The main contributions of this paper are summarized as follows:

Original: • We tackle the challenging problem of RIS via integrating the mask-aware constraint into thepixel-level semantic alignment for capturing the pixel coherence within objects.• We propose CoupAlign, a simple yet effective alignment framework, to couple sentencemask alignment with word-pixel alignment to hierarchically align pixels and masks withlinguistic information.•
Revised: • We tackle the challenging problem of RIS via integrating the mask-aware constraint into thepixel-level semantic alignment for capturing the pixel coherence within objects.• We propose CoupAlign, a simple yet effective alignment framework, to couple sentencemask alignment with word-pixel alignment to hierarchically align pixels and masks withlinguistic information.•

Original: We conduct comprehensive experiments to demonstrate that our CoupAlign predicts moreaccurate object locations and masks and significantly outperforms state-of-the-art RISmethods on popular benchmarks.
Revised: We conduct comprehensive experiments to demonstrate that our CoupAlign predicts moreaccurate object locations and masks and significantly outperforms state-of-the-art RISmethods on popular benchmarks.

Original:  Related Work
Revised:  Related Work

Original: Referring Image Segmentation aims to segment the objects referred by natural language expressions.
Revised: Referring Image Segmentation aims to segment the objects referred by natural language expressions.

Original: RIS Methods can be summarized into two categories.
Revised: RIS Methods can be summarized into two categories.

Original: One focuses on cross-model alignment andrepresentation learning [14, 23, 9].
Revised: One focuses on cross-model alignment andrepresentation learning [15, 24, 10].

Original: Another views this problem as a language-conditioned visualreasoning [43, 12], which utilizes a dependency parsing tree to iteratively select the proposal regions.
Revised: Another views this problem as a language-conditioned visualreasoning [44, 13], which utilizes a dependency parsing tree to iteratively select the proposal regions.

Original: Due to the breakthrough of Transformer [7] in the computer vision field and its remarkable fusionpower for multi-modality, many works [15, 6, 44] begin to explore the cross-model alignment basedon Transformer.
Revised: Due to the breakthrough of Transformer [7] in the computer vision field and its remarkable fusionpower for multi-modality, many works [16, 6, 45] begin to explore the cross-model alignment basedon Transformer.

Original: MDETR [15] shows that simply concatenating vision and language features andthen feeding them into the Transformer encoder and decoder can achieve surprising results in manyvision-language downstream tasks.
Revised: MDETR [16] shows that simply concatenating vision and language features andthen feeding them into the Transformer encoder and decoder can achieve surprising results in manyvision-language downstream tasks.

Original: VLT [6] is the first work to introduce the Transformer architectureto referring segmentation.
Revised: VLT [6] is the first work to introduce the Transformer architectureto referring segmentation.

Original: It designs a query generation module to utilize the contextual informationof images to enrich language expressions and improve robustness.
Revised: It designs a query generation module to utilize the contextual informationof images to enrich language expressions and improve robustness.

Original: EFN [9] and LAVT [44] insertlanguage-aware attention modules into the image encoding process to achieve early fusion of crossmodal features.
Revised: EFN [10] and LAVT [45] insertlanguage-aware attention modules into the image encoding process to achieve early fusion of crossmodal features.

Original: MaiL [23] abandons the dual-stream encoders and adopts an unified cross-model
Revised: MaiL [24] abandons the dual-stream encoders and adopts an unified cross-model

Original: Transformer to achieve word-pixel alignment.
Revised: Transformer to achieve word-pixel alignment.

Original: CRIS [40] focuses on how to transfer CLIP’s
Revised: CRIS [41] focuses on how to transfer CLIP’s

Original: [35]strong image-text alignment ability to benefit sentence-pixel alignment required by RIS.
Revised: [36]strong image-text alignment ability to benefit sentence-pixel alignment required by RIS.

Original: CRIS uses a
Revised: CRIS uses a

Original: Transformer decoder with word queries and a sentence-pixel contrastive loss to achieve this goal.
Revised: Transformer decoder with word queries and a sentence-pixel contrastive loss to achieve this goal.

Original: The above mentioned RIS methods mainly focus on aligning each pixel to word or sentence independently while ignoring the pixel coherence of the target object, thus resulting in inaccurate masks andlocalization.
Revised: The above mentioned RIS methods mainly focus on aligning each pixel to word or sentence independently while ignoring the pixel coherence of the target object, thus resulting in inaccurate masks andlocalization.

Original: In contrast, our method couples sentence-mask alignment with word-pixel alignment tohierarchically align pixels and masks with linguistic information, and improving the segmentationand localization accuracy.
Revised: In contrast, our method couples sentence-mask alignment with word-pixel alignment tohierarchically align pixels and masks with linguistic information, and improving the segmentationand localization accuracy.

Original: Vision Language Modeling targets at seeking an unified representation for the two modalities.
Revised: Vision Language Modeling targets at seeking an unified representation for the two modalities.

Original: Fromthe aspect of architecture, vision language models can be divided into single-stream and dual-streamarchitecture structures.
Revised: Fromthe aspect of architecture, vision language models can be divided into single-stream and dual-streamarchitecture structures.

Original: Single-stream models such as VisualBERT [20], Uniter [2], Uniencodervl [19] and Oscar [22], utilizes an unified encoder based on self-attention to fuse the vision andlanguage embeddings.
Revised: Single-stream models such as VisualBERT [21], Uniter [2], Uniencodervl [20] and Oscar [23], utilizes an unified encoder based on self-attention to fuse the vision andlanguage embeddings.

Original: For dual-stream models such as CLIP [35], ALIGN [13], FILIP [45] and
Revised: For dual-stream models such as CLIP [36], ALIGN [14], FILIP [46] and

Original: GLIP [21], they adopt two unimodality encoders to encode vision and language embeddings andutilizes the dot product to align the outputs.
Revised: GLIP [22], they adopt two unimodality encoders to encode vision and language embeddings andutilizes the dot product to align the outputs.

Original: Others like ViLBERT [27] and LXMERT [36] usestwo self-attention-based encoders to model intra-modality interactions and employ cross-attentionto model cross-modality interactions.
Revised: Others like ViLBERT [28] and LXMERT [37] usestwo self-attention-based encoders to model intra-modality interactions and employ cross-attentionto model cross-modality interactions.

Original: Inspired by GLIP [21], we adopt a hybrid encoder design,where dual encoders are used to extract vision and language features and the word-pixel alignment isrealized at the intermediate stages of the encoders.
Revised: Inspired by GLIP [22], we adopt a hybrid encoder design,where dual encoders are used to extract vision and language features and the word-pixel alignment isrealized at the intermediate stages of the encoders.

Original:  Methodology
Revised:  Methodology

Original: 3.1 Overview
Revised: 3.1 Overview

Original: The overall architecture of our CoupAlign is shown in Fig.
Revised: The overall architecture of our CoupAlign is shown in Fig.

Original: It is designed following the encoderdecoder paradigm.
Revised: It is designed following the encoderdecoder paradigm.

Original: The CoupAlign encoder consists of an image encoder f v and a language encoderf l for extracting vision and language embeddings, a Word-Pixel Alignment (WPA) module thatenables the cross-modal interaction at the intermediate layers of the visual and language encoders,and a cross attention module for cross-modal fusion after encoders.
Revised: The CoupAlign encoder consists of an image encoder f v and a language encoderf l for extracting vision and language embeddings, a Word-Pixel Alignment (WPA) module thatenables the cross-modal interaction at the intermediate layers of the visual and language encoders,and a cross attention module for cross-modal fusion after encoders.

Original: The CoupAlign decoder includesa mask generator module that produces N segment proposals, a segmentation head (SegHead)
Revised: The CoupAlign decoder includesa mask generator module that produces N segment proposals, a segmentation head (SegHead)

Original: that
Revised: that

Original: Language Encoder
Revised: Language Encoder

Original: Image Encoder
Revised: Image Encoder

Original: WPA
Revised: WPA

Original: C r o ssA tt e n t i o n ⨂ queries
Revised: C r o ssA tt e n t i o n ⨂ queries

Original: WPA …
Revised: WPA …

Original: SMA
Revised: SMA

Original: CoupAlign Encoder
Revised: CoupAlign Encoder

Original: CoupAlign Decoder
Revised: CoupAlign Decoder

Original: W ord- P ixel
Revised: W ord- P ixel

Original: A lignment
Revised: A lignment

Original: S entence-
Revised: S entence-

Original: M ask A lignment
Revised: M ask A lignment

Original: The closest girl on right …
Revised: The closest girl on right …

Original: The closest girl right … … … … … … … …
Revised: The closest girl right … … … … … … … …

Original: 𝑄 !
Revised: 𝑄 !

Original: 𝑌 " 𝑌 # 𝑄 $ 𝑆 !
Revised: 𝑌 " 𝑌 # 𝑄 $ 𝑆 !

Original: 𝑉 " 𝑉 % 𝑉 & 𝑉 ’ 𝐿 ( 𝐿 ) 𝑉 !
Revised: 𝑉 " 𝑉 % 𝑉 & 𝑉 ’ 𝐿 ( 𝐿 ) 𝑉 !

Original: 𝑄 𝑀
Revised: 𝑄 𝑀

Original: [Figure]
Revised: Mask Generator

Original: S e g H e a d
Revised: S e g H e a d

Original: [Figure]

Original: closest

Original: [Figure]

Original: on

Original: [Figure] 𝑁 … [Figure] ⨂
Revised: The closest girl on right 𝑁 … … ⨂

Original: Aux Loss
Revised: Aux Loss

Original: Seg Loss
Revised: Seg Loss

Revised: Auxiliary Loss

Original: [Figure]
Revised: [Figure]

Original: upsamples the pixel-level embeddings, and a Sentence-Mask Alignment module (SMA) to weight themasks via language sentence and then project back to the pixel-level embeddings to enforce objectmask constraint.

Original: Together with the segmentation loss, an auxiliary loss that compares foregroundpixels with background ones to enhance the hierarchical aligning at word-pixel and sentence-masklevels.

Original: 3.2 CoupAlign Encoder
Revised: 3.2 CoupAlign Encoder

Original: [Figure]
Revised: [Figure]

Original: Image Encoder .
Revised: Image Encoder .

Original: Given an image I ∈
Revised: Given an image I ∈

Original: R H
Revised: R H

Original: × W × 3 , we adopt Swin Transformer [25] as the imageencoder backbone to extract multi-level visual features.
Revised: × W × 3 , we adopt Swin Transformer [26] as the imageencoder backbone to extract multi-level visual features.

Original: We utilize the features from the 1 st– 4 thstages of Swin Transformer, denoted as { V i ∈ R H i × W i × C i } 4 i =1 , where C i , H i , W
Revised: We utilize the features from the 1 st– 4 thstages of Swin Transformer, denoted as { V i ∈ R H i × W i × C i } 4 i =1 , where C i , H i , W

Original: i are the numberof channels, the height, and the width of the feature map at the i -th stage, respectively.
Revised: i are the numberof channels, the height, and the width of the feature map at the i -th stage, respectively.

Original: Language Encoder.
Revised: Language Encoder.

Original: We use BERT [5] with 12 layers as our language encoder backbone.
Revised: We use BERT [5] with 12 layers as our language encoder backbone.

Original: Givena referring expression { w i } Ti =1 , where T is the length of the expression, we first use the BERTtokenizer based on WordPiece embeddings [42] to get the word embeddings E ∈ R T × D , where
Revised: Givena referring expression { w i } Ti =1 , where T is the length of the expression, we first use the BERTtokenizer based on WordPiece embeddings [43] to get the word embeddings E ∈ R T × D , where

Original: D is the hidden feature dimensionality of BERT.
Revised: D is the hidden feature dimensionality of BERT.

Original: Then the word embeddings E are fed into thehidden layers of BERT to extract language features.
Revised: Then the word embeddings E are fed into thehidden layers of BERT to extract language features.

Original: In order to make word-pixel alignment withvision features from Swin Transformer, we aggregate every 3 hidden layers into a stage, so that wecan have the same stage number of language features as the vision features, which are denoted as{ L i ∈ R T × D } 4 i =1 .
Revised: In order to make word-pixel alignment withvision features from Swin Transformer, we aggregate every 3 hidden layers into a stage, so that wecan have the same stage number of language features as the vision features, which are denoted as{ L i ∈ R T × D } 4 i =1 .

Original: We also utilize the output [CLS] token as the global semantic representation ofthe referring expression, which is denoted as L g ∈ R 1 × D .
Revised: We also utilize the output [CLS] token as the global semantic representation ofthe referring expression, which is denoted as L g ∈ R 1 × D .

Original: Word-Pixel Alignment (WPA).
Revised: Word-Pixel Alignment (WPA).

Original: For segmentation task, we need to obtain a fine-grained segmentationmask, which means that low-level features like words in sentences and edges and shapes of objectsin images are crucial for finding correct segments.
Revised: For segmentation task, we need to obtain a fine-grained segmentationmask, which means that low-level features like words in sentences and edges and shapes of objectsin images are crucial for finding correct segments.

Original: The cross-modal alignment at early encoderstage is proved to be effective for referring image segmentation [6, 44].
Revised: The cross-modal alignment at early encoderstage is proved to be effective for referring image segmentation [6, 45].

Original: Their early alignmentsare in single-direction of language to vision.
Revised: Their early alignmentsare in single-direction of language to vision.

Original: In contrast, our WPA module is implemented with bidirectional cross attention to achieve both language-to-vision and vision-to-language alignment, as:
Revised: In contrast, our WPA module is implemented withbidirectional cross attention to achieve both language-to-vision and vision-to-language alignment, as:

Original: [Equation]
Revised: [Equation]

Original: where SwinStage i ( · ) represents the Stage i defined in Swin Transformer.
Revised: where SwinStage i ( · ) represents the Stage i defined in Swin Transformer.

Original: BERTStage i ( · ) representsthe Stage i of BERT.
Revised: BERTStage i ( · ) representsthe Stage i of BERT.

Original: The cross-modality interaction is achieved by the bidirectional cross attention(BiAttn), which computes the context vectors of one modality by attending to the other modality:
Revised: The cross-modality interaction is achieved by the bidirectional cross attention(BiAttn), which computes the context vectors of one modality by attending to the other modality:

Original: [Equation]
Revised: [Equation]

Original: where d is the dimension of the joint embedding space, and W vi ∈ R C i × d ,
Revised: where d is the dimension of the joint embedding space, and W vi ∈ R C i × d ,

Original: W li ∈ R D
Revised: W li ∈ R D

Original: × d , ˆ W li ∈
Revised: × d , ˆ W li ∈

Original: R d × C i , ˆ W vi ∈ R d × D are all projection matrices.
Revised: R d × C i , ˆ W vi ∈ R d × D are all projection matrices.

Original: In order to prevent the fused features V ′ i and L ′ i from overwhelming the original signals V i and L irespectively.
Revised: In order to prevent the fused features V ′ i and L ′ i from overwhelming the original signals V i and L irespectively.

Original: We design a Gate network [44] to control the fused information flow:
Revised: We design a Gate network [45] to control the fused information flow:

Original: [Equation]
Revised: [Equation]

Original: where F i is the fused feature from BiAttn, ⊙ indicates element-wise multiplication, and MLP is atwo-layer perceptron with the first layer being a linear layer followed by the ReLU activation [31]and the second layer being a linear layer followed by a hyperbolic tangent activation.
Revised: where F i is the fused feature from BiAttn, ⊙ indicates element-wise multiplication, and MLP is atwo-layer perceptron with the first layer being a linear layer followed by the ReLU activation [32]and the second layer being a linear layer followed by a hyperbolic tangent activation.

Original: Cross Attention.
Revised: Cross Attention.

Original: To better capture high-level semantics and generate the pixel-level fused features,we use a plain multi-head attention layer [39] to fuse the high-level vision and language featuresfrom the image and language encoders.
Revised: To better capture high-level semantics and generate the pixel-level fused features,we use a plain multi-head attention layer [40] to fuse the high-level vision and language featuresfrom the image and language encoders.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 2, first, we obtain the high-level visionfeatures V o ∈
Revised: 2, first, we obtain the high-level visionfeatures V o ∈

Original: R H o ×
Revised: R H o ×

Original: W o × C o from the output of the image encoder and the high-level language features
Revised: W o × C o from the output of the image encoder and the high-level language features

Original: L o ∈ R T × D from the language encoder.
Revised: L o ∈ R T × D from the language encoder.

Original: Then we map the vision and language features to R D .
Revised: Then we map the vision and language features to R D .

Original: Afterthat we concatenate them to produce a F o ∈ R ( H o W o + T ) × D , which is fed into the cross attentionlayer.
Revised: Afterthat we concatenate them to produce a F o ∈ R ( H o W o + T ) × D , which is fed into the cross attentionlayer.

Original: We add learnable positional emdeddings e p ∈
Revised: We add learnable positional emdeddings e p ∈

Original: R H o × W
Revised: R H o × W

Original: o × D to the vision features before theconcatenation.
Revised: o × D to the vision features before theconcatenation.

Original: Finally the output of the cross attention layer is added with the original input visionfeatures to produce a low-resolution feature map S o ∈ R H
Revised: Finally the output of the cross attention layer is added with the original input visionfeatures to produce a low-resolution feature map S o ∈ R H

Original: o × W o × C
Revised: o × W o × C

Original: o , as:
Revised: o , as:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where W vo ∈ R C o × D and W lo ∈ R D × D are projection matrices and [ ; ] denotes concatenation.
Revised: (8) where W vo ∈ R C o × D and W lo ∈ R D × D are projection matrices and [ ; ] denotes concatenation.

Original: 3.3 CoupAlign Decoder
Revised: 3.3 CoupAlign Decoder

Original: Mask Generator.
Revised: Mask Generator.

Original: To provide object mask constraints for the word-pixel alignment, we first design amask generator that produces N segment proposals based on the output feature of the CoupAlignencoder.
Revised: To provide object mask constraints for the word-pixel alignment, we first design amask generator that produces N segment proposals based on the output feature of the CoupAlignencoder.

Original: The generator is built upon a 6-layer transformer decoder.
Revised: The generator is built upon a 6-layer transformer decoder.

Original: It takes S o ∈ R H o × W o × C o and
Revised: It takes S o ∈ R H o × W o × C o and

Original: N learnable queries Q ∈ R N ×
Revised: N learnable queries Q ∈ R N ×

Original: d q as input, where d q is the hidden dimensionality of the transformerdecoder, and outputs the hidden feature Q o ∈ R N × d q of the N segment tokens, as:
Revised: d q as input, where d q is the hidden dimensionality of the transformerdecoder, and outputs the hidden feature Q o ∈ R N × d q of the N segment tokens, as:

Original: [Equation]
Revised: [Equation]

Original:  Q
Revised: ( Q

Original: [Equation]
Revised: [Equation]

Revised: S

Original: [Equation]
Revised: [Equation]

Original: Segmetation Head (SegHead).
Revised: Segmetation Head (SegHead).

Original: To upsample the pixel-level features for generating final segmentationmaps, we construct a segmentation head that takes the CoupAlign encoder output feature S o and themulti-level visual features { V i } 

Revised: To upsample the pixel-level features for generating final segmentationmaps, we construct a segmentation head that takes the CoupAlign encoder output feature S o and themulti-level visual features { V i }

Original: i
Revised: i

Original: =1 as input, and obtains output features as:
Revised: =1 as input, and obtains output features as:

Original: [Equation]
Revised: [Equation]

Original: where ρ is a two-layer convolution network where each layer performs a 3 × 3 convolution followedby ReLU and batch normalization, Up denotes 2 × bilinear upsampling, γ is a 1 × 1 convolution thatmaps the vision features to R d s .
Revised: where ρ is a two-layer convolution network where each layer performs a 3 × 3 convolution followedby ReLU and batch normalization, Up denotes 2 × bilinear upsampling, γ is a 1 × 1 convolution thatmaps the vision features to R d s .

Original: We take Y 1 ∈
Revised: We take Y 1 ∈

Original: R H 

Revised: R H

Original: × W 4 ×
Revised: × W 4 ×

Original: d s as the output of SegHead.
Revised: d s as the output of SegHead.

Original: Sentence-Mask Alignment (SMA).
Revised: Sentence-Mask Alignment (SMA).

Original: After generating mask proposals and obtaining upsampledpixel-level embeddings, the sentence-mask alignment assigns each mask a score according to the language query and then project the weighted masks back to the pixel-level feature maps to introduceobject mask constraint for word-pixel alignment.
Revised: After generating mask proposals and obtaining upsampledpixel-level embeddings, the sentence-mask alignment assigns each mask a score according to thelanguage query and then project the weighted masks back to the pixel-level feature maps to introduceobject mask constraint for word-pixel alignment.

Original: At first, We project Q o and Y 1 to ˆ Q o ∈ R N ×
Revised: At first, We project Q o and Y 1 to ˆ Q o ∈ R N ×

Original: Dand ˆ Y 1 ∈ R H 

Revised: Dand ˆ Y 1 ∈ R H

Original: × W 4 × D via learnable weights W Q ∈ R d q × D and W Y ∈ R d s × D .
Revised: × W 4 × D via learnable weights W Q ∈ R d q × D and W Y ∈ R d s × D .

Original: We use the maskembeddings ˆ Q o and the sentence representation L g generated from the language encoder to computeattention weight Q w ∈ R 1 × N .
Revised: We use the maskembeddings ˆ Q o and the sentence representation L g generated from the language encoder to computeattention weight Q w ∈ R 1 × N .

Original: Higher weight in Q w indicate that the corresponding mask is mostlikely to contain the object referred by the language.
Revised: Higher weight in Q w indicate that the corresponding mask is most likely to contain the object referred by the language.

Original: Then, ˆ Q o and ˆ Y 1 are multiplied to get N maskpredictions Y N ∈ R N
Revised: Then, ˆ Q o and ˆ Y 1 are multiplied to get N maskpredictions Y N ∈ R N

Original: × H 4 × W 4 .
Revised: × H 4 × W 4 .

Original: Finally, Y N is multiplied by Q w to generate the final mask prediction
Revised: Finally, Y N is multiplied by Q w to generate the final mask prediction

Original: M ∈ R H 

Revised: M ∈ R H

Original: × W 4 , which is further upsampled to the input image size.
Revised: × W 4 , which is further upsampled to the input image size.

Original: This procedure is represented as:
Revised: This procedure is represented as:

Original: [Equation]
Revised: [Equation]

Original: where sim ( · ) is cosine similarity and ⊗ denotes the broadcast tensor multiplication.
Revised: where sim ( · ) is cosine similarity and ⊗ denotes the broadcast tensor multiplication.

Original: 3.4 Loss Functions
Revised: 3.4 Loss Functions

Original: Segmentation Loss.
Revised: Segmentation Loss.

Original: In the context of referring image segmentation, each pixel in the image I isdiscriminated into background or foreground.
Revised: In the context of referring image segmentation, each pixel in the image I isdiscriminated into background or foreground.

Original: Thus this task can be viewed as a pixel-wise binaryclassification problem.
Revised: Thus this task can be viewed as a pixel-wise binaryclassification problem.

Original: Specifically, we use bilinear interpolation to upsample M to the originalimage size, obtaining M ′ .
Revised: Specifically, we use bilinear interpolation to upsample M to the originalimage size, obtaining M ′ .

Original: Let m ′ i
Revised: Let m ′ i

Original: and ˆ m i be the values of pixel i from M ′ and ˆ M (ground truthmask) respectively.
Revised: and ˆ m i be the values of pixel i from M ′ and ˆ M (ground truthmask) respectively.

Original: Then the segmentation loss are implemented via a binary cross entropy loss as:
Revised: Then the segmentation loss are implemented via a binary cross entropy loss as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where σ denotes sigmoid function, and j is the image index in a training batch.
Revised: where σ denotes sigmoid function, and j is the image index in a training batch.

Original: Auxiliary Loss.
Revised: Auxiliary Loss.

Original: The word-pixel and sentence-mask alignment collaborate in a hierarchically learningmanner to enforce object mask constraint for improving referring image segmentation.
Revised: The word-pixel and sentence-mask alignment collaborate in a hierarchically learningmanner to enforce object mask constraint for improving referring image segmentation.

Original: We adopta pixel-level contrastive loss as auxiliary to further enhance the learning of the two alignments andenable the aligning consistency at both the fine-grained and coarse-grained level.
Revised: We adopta pixel-level contrastive loss as auxiliary to further enhance the learning of the two alignments andenable the aligning consistency at both the fine-grained and coarse-grained level.

Original: This loss forcesthe pixels within the ground truth masks to get closer, and the rest pixels to get away.
Revised: This loss forcesthe pixels within the ground truth masks to get closer, and the rest pixels to get away.

Original: Thereforethe foreground pixels are aggregated together to generate more accurate predictions.
Revised: Thereforethe foreground pixels are aggregated together to generate more accurate predictions.

Original: Specifically,we resize the ground truth mask ˆ M to the same shape as the segmentation map Y 1 .
Revised: Specifically,we resize the ground truth mask ˆ M to the same shape as the segmentation map Y 1 .

Original: Let y i be thevector of Y 1 in pixel i .
Revised: Let y i be thevector of Y 1 in pixel i .

Original: Then we use the resized mask to divide positive and negative samples in eachimage.
Revised: Then we use the resized mask to divide positive and negative samples in eachimage.

Original: For pixel i , if ˆ m i is 0, then y i belongs to the negative set N and is denoted as y − i ; otherwisey i belongs to the positive set P and is denoted as y + i .
Revised: For pixel i , if ˆ m i is 0, then y i belongs to the negative set N and is denoted as y − i ; otherwisey i belongs to the positive set P and is denoted as y + i .

Original: Then we average the negative vectors and thepositive vectors to get the negative prototype ˆ y − and the positive prototype ˆ y + .
Revised: Then we average the negative vectors and thepositive vectors to get the negative prototype ˆ y − and the positive prototype ˆ y + .

Original: Afterward we adopt
Revised: Afterward we adopt

Original: InfoNCE [38] to compute the contrastive loss, as:
Revised: InfoNCE [39] to compute the contrastive loss, as:

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 16) where τ is the temperature parameter.
Revised: where τ is the temperature parameter.

Original: Finally, the total loss for the training is that:
Revised: Finally, the total loss for the training is that:

Original: [Equation]
Revised: [Equation]

Original: 17) where B is the batch size, λ is a hyper-parameter.
Revised: where B is the batch size, λ is a hyper-parameter.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Settings
Revised: 4.1 Settings

Original: Dataset.
Revised: Dataset.

Original: We evaluate our method on two widely-used datasets: RefCOCO [16] and G-Ref [30].
Revised: We evaluate our method on four widely-used datasets: RefCOCO [47], RefCOCO+[47], G-Ref [31] and ReferIt [17].

Original: RefCOCO consists of 142,209 referring expressions for 50,000  objects in 19,994 images.
Revised: RefCOCO consists of 142,209 referring expressions for 50,000 objectsin 19,994 images and RefCOCO+ has 141,564 referring expressions for 49,856 objects in 19,992images.

Original: G-Ref [46]
Revised: G-Ref was collected on Amazon Mechanical Turk in a non-interactive setting which consists

Original: [Table]
Revised: [Table]

Original: non-interactive setting which consists of 85,474referring expressions for 54,822 objects in 26,711 images.
Revised: of 85,474 referring expressions for 54,822 objects in 26,711 images.

Original: There are two different partitions of the G-Ref dataset, one by UMD and the other by Google.
Revised: ReferIt has 130,525 referringexpressions in 19,894 images which are collected from IAPR TC-12 [8].

Revised: RefCOCO+ is disallowedfrom using location words in referring expressions.

Original: The languages used in RefCOCO tend tobe more concise and less flowery than the languages used in the G-Ref.
Revised: The languages used in RefCOCO and RefCOCO+tend to be more concise and less flowery than the languages used in the G-Ref.

Original: The expressions in G-Reftend to be longer with an average of 8.43 words, which makes it more challenging compared with RefCOCO.
Revised: The expressions in G-Ref tend to be longer with an average of 8.43 words, which makes it more challenging comparedwith RefCOCO and RefCOCO+. Expressions in ReferIt are usually shorter than the other datasets.

Original: Metrics.
Revised: Metrics.

Original: We adopt three common used metrics in referring image segmentation: the overallintersection-over-union (oIoU), the mean intersection-over-union (mIoU) and prec@X. The oIoUmeasures as the ratio between the total intersection area and the total union area of all test samples.
Revised: We adopt three common used metrics in referring image segmentation: the overallintersection-over-union (oIoU), the mean intersection-over-union (mIoU) and prec@X. The oIoUmeasures as the ratio between the total intersection area and the total union area of all test samples.

Original: By default, we use this metric to compare CoupAlign with other works.
Revised: By default, we use this metric to compare CoupAlign with other works.

Original: The mIoU is the averageof the IoU over all test samples.
Revised: The mIoU is the averageof the IoU over all test samples.

Original: prec@X measures the percentage of test images with an IoU scorehigher than a threshold ϵ , where ϵ ∈ { 0 . 5 , 0 . 7 , 0 . 9 } .
Revised: prec@X measures the percentage of test images with an IoU scorehigher than a threshold ϵ , where ϵ ∈ { 0 . 5 , 0 . 7 , 0 . 9 } .

Original: Implementation Details.
Revised: Implementation Details.

Original: We implement our model in the PyTorch [32] framework.
Revised: We implement our model in the PyTorch [33] framework.

Original: We choose
Revised: We choose

Original: BERT BASE [5] with 12 layers and hidden size 768 as our language encoder.
Revised: BERT BASE [5] with 12 layers and hidden size 768 as our language encoder.

Original: We slightly modify theimplementation of BERT in HuggingFace’s Transformer library [41] and initialize it with officialpre-trained weights.
Revised: We slightly modify theimplementation of BERT in HuggingFace’s Transformer library [42] and initialize it with officialpre-trained weights.

Original: The image encoder layers are initialized with the classification weights pretrained on ImageNet-22K [4].
Revised: The image encoder layers are initialized with the classification weights pretrained on ImageNet-22K [4].

Original: The number of queries of the mask generator N is 100.
Revised: The number of queries of the mask generator N is 100.

Original: The rest ofthe weights in our model are randomly initialized.
Revised: The rest ofthe weights in our model are randomly initialized.

Original: We adopt AdamW [26] optimizer with weightdecay 0.01.
Revised: We adopt AdamW [27] optimizer with weightdecay 0.01.

Original: We adopt the polynomial learning rate decay schedule and set the initial learning rateto 3e-5, the end learning rate to 1.5e-5 and the max decay epoch to 25.
Revised: We adopt the polynomial learning rate decay schedule and set the initial learning rateto 3e-5, the end learning rate to 1.5e-5 and the max decay epoch to 25.

Original: Our model is trained for 50epochs with batch size 16.
Revised: Our model is trained for 50epochs with batch size 16.

Original: The images are resized to 448 × 448 without data augmentation and themaximum sentence length is set to 30.
Revised: The images are resized to 448 × 448 without data augmentation and themaximum sentence length is set to 30.

Original: The weight of the auxiliary loss λ is set to 0 .
Revised: The weight of the auxiliary loss λ is set to 0 .

Original: 4.2 Comparison with State-of-the-Arts
Revised: 4.2 Comparison with State-of-the-Arts

Original: Results on RefCOCO.
Revised: Results on RefCOCO.

Original: In Tab. 1, we compare CoupAlign with state-of-the-art (SOTA) RIS methodson two widely-used datasets (RefCOCO and G-Ref) on the oIoU metric.
Revised: In Tab. 1, we compare CoupAlign with state-of-the-art (SOTA) RIS methodsby oIoU metric.

Original: The language expressionsprovided by RefCOCO contain many words of positional relationships, e.g., “The closet girl on theright”, which requires the RIS models to not only capture the pixel coherence within objects, but alsothe spatial relationship among multiple objects of the same class.
Revised: The language expressions provided by RefCOCO contain many words of positionalrelationships, e.g., “The closet girl on the right”, which requires the RIS models to not only capturethe pixel coherence within objects, but also the spatial relationship among multiple objects of thesame class.

Original: Compared with the SOTA method LAVT [44], CoupAlign achieves better performance with absolute oIoU increases of 1.97%, 1.94%,and 1.79% on the validation, testA, and testB sets of RefCOCO, respectively.
Revised: Compared with the SOTA method LAVT [45], CoupAlign achieves better performancewith absolute oIoU increases of 1.97%, 1.94%, and 1.79% on the validation, testA, and testB sets of RefCOCO, respectively.

Original: Results on G-Ref.
Revised: Results on RefCOCO+.

Revised: As shown in the Tab. 1, the performance of CoupAlign on RefCOCO+is comparable to or better than LAVT which is the most recent state-of-the-art method (publishedin CVPR2022).

Original: The language annotations of G-Ref have more complex language structurewith more words, e.g., “chocolate dessert directly in front ofus in the center”, which requiresmore fine-detailed word-pixel alignment, more discriminative sentence-mask alignment, and a morecomprehensive understanding of  the sentence.
Revised: The improvements on RefCOCO+ are less significant than those on G-Ref and RefCOCO. This is because the language captions in RefCOCO+ hardly contain the descriptionsof the relative or absolute spatial locations (e.g., "closest" or "right bottom") of the target object inimages, and the effectiveness of our method mainly lies in the ability of localizing objects in suchchallenging scenarios, e.g., "middle row second kid from right" in Fig.

Original: As shown in Tab. 1 on G-Ref (UMD partition), our
Revised: Results on G-Ref.

Original: CoupAlign outperforms LAVT by 1.6% and 0.13% on the validation and test sets, respectively. Theconsistent improvements on all the evaluated datasets demonstrate the capability of our CoupAlign inmodeling hierarchical cross-modal alignment at pixel and mask levels.
Revised: The language annotations of G-Ref have more complex language structurewith more words, e.g., “chocolate dessert directly in front of us in the center”, which requiresmore fine-detailed word-pixel alignment, more discriminative sentence-mask alignment, and a more

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: comprehensive understanding of the sentence.

Revised: As shown in Tab. 1 on G-Ref (UMD partition), our

Revised: CoupAlign outperforms LAVT by 1.6% and 0.13% on the validation and test sets, respectively.

Revised: Theconsistent improvements on all the evaluated datasets demonstrate the capability of our CoupAlign inmodeling hierarchical cross-modal alignment at pixel and mask levels.

Original: Analysis of Localization Ability.
Revised: Analysis of Localization Ability.

Original: If the IoU between a mask prediction and the Ground Truth maskis smaller than 0.5, we consider the prediction as a failure case for localization.
Revised: If the IoU between a mask prediction and the Ground Truth maskis smaller than 0.5, we consider the prediction as a failure case for localization.

Original: In Tab. 2, we can seethat our CoupAlign has fewer failure cases than LAVT in most IoU ranges.
Revised: In Tab. 2, we can seethat our CoupAlign has fewer failure cases than LAVT in most IoU ranges.

Original: Especially, in the IoUrange of [0,0.1), the number of failure cases of CoupAlign decreases the most, which further validatesthe remarkable ability of CoupAlign.

Original: We also show some visualization examples in Fig. 3(b).

Original: Mask Number N .
Revised: Mask Number N .

Original: From Tab. 3, we find that the mask number N in mask generator is correlated tothe model performance.
Revised: From Tab. 3, we find that the mask number N in mask generator is correlated tothe model performance.

Original: To balance the computational cost and performance, N is set to 100.
Revised: To balance the computational cost and performance, N is set to 100.

Original: 4.3 Visualization
Revised: 4.3 Visualization

Original: Mask Quality and Localization Ability.
Revised: Success cases and failure cases.

Original: In Fig.
Revised: In Fig.

Original: 3, we show some representative visualizationexamples of success predictions by CoupAlign while the SOTA method LAVT typically fails.
Revised: 3, we show some representative visualization examples ofsuccess predictions and failure cases of CoupAlign.

Original: Theexamples are divided into two groups according to the failure types of LAVT, including inaccuratemasks and localization results.

Original: The examples come from the RefCOCO validation set.
Revised: The examples are from the RefCOCO val. set.

Original: From Fig. 3(a),we can see thatour CoupAlign predicts more accurate masks than LAVT, especially in the sceneswhere crowded objects have similar color and context and stay close to each other, e.g., the “hotdog”and “fridge”. It can be observed from Fig. 3(b) that our CoupAlign has remarkable ability in localizingthe target from crowded objects, even when the language expression describes multiple conditions ofdifferent types, e.g., “left baby elephant face towards us”.
Revised: We can see that CoupAlign works well in the scenes where crowded objects have similar color andcontext and stay close to each other. However, if the boundary of the object is blurred, or there existsocclusion, the segmentation prediction is roughly accurate, but not precise enough.

Original: Word-Pixel and Sentence-Mask Alignment.
Revised: Word-Pixel and Sentence-Mask Alignment.

Original: To validate whether our WPA and SMA modulescan perform accurate and consistent alignments, we visualize the intermediate attention maps ofthe alignment modules in Fig.
Revised: To validate whether our WPA and SMA modulescan perform accurate and consistent alignments, we visualize the intermediate attention maps ofthe alignment modules in Fig.

Original: It can be observed that our WPA module accurately highlightsthe most relevant pixels corresponding to the word.
Revised: It can be observed that our WPA module accurately highlightsthe most relevant pixels corresponding to the word.

Original: Note that the vocabulary of RIS languageannotations is very large compared to the small fixed label set of semantic segmentation.
Revised: Note that the vocabulary of RIS languageannotations is very large compared to the small fixed label set of semantic segmentation.

Original: Surprisingly,our CoupAlign can not only capture the semantic meaning of words in various types, e.g., nouns,positions, and attributes, but also distinguish synonyms, e.g., child, man, and lady, which are roughlydefined as “person” in many semantic segmentation datasets like PASCAL VOC [8].
Revised: Surprisingly,our CoupAlign can not only capture the semantic meaning of words in various types, e.g., nouns,positions, and attributes, but also distinguish synonyms, e.g., child, man, and lady, which are roughlydefined as “person” in many semantic segmentation datasets like PASCAL VOC [9].

Original: As for the SMAmodule, we show three masks with their attention weights ranked in descending order.
Revised: As for the SMAmodule, we show three masks with their attention weights ranked in descending order.

Original: The maskwith the highest weight is the most accurate one and the masks with lower weights are less accurate.
Revised: The maskwith the highest weight is the most accurate one and the masks with lower weights are less accurate.

Original: Fig.
Revised: Fig.

Original: 4(right) demonstrates that our SMA module provides the effective mask constraint for benefitingthe hierarchical aligning at pixel and mask levels.[Figure] [Figure] oIoU
Revised: 4(right) demonstrates that our SMA module provides the effective mask constraint for benefitingthe hierarchical aligning at pixel and mask levels.

Original: [Table]

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Original: Effect of WPA.
Revised: Effect of WPA Types.

Original: Our WPA module enables cross-modal interaction at both low-level and high-levelencoding stages.
Revised: Our WPA module enables cross-modal interaction at both low-level and highlevel encoding stages.

Original: In Tab. 4, Bi-WPA indicates our WPA which is implemented via bidirectional crossattention, Uni-WPA represent unidirectional attention only from language to image features, as in LAVT [44].
Revised: In Tab. 4, Bi-WPA indicates our WPA which is implemented via bidirectionalcross attention, Uni-WPA represent unidirectional attention only from language to image features, asin LAVT [45].

Original: The performance dramatically drops by 4.3% (74.70% vs. 70.43%) on oIoU when WPAis removed.
Revised: The performance dramatically drops by 4.3% (74.70% vs. 70.43%) on oIoU when WPA is removed.

Original: And also when we replace our Bi-WPA with Uni-WPA, the oIoU result drops by 2%(74.70% vs. 72.70%).
Revised: And also when we replace our Bi-WPA with Uni-WPA, the oIoU result drops by2% (74.70% vs. 72.70%).

Original: These results demonstrate the effectiveness of our WPA module.
Revised: These results demonstrate the effectiveness of our WPA module.

Revised: Effect of WPA’s Number and Position.

Revised: In our experiment, we used four WPA modules, two ofwhich are in the early encoding stage and the other two are in the late encoding stage.

Original: Effect of SMA.
Revised: To study the [Figure] [Figure] [Table]

Revised: [Table]

Revised: effects of the numbers of WPA modules, we first conduct two baseline models that alternativelyremove two WPA at early or late encoding stages.

Revised: As shown in the Tab. 5, when we remove thelast two WPA modules the performance drops about 2%, and when we remove the first two WPAmodules the performance drops about 0.8%.

Revised: These results validate the effectiveness of WPA modulesat both early and late stages and indicate that the latter WPA modules play a more important role inour modules.

Revised: Then, we only use one WPA module, which is inserted at different encoding stages.

Revised: The WPA module at the 4-th stage is more effective than those inserted at other stages.

Original:  In Tab. 4, when SMA is removed, the oIoU result decreases by 1.7% (74.70% vs.73.02%), since the cross-modal alignment becomes less accurate without mask constraints.
Revised: Effect of SMA. In Tab. 4, when SMA is removed, the oIoU result decreases by 1.7% (74.70% vs.73.02%), since the cross-modal alignment becomes less accurate without mask constraints.

Original: Effect of Auxiliary Loss.
Revised: Effect of Auxiliary Loss.

Original: From the last two columns of Tab. 4, we observe a performance drop of1% (74.70% vs. 73.70%) on oIoU when the auxiliary loss is disabled, which verifies the ability of theloss in enhancing the word-pixel and sentence-mask alignments.
Revised: From the last two columns of Tab. 4, we observe a performance drop of1% (74.70% vs. 73.70%) on oIoU when the auxiliary loss is disabled, which verifies the ability of theloss in enhancing the word-pixel and sentence-mask alignments.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose CoupAlign which is a simple yet effective cross-modal alignment frameworkthat couples sentence-mask alignment with word-pixel alignment to enforce mask-aware constraintfor referring segmentation.
Revised: In this work, we propose CoupAlign which is a simple yet effective cross-modal alignment frameworkthat couples sentence-mask alignment with word-pixel alignment to enforce mask-aware constraintfor referring segmentation.

Original: The word-pixel alignment module enables fine-grained cross-modalinteractions at the early encoding stage.
Revised: The word-pixel alignment module enables fine-grained cross-modalinteractions at the early encoding stage.

Original: The sentence-mask alignment module uses the sentence toweight the mask proposals generated based on the word-pixel aligned features, and then projects themask constraints back to guide the word-pixel alignment.
Revised: The sentence-mask alignment module uses the sentence toweight the mask proposals generated based on the word-pixel aligned features, and then projects themask constraints back to guide the word-pixel alignment.

Original: The auxiliary contrastive loss that comparesamong pixel pairs to further enhance the hierarchical alignments at word-pixel and sentence-masklevels.
Revised: The auxiliary contrastive loss that comparesamong pixel pairs to further enhance the hierarchical alignments at word-pixel and sentence-masklevels.

Original: Benefiting from the collaboration of the alignment learning at different levels, CoupAligncaptures both visual and semantic coherence of pixels within the referred object, and significantlyoutperforms state-of-the-art RIS methods.
Revised: CoupAlign captures both visual and semantic coherence of pixels within the referred object,and significantly outperforms state-of-the-art RIS methods.

Original: Especially, CoupAlign has surprising ability in localizingthe target from crowded objects of the same class, showing great potential on segmenting naturallanguage referred objects in real-world scenarios.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] Please see the Abstract and Introduction sections.
Revised: [Yes] Please see the Abstract and Introduction sections.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Please see the Conclusion section.
Revised: [Yes] Please see the Conclusion section.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Please seethe Conclusion section.(d)
Revised: [Yes] Please seethe Conclusion section.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Our code is
Revised: [Yes] Our code is

Original: enclosed in the supplementary material.
Revised: enclosed in the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Please see Section ??
Revised: [Yes] Please see Section ??

Original: for more experimental details.
Revised: for more experimental details.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] The experiments are conducted under a fixed seed, and
Revised: [Yes] The experiments are conducted under a fixed seed, and

Original: have the same configuration as previous works.
Revised: have the same configuration as previous works.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] All experiments are conducted on
Revised: [Yes] All experiments are conducted on

Original: an NVIDIA 3090 GPU.
Revised: an NVIDIA 3090 GPU.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Please see the
Revised: [Yes] Please see the

Original: Experiment and Reference sections.
Revised: Experiment and Reference sections.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] The data and models used in our work[Table]
Revised: [No] The data and models used in our work

Original: are publicly released.
Revised: are publicly released.

Original: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes] The data and models used in our work are publicly released.[Table]
Revised: [Yes] The data and models used in our work are publicly released.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] The data used in our work does not contain[Table]
Revised: [Yes] The data used in our work does not contain

Original: personally identifiable information or offensive content.
Revised: personally identifiable information or offensive content.

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]