Original: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency
Revised: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.
Revised: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.

Original: It is a recently introducedtask attracting growing research attention.
Revised: It is a recently introducedtask attracting growing research attention.

Original: However, all existing works make astrong assumption:
Revised: However, all existing works make astrong assumption:

Original: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.
Revised: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.

Original: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.
Revised: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.

Original: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.
Revised: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.

Original: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).
Revised: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).

Original: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).
Revised: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).

Original: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.
Revised: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.

Original: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.
Revised: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.

Original: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.
Revised: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.

Original: A new evaluation dataset,
Revised: A new evaluation dataset,

Original: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.
Revised: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.

Original: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.
Revised: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.

Original: Our model achieves the state-of-the-art performance on
Revised: Our model achieves the state-of-the-art performance on

Original: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.
Revised: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.

Original:  Introduction
Revised:  Introduction

Original: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.
Revised: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.

Original: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.
Revised: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.

Original: Unlikeother video segmentation tasks [42, 34, 33, 43] that only rely on visual cues, R-VOS [10] pairs atarget video with a linguistic expression referring to an object.
Revised: Unlikeother video segmentation tasks [45, 36, 35, 46] that only rely on visual cues, R-VOS [13] pairs atarget video with a linguistic expression referring to an object.

Original: Previous works [1, 41] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.
Revised: Previous works [1, 44] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.

Original: However, this assumption does not always hold in practice.
Revised: However, this assumption does not always hold in practice.

Original: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.
Revised: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.

Original: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.
Revised: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.

Revised: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.

Revised: Recently, MTTR [1]

Original: Query expression: “a kangaroo in the middle of two” others
Revised: Query expression: “a kangaroo in the middle of two” others

Original: [Figure]
Revised: [Figure]

Original:  v e[Figure] v
Revised: P o s iti v e

Original: [Figure]
Revised: [Figure]

Original: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.

Original: Recently, MTTR [1]employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.
Revised: employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.

Original: ReferFormer[41] follows the image-level method, ReTR [14], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.
Revised: ReferFormer[44] follows the image-level method, ReTR [19], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.

Original: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.
Revised: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.

Original: As shown in Figure 1, the segmented objects by MTTR and
Revised: As shown in Figure 1, the segmented objects by MTTR and

Original: ReferFormer are often not the object referred to by the linguistic expression.
Revised: ReferFormer are often not the object referred to by the linguistic expression.

Original: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.
Revised: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.

Original: We extend the current task definition of R-VOS [10] toaccept both paired and unpaired video and language inputs.
Revised: We extend the current task definition of R-VOS [13] toaccept both paired and unpaired video and language inputs.

Original: This new task, which we term Robust
Revised: This new task, which we term Robust

Original: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.
Revised: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.

Original: We reveal that this task is essentiallyrelated to two problems that are interrelated [27]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.
Revised: We reveal that this task is essentiallyrelated to two problems that are interrelated [31]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.

Original: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.
Revised: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.

Original: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.
Revised: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.

Original: We address this problem by incorporating an earlygrounding scheme, serving as a medium, to efficiently model the two tasks in a parallel manner.
Revised: We address this problem by incorporating an earlygrounding scheme, serving as a proxy, to efficiently model the two tasks in a parallel manner.

Original: In addition, we discriminate the semantic misalignment between the video and text by assessingthe cycle consistency between the original and reconstructed texts, thus alleviating the false-alarmproblem.
Revised: Inaddition, we discriminate the semantic misalignment between the video and text by assessing the cycleconsistency between the original and reconstructed texts, thus alleviating the false-alarm problem.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.
Revised: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.

Original: To investigate the robustness of current referring segmentation models, we introducethe R 2 -VOS task that accepts unpaired video and text as inputs.
Revised: To investigate the robustness of current referring segmentation models, we introduce the R 2 -VOS task that accepts unpaired video and text as inputs.•

Original: • We propose a pipeline that jointly optimizes the primary referring segmentation and dual expression reconstruction task and introduces a relational cycle consistency constraint to enhance the semantic alignment between visual and textual modalities.
Revised: We propose a pipeline that jointly optimizes the primary referring segmentation and dualexpression reconstruction task and introduces a relational cycle consistency constraint toenhance the semantic alignment between visual and textual modalities.•

Original: • Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS,and R 2 -Youtube-VOS dataset in terms of both performance and speed.
Revised: Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS, and R 2 -Youtube-VOS dataset in terms of both performance and speed.

Original:  Related Works
Revised:  Related Works

Original: Vision and language representation learning.
Revised: Vision and language representation learning.

Original: For vision-language representation learning, visualsemantic embedding (VSE) [6] and its improved versions [30, 13, 28, 3] have been proposed tofacilitate multimodal tasks, e.g., image-text matching. Recent works [21, 26, 3] introduce crossmodal attention layers and achieve superior improvement.
Revised: There have been a long line of studies on how tolearn better vision-language representation, e.g., multimodal attention [30, 50, 8, 3], fusion scheme[7, 14, 15, 51], multi-step reasoning [47, 10] and pretraining [37, 5, 17].

Original: KAC Net [2] leverages knowledge-aidedconsistency constraints to enhance semantic alignment for weakly supervised phrase grounding.
Revised: KAC Net [2] leverages knowledge-aided consistency constraints to enhance semantic alignment for weakly supervised phrasegrounding.

Original: A structure-preserving constraint [39] is proposed to preserve some intra-modal properties whenlearning vision-language representation for image-text retrieval.
Revised: A structure-preserving constraint [42] is proposed to preserve some intra-modal propertieswhen learning vision-language representation for image-text retrieval.

Original: Referring video object segmentation.
Revised: Referring video object segmentation.

Original: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.
Revised: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.

Original: URVOS [36] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.
Revised: URVOS [39] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.

Original: ClawCraneNet [16] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.
Revised: ClawCraneNet [21] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.

Original: ReferFormer [41] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.
Revised: ReferFormer [44] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.

Original: ReferFormer [41] employs a linguistic prior to the transformerdecoder to focus on the referred object.
Revised: ReferFormer [44] employs a linguistic prior to the transformerdecoder to focus on the referred object.

Original: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.
Revised: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.

Original: Different from other vision-language tasks, e.g., image-textretrieval [20, 22, 29] and video question answering [12, 37], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.
Revised: Different from other vision-language tasks, e.g., image-textretrieval [25, 26, 32] and video question answering [18, 40], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.

Original:  R 2 -VOS3.1 Task Definition
Revised:  R 2 -VOS3.1 Task Definition

Original: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .
Revised: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .

Original: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .
Revised: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .

Original: We define a video V and an expression E o to have semanticconsensus if the object
Revised: We define a video V and an expression E o to have semanticconsensus if the object

Original: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .
Revised: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .

Original: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.
Revised: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.

Original: 3.2 Problem Analysis
Revised: 3.2 Problem Analysis

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation][Figure] [Equation] [Figure]
Revised: [Equation]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We enable the joint optimization for cycle consistency with a cross-modal medium f m defined between all single-modal operations (i.e., Π encv , Π ence , Π decv and Π dece ).

Original: Point-wise consistency is not suitable in R 2 -VOS because the mapping between E and E ′ are not necessarily bijective because an object can be referred to by various textual expressions.

Original: (d) Instead, we apply a relational consistency to preserve distances and angles.

Original: Primary and dual problems for R 2 -VOS.
Revised: Primary and dual problems for R 2 -VOS.

Original: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .
Revised: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .

Original: By applying the Bayes rule, we obtain:
Revised: By applying the Bayes rule, we obtain:

Original: [Equation]
Revised: [Equation]

Original: As the prior p ( M o | V ) is not affected 
Revised: As the prior p ( M o | V ) is not affected by the expression E o , we consider maximizing p ( E o | V, M o )as a dual problem of the referring segmentation (primary problem), which is to reconstruct the textexpression given the video and object masks. We note that for negative videos, p ( E o | V, M o ) isundefined because the mask M o is empty. Thus, we only investigate the dual problem for positivevideos. The primary problem and the dual problem can be connected in a cycle path, i.e., from theoriginal expression E o to the reconstructed expression E ′

Original:  Video 𝑉 = 𝐈 ! !"
Revised: o through positive video queries, as shownin Figure 2 (a). We believe that the cycle constraint benefits to optimize the primary problem byenhancing the semantic consensus. Video 𝑉 = 𝐈 ! !"

Original: #$
Revised: #$

Original: Text exp.
Revised: Text exp.

Original: 𝐸 %
Revised: 𝐸 %

Original: Instance emb.
Revised: Instance emb.

Original: 𝐳 & &"#$
Revised: 𝐳 & &"#$

Original: Box 𝐁 ! !"
Revised: Box 𝐁 ! !"

Original: #$ Score 𝐒 ! !"
Revised: #$ Score 𝐒 ! !"

Original: #$
Revised: #$

Original: [Figure]
Revised: Visual Encoder

Original:  Encoder
Revised: Textual Encoder

Original: Visual Decoder
Revised: Visual Decoder

Original: Seg.
Revised: Seg.

Original: 𝐌 ! !"
Revised: 𝐌 ! !"

Original: #$ a cat laying by an owl
Revised: #$ a cat laying by an owl

Original: Skip connections
Revised: Skip connections

Original: Pos. / Neg.
Revised: Pos. / Neg.

Original: R
Revised: R

Original: Dot Product
Revised: Dot Product

Original: Repeat R
Revised: Repeat R

Original: Concatenation C
Revised: Concatenation C

Original: Reconstructed [Figure] emb [Figure]
Revised: Reconstructed text emb. 𝐞′

Original: 𝐞′ Relational Cycle 
Revised: Relational Cycle Consistency

Original: [Figure]

Original: Multi [Figure] scale feat.
Revised: Multi-scale feat.

Original: 𝐅 𝐅
Revised: 𝐅 𝐅

Original: Instance query 𝐳 ’
Revised: Instance query 𝐳 ’

Original: Sentence- [Figure] emb.
Revised: Sentence-level text emb.

Original: 𝐞
Revised: 𝐞

Original: C
Revised: C

Original: Conv Grounded feat.
Revised: Conv Grounded feat.

Original: 𝐟 ()*+,
Revised: 𝐟 ()*+,

Original: Dynamic kernel Θ 𝐠𝐟 - Video feat.
Revised: Dynamic kernel Θ 𝐠𝐟 - Video feat.

Original: 𝐟
Revised: 𝐟

Original: Word-level feat.
Revised: Word-level feat.

Original: 𝐠
Revised: 𝐠

Original: [Figure]
Revised: Multimodal Attention

Original: Transformer Decoder 𝒟 !
Revised: Transformer Decoder 𝒟 !

Original: [Figure]

Original:  query 𝐞 ’
Revised: Text query 𝐞 ’

Original: Early Grounding
Revised: Early Grounding

Original: V
Revised: V-T Projection Pool

Original: [Figure]
Revised: [Figure]

Original: Projection
Revised: Transformer Decoder

Original: [Figure]
Revised: [Figure]

Original: ℋ because the is

Original: , we only investigate the problem for The problem and the can be connected in a i. the o to the reconstructed E o through positive video queries as shown in Figure 2 (a). We that the cycle constraint benefits to optimize the primary problem by
Revised: ℋ

Original: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textural embedding space E ′ induced by positive videos.
Revised: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textual embedding space E ′ induced by positive videos.

Original: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.
Revised: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.

Original: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal medium f m to convey information of both  theinput ofthe primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).
Revised: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal intermediate feature f m to convey information of both theinput of the primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).

Original: f m is definedbetween the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv , Π dece , toonly focus on the multi-modal interaction.
Revised: f m isdefined between the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv ,Π dece , to only focus on the multi-modal interaction.

Original: Relational cycle consistency.
Revised: Relational cycle consistency.

Original: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.
Revised: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.

Original: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.
Revised: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.

Original: Instead, we take inspiration from relational knowledgedistillation [31], and introduce relational cycle consistency for E and E ′ .
Revised: Instead, we take inspiration from relational knowledgedistillation [33], and introduce relational cycle consistency for E and E ′ .

Original: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).
Revised: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).

Original: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e
Revised: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e

Original: ′ ∈ E ′ :
Revised: ′ ∈ E ′ :

Original: [Equation]
Revised: [Equation]

Original: ) where | · | and ∠ ( · ) denote distance and angle metrics.
Revised: where | · | and ∠ ( · ) denote distance and angle metrics.

Original:  Method
Revised:  Method

Original: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.
Revised: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.

Original: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .
Revised: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .

Original: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .
Revised: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .

Original: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.
Revised: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.

Original: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instance embedding { z t } Tt =1 .
Revised: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instanceembedding { z t } Tt =1 .

Original: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f
Revised: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f

Original: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .
Revised: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .

Original: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .
Revised: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .

Original: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .
Revised: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .

Original: We add relational constraint betweene ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.
Revised: We add relational constraint between e ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.

Original: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.
Revised: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.

Original: 4.1 Single-modal Feature Extraction
Revised: 4.1 Single-modal Feature Extraction

Revised: [Figure]

Original: Visual encoder.
Revised: Visual encoder.

Original: Following previous methods [1, 41, 40], we build the visual encoder with a visualbackbone and a deformable transformer encoder [46] on top of it.
Revised: Following previous methods [1, 44, 43], we build the visual encoder with a visualbackbone and a deformable transformer encoder [52] on top of it.

Original: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [9], and thenfed into a deformable transformer encoder [46] similar to the previous method [41].
Revised: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [12], andthen fed into a deformable transformer encoder [52] similar to the previous method [44].

Original: We denote themulti-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.
Revised: We denotethe multi-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.

Original: Textual encoder.
Revised: Textual encoder.

Original: We leverage a pre-trained linguistic model RoBERTa
Revised: We leverage a pre-trained linguistic model RoBERTa

Original: [23] to map the input textualexpression E o to a textual embedding space.
Revised: [27] to map the input textualexpression E o to a textual embedding space.

Original: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively.4.2 Early Grounding Frame
Revised: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively.

Original: CAM of 𝐟 𝐞𝐚𝐫𝐥𝐲 a black bear standing on a rock in

Original: [Figure]
Revised: [Figure]

Original: We propose an early grounding module to coarsely
Revised: We propose an early grounding module to coarsely

Original: [Figure]
Revised: [Figure]

Original: locate the referred object o and filter out irrelevant fea-
Revised: locate the referred object o and filter out irrelevant fea-

Original: [Figure]
Revised: [Figure]

Original: tures.
Revised: tures.

Original: Then the grounded feature f early encoding in-
Revised: Then the grounded feature f early encoding in-

Original: [Figure]
Revised: [Figure]

Original: formation of o can not only be utilized for the primarysegmentation problem, but also for the dual expressionreconstruction task, which serves as a medium connecting the two problems.
Revised: formation of o can not only be utilized for the primary [Figure] segmentation problem, but also for the dual expression [Figure]

Revised: reconstruction task, which serves as a proxy connecting

Revised: [Figure]

Revised: the two problems.

Original:  Specifically, we utilize the powerof dynamicconvolution [4] to discriminate visual features in the early stage.
Revised: Figure 4 shows a visualization of [Figure] f early . Specifically, we utilize the power of dynamic

Original: As shown in the blue part of
Revised: [Figure]

Revised: convolution [4] to discriminate visual features in theearly stage.

Original:  Figure 3, we first enable the multimodal interaction between video and text features, then apply thedynamic convolution with kernels learned from text feature to discriminate the object-level semantics.
Revised: As shown in the blue part of Figure 3, we first enable the multimodal interaction betweenvideo and text features, then apply the dynamic convolution with kernels learned from text featureto discriminate the object-level semantics.

Original: In particular, multi-head cross-attention (MCA) [38] is leveraged to conduct the multimodalinteraction:
Revised: In particular, multi-head cross-attention (MCA) [41] isleveraged to conduct the multimodal interaction:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where MCA( X , Y ) =
Revised: where MCA( X , Y ) =

Original: Attention( W Q X , W K Y , W V Y ) .
Revised: Attention( W Q X , W K Y , W V Y ) .

Original: W represents learnable weight.
Revised: W represents learnable weight.

Original: LNand FFN denote layer normalization and feed-forward network respectively.
Revised: LNand FFN denote layer normalization and feed-forward network respectively.

Original: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =
Revised: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =

Original: { θ i } Ki =1 .
Revised: { θ i } Ki =1 .

Original: K is the kernel number and θ i ∈ R C × 1 .
Revised: K is the kernel number and θ i ∈ R C × 1 .

Original: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW
Revised: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW

Original: [Equation]
Revised: [Equation]

Original: (6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.
Revised: 6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.

Original: BN denotes batch normalization.
Revised: BN denotes batch normalization.

Revised: 4.3 Text Reconstruction

Original: V-T projection.
Revised: V-T projection.

Original: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.
Revised: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.

Original: As shown in Figure 3, a learnable textquery e
Revised: As shown in Figure 3, a learnable textquery e

Original: 0 ∈ R C
Revised: 0 ∈ R C

Original: e × 1 is employed to query the f
Revised: e × 1 is employed to query the f

Original: early .
Revised: early .

Original: The output of the transformer decoder is thereconstructed text embedding e ′ =
Revised: The output of the transformer decoder is thereconstructed text embedding e ′ =

Original: D E ( f early  e 0 )
Revised: D E ( f early , e 0 )

Original:  R C e × 1 .
Revised: ∈ R C e × 1 .

Original: 4.4 Referring Segmentation
Revised: 4.4 Referring Segmentation

Original: Mask segmentation.
Revised: Mask segmentation.

Original: Similar to previous methods [41, 1, 8], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.
Revised: Similar to previous methods [44, 1, 11], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.

Original: As shown in Figure 3, we first fusethe reconstructed text embedding e ′ to text embedding e .
Revised: As shown in Figure 3, we first fuse the reconstructed text embedding e ′ to text embedding e .

Original: The fused text embedding e is then repeated
Revised: The fused text embedding e is then repeated

Original: N times to form the instance query [40] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.
Revised: N times to form the instance query [43] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.

Original: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .
Revised: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .

Original: F t is the multiscale visual feature from visual encoder at time t .
Revised: F t is the multiscale visual feature from visual encoder at time t .

Original: A dynamic kernel w t is furtherlearned from the instance embedding z t .
Revised: A dynamic kernel w t is furtherlearned from the instance embedding z t .

Original: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [18] in the visual decoder.
Revised: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [23] in the visual decoder.

Original: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .
Revised: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .

Original: Auxiliary heads.
Revised: Auxiliary heads.

Original: We build a set of auxiliary heads to obtain the final object masks across frames.
Revised: We build a set of auxiliary heads to obtain the final object masks across frames.

Original: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .
Revised: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .

Original: The box and scoring head are two fully-connected layers upon theinstance embedding e t .
Revised: The box and scoring head are two fully-connected layers upon theinstance embedding e t .

Original: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .
Revised: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .

Original: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.
Revised: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.

Original: 4.5 Loss Function
Revised: 4.5 Loss Function

Original: The loss function of our method can be boiled down to three parts:
Revised: The loss function of our method can be boiled down to three parts:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.
Revised: ) where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.

Revised: A ground-truth semantic alignment ˆ A

Revised: = { 0 , 1 } is utilizedto discriminate positive and negative pairs.

Original: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .
Revised: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .

Original: The other two terms are computed as follows:
Revised: The other two terms are computed as follows:

Original: Loss for text reconstruction.
Revised: Loss for text reconstruction.

Original: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .
Revised: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .

Original: We calculate theloss by
Revised: We calculate theloss by

Original: [Equation]
Revised: [Equation]

Original: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss
Revised: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss

Original: L angle .
Revised: L angle .

Original: We elaborate these two losses according to the relational cycle consistency Equation 2.
Revised: We elaborate these two losses according to the relational cycle consistency Equation 2.

Original: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )
Revised: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )

Original: | x ∈ X n , x ′ ∈
Revised: | x ∈ X n , x ′ ∈

Original: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and
Revised: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and

Original: X ′ .
Revised: X ′ .

Original: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ 

Revised: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ

Original: = { ( x , x ′ )
Revised: = { ( x , x ′ )

Original: | x = ( e i , e j ) , x ′ =
Revised: | x = ( e i , e j ) , x ′ =

Original: ( e ′ i , e ′ j ) , i ̸
Revised: ( e ′ i , e ′ j ) , i ̸

Original: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =
Revised: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =

Original: ( e ′ i , e ′ j , e ′ k ) , i ̸
Revised: ( e ′ i , e ′ j , e ′ k ) , i ̸

Original: =
Revised: =

Original: j ̸ = k } .
Revised: j ̸ = k } .

Original: Then the losses are given by:
Revised: Then the losses are given by:

Original: [Equation]
Revised: [Equation]

Original: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )
Revised: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )

Original: =
Revised: =

Original: 12 ( x − x ′ ) 

Revised: 12 ( x − x ′ )

Original: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .
Revised: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .

Original: Loss for referring segmentation.
Revised: Loss for referring segmentation.

Original: Given a set of predictions y
Revised: Given a set of predictions y

Original: =
Revised: =

Original: { y i } Ni =1 and ground-truth ˆy ,where y i =
Revised: { y i } Ni =1 and ground-truth ˆy ,where y i =

Original: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).
Revised: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).

Original: The similarity can be computed as
Revised: The similarity can be computed as

Original: [Equation]
Revised: [Equation]

Original: where λ box , λ conf , and λ mask are weights to balance losses.
Revised: where λ box , λ conf , and λ mask are weights to balance losses.

Original: Following previous works [5, 40], weleverage a combination of Dice [15] and BCE loss as L mask , focal loss [19] as L conf , and GIoU [35] and L1 loss as L box .
Revised: Following previous works [6, 43], weleverage a combination of Dice [20] and BCE loss as L mask , focal loss [24] as L conf , and GIoU[38] and L1 loss as L box .

Original: The best assignment ˆ σ is solved by Hungarian algorithm [11].
Revised: The best assignment ˆ σ is solved by Hungarian algorithm [16].

Original: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as
Revised: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as

Original: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .
Revised: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .

Original: 4.6 Inference
Revised: 4.6 Inference

Original: During inference, we select the candidate with the highest confidence to predict the final masks:
Revised: During inference, we select the candidate with the highest confidence to predict the final masks:

Original: [Equation]
Revised: [Equation]

Original: ) where { ¯M t } Tt =1 is the masks of referred object.
Revised: where { ¯M t } Tt =1 is the masks of referred object.

Original: S i,t and M i,t represent the i -th candidate in S t and
Revised: S i,t and M i,t represent the i -th candidate in S t and

Original: M t respectively.
Revised: M t respectively.

Original: ¯s is the slot with the highest confidence to be the target object.
Revised: ¯s is the slot with the highest confidence to be the target object.

Original: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm. {+1 ( A ) = 
+}
Revised: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm. {+1 ( A ) = 
+}

Revised: if A > 0 . 5 , else 0.

Original:  Experiment5.1 Dataset and Metrics
Revised:  Experiment

Revised: 5.1 Dataset and Metrics

Original: Dataset.
Revised: Dataset.

Original: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.
Revised: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.

Original: Ref-Youtube-VOS [36] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.
Revised: Ref-Youtube-VOS [39] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.

Original: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.
Revised: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.

Original: Ref-DAVIS-17 [10] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.
Revised: Ref-DAVIS-17 [13] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.

Original: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.
Revised: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.

Original: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.
Revised: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.

Original: Metrics.
Revised: Metrics.

Original: We employ commonly-used region similarity J and contour accuracy F [34] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.
Revised: We employ commonly-used region similarity J and contour accuracy F [36] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.

Original: For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 − (cid:80) M ∈M neg | M | (cid:80) M ∈M pos | M | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively.
Revised: For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 − [Equation] [Figure] [Equation]

Original:  | M | denotes the foreground area of mask M .
Revised: [Figure] [Equation] | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively. | M | denotes the foreground area of mask M .

Original: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.
Revised: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.

Original: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .
Revised: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .

Original: 5.2 Implementation Details
Revised: 5.2 Implementation Details

Original: Following previous methods [5, 41], our model is first pre-trained on Ref-COCO/+/g dataset [45, 27]and then finetuned on Ref-Youtube-VOS.
Revised: Following previous methods [6, 44], our model is first pre-trained on Ref-COCO/+/g dataset [49, 31]and then finetuned on Ref-Youtube-VOS.

Original: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.
Revised: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.

Original: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.
Revised: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.

Original: We adopt a batchsize of 8 and an AdamW [25] optimizerwith weight decay 1 × 10 − 4 .
Revised: We adopt a batchsize of 8 and an AdamW [29] optimizerwith weight decay 1 × 10 − 4 .

Original: Following convention [41, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.
Revised: Following convention [44, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.

Original: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels duringtraining and evaluation.
Revised: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels during evaluation.

Original: Thewindow size is set to 5.
Revised: The window size isset to 5 for all backbones.

Revised: We create negative pairs by shuffling positive pairs in each batch.

Original: Our method is implemented with PyTorch [32].
Revised: Ourmethod is implemented with PyTorch [34].

Original: 5.3 Main Results
Revised: 5.3 Main Results

Original: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.
Revised: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.

Original: Comparison on Ref-Youtube-VOS.
Revised: Comparison on Ref-Youtube-VOS.

Original: In Table 1, we first compare our method on Ref-Youtube-VOS.
Revised: In Table 1, we first compare our method on Ref-Youtube-VOS.

Original: For results of ResNet [7] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [41] by 1.7 J & F .
Revised: For results of ResNet [9] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [44] by 1.7 J & F .

Original: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).
Revised: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).

Original: Forresults of Swin-Transformer [24, 24] backbones, our method achieves 60.2 J & F and 61.3 J & Fwith Swin-Tiny and Video-Swin-Tiny backbones respectively, which outperforms ReferFormer [41]and MTTR [1] by a clear margin.
Revised: For

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison on Ref-DAVIS-17.

Original: Our

Original: 59.7 J & outperforms ReferFormer

Original: Comparison on R 2 -VOS.
Revised: Comparison on R 2 -VOS.

Original: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.
Revised: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.

Original: Compared to the severe false alarm of previous
Revised: Compared to the severe false alarm of previous

Original: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.
Revised: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.

Original: [Figure]
Revised: [Figure]

Original: Qualitative results.
Revised: Qualitative results.

Original: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.
Revised: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.

Original: For positive videos : The result indicates that our method predicts accurate andtemporal-consistent results, while ReferFormer [41] and MTTR [1] fail to locatethe correct object.
Revised: For positive videos : The result indicates that our method predicts accurate andtemporally-consistent results, while ReferFormer [44] and MTTR [1] fail to locatethe correct object.

Original: For negative videos :
Revised: For negative videos :

Original: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.
Revised: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.

Original: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.
Revised: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.

Original: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.
Revised: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.

Original: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.
Revised: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.

Original: 5.4 Ablation Study
Revised: 5.4 Ablation Study

Original: Module effectiveness.
Revised: Module effectiveness.

Original: To investigate the effectiveness of different components in our method, weconduct experiments with the ResNet-50 backbone on Ref-Youtube-VOS validation set.
Revised: To investigate the effectiveness of different components in our method,we conduct experiments with the ResNet-50 backbone on R 2 -Youtube-VOS dataset.

Original: We build atransformer-based baseline model and equip our proposed components step-by-step.
Revised: We build atransformer-based baseline model and equip our proposed components step-by-step.

Original: As shown in
Revised: As shown in

Original: Table 3, the baseline model achieves 52.4 J & F .
Revised: Table 3, the baseline model achieves 52.4 J & F .

Revised: Then, we add our proposed components step-bystep to demonstrate the module effectiveness.

Original: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.
Revised: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.

Original: Since the reconstructed text embedding is generated with visual features injected, we consider it can
Revised: Since the reconstructed text embedding is generated with visual features injected, we consider it can

Original: F r a m e
Revised: F r a m e

Original: M TT R
Revised: M TT R

Original: R e f e r F o r m e r
Revised: R e f e r F o r m e r

Original: O u r 
Revised: O u r s

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: encode some visual information, thus augmenting the original text embedding.
Revised: encode some visual information, thus augmenting the original text embedding.

Original: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .
Revised: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .

Original: Consistency constraint.
Revised: Consistency constraint.

Original: We conduct experiments to ablate the influence of cycle-consistencyconstraints.
Revised: We conduct experiments to ablate the influence of cycle-consistencyconstraints.

Original: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.
Revised: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.

Original: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.
Revised: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.

Original: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.
Revised: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.

Original: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.
Revised: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.

Original: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.
Revised: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.

Original: Instance query number.
Revised: Instance query number.

Original: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.
Revised: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.

Original: Table 5indicates that a query number of 5 brings the best result.
Revised: Table 5indicates that a query number of 5 brings the best result.

Original: Frame number.
Revised: Frame number.

Original: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporal-consistent results.
Revised: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporally-consistent results.

Original: We ablate on the bestwindow size of input videos during training.
Revised: We ablate on the bestwindow size of input videos during training.

Original: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .
Revised: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we investigate the semantic misalignment problem in R-VOS task.
Revised: In this paper, we investigate the semantic misalignment problem in R-VOS task.

Original: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.
Revised: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.

Original: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.
Revised: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.

Original: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.
Revised: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.

Original: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.
Revised: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.

Original: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation.
Revised: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (b) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you describe the limitations of your work?
Revised: (d) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: We are not including theoretical results.
Revised: We are not including theoretical results.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] The code areproprietary; most of our used dataset are public available; we plan to release code and
Revised: [No] The code are proprietary; most of our used dataset are public available; we plan to release code and

Original: novel dataset upon acceptance.
Revised: novel dataset upon acceptance.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No] We conduct several experiments multiple times and findthe results are very close.
Revised: [No] We conduct several experiments multiple times and find the results are very close.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]4.
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] The data we used are public available datasets for academic
Revised: [Yes] The data we used are public available datasets for academic

Original: purposes.
Revised: purposes.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [No] The dataset that we used does not contain such
Revised: [No] The dataset that we used does not contain such

Original: content.
Revised: content.

Original: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.
Revised: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.