Original: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency
Revised: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.
Revised: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.

Original: It is a recently introducedtask attracting growing research attention.
Revised: It is a recently introducedtask attracting growing research attention.

Original: However, all existing works make astrong assumption:
Revised: However, all existing works make astrong assumption:

Original: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.
Revised: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.

Original: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.
Revised: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.

Original: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.
Revised: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.

Original: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).
Revised: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).

Original: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).
Revised: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).

Original: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.
Revised: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.

Original: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.
Revised: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.

Original: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.
Revised: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.

Original: A new evaluation dataset,
Revised: A new evaluation dataset,

Original: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.
Revised: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.

Original: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.
Revised: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.

Original: Our model achieves the state-of-the-art performance on
Revised: Our model achieves the state-of-the-art performance on

Original: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.
Revised: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.

Original:  Introduction
Revised:  Introduction

Original: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.
Revised: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.

Original: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.
Revised: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.

Original: Unlikeother video segmentation tasks [45, 36, 35, 46] that only rely on visual cues, R-VOS [13] pairs atarget video with a linguistic expression referring to an object.
Revised: Unlikeother video segmentation tasks [42, 34, 33, 43] that only rely on visual cues, R-VOS [10] pairs atarget video with a linguistic expression referring to an object.

Original: Previous works [1, 44] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.
Revised: Previous works [1, 41] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.

Original: However, this assumption does not always hold in practice.
Revised: However, this assumption does not always hold in practice.

Original: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.
Revised: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.

Original: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.
Revised: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.

Original: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.

Original: Recently, MTTR [1]

Original: Query expression: “a kangaroo in the middle of two” others
Revised: Query expression: “a kangaroo in the middle of two” others

Original: [Figure]
Revised: [Figure]

Original: P o s iti v e 
Revised: v e [Figure] v

Original: [Figure]
Revised: [Figure]

Revised: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.

Original: employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.
Revised: Recently, MTTR [1]employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.

Original: ReferFormer[44] follows the image-level method, ReTR [19], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.
Revised: ReferFormer[41] follows the image-level method, ReTR [14], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.

Original: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.
Revised: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.

Original: As shown in Figure 1, the segmented objects by MTTR and
Revised: As shown in Figure 1, the segmented objects by MTTR and

Original: ReferFormer are often not the object referred to by the linguistic expression.
Revised: ReferFormer are often not the object referred to by the linguistic expression.

Original: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.
Revised: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.

Original: We extend the current task definition of R-VOS [13] toaccept both paired and unpaired video and language inputs.
Revised: We extend the current task definition of R-VOS [10] toaccept both paired and unpaired video and language inputs.

Original: This new task, which we term Robust
Revised: This new task, which we term Robust

Original: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.
Revised: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.

Original: We reveal that this task is essentiallyrelated to two problems that are interrelated [31]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.
Revised: We reveal that this task is essentiallyrelated to two problems that are interrelated [27]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.

Original: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.
Revised: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.

Original: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.
Revised: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.

Original: We address this problem by incorporating an earlygrounding scheme, serving as a proxy, to efficiently model the two tasks in a parallel manner.
Revised: We address this problem by incorporating an earlygrounding scheme, serving as a medium, to efficiently model the two tasks in a parallel manner.

Original: Inaddition, we discriminate the semantic misalignment between the video and text by assessing the cycleconsistency between the original and reconstructed texts, thus alleviating the false-alarm problem.
Revised: In addition, we discriminate the semantic misalignment between the video and text by assessingthe cycle consistency between the original and reconstructed texts, thus alleviating the false-alarmproblem.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.
Revised: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.

Original: To investigate the robustness of current referring segmentation models, we introduce the R 2 -VOS task that accepts unpaired video and text as inputs.•
Revised: To investigate the robustness of current referring segmentation models, we introducethe R 2 -VOS task that accepts unpaired video and text as inputs.

Original:  We propose a pipeline that jointly optimizes the primary referring segmentation and dualexpression reconstruction task and introduces a relational cycle consistency constraint toenhance the semantic alignment between visual and textual modalities.•
Revised: • We propose a pipeline that jointly optimizes the primary referring segmentation and dual expression reconstruction task and introduces a relational cycle consistency constraint to enhance the semantic alignment between visual and textual modalities.

Original:  Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS, and R 2 -Youtube-VOS dataset in terms of both performance and speed.
Revised: • Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS,and R 2 -Youtube-VOS dataset in terms of both performance and speed.

Original:  Related Works
Revised:  Related Works

Original: Vision and language representation learning.
Revised: Vision and language representation learning.

Original: There have been a long line of studies on how tolearn better vision-language representation, e.g., multimodal  attention [30, 50, 8, 3], fusion scheme[7, 14, 15, 51], multi-step reasoning [47, 10] and pretraining [37, 5, 17].
Revised: For vision-language representation learning, visualsemantic embedding (VSE) [6] and its improved versions [30, 13, 28, 3] have been proposed tofacilitate multimodal tasks, e.g., image-text matching. Recent works [21, 26, 3] introduce crossmodal attention layers and achieve superior improvement.

Original: KAC Net [2] leverages knowledge-aided consistency constraints to enhance semantic alignment for weakly supervised phrasegrounding.
Revised: KAC Net [2] leverages knowledge-aidedconsistency constraints to enhance semantic alignment for weakly supervised phrase grounding.

Original: A structure-preserving constraint [42] is proposed to preserve some intra-modal propertieswhen learning vision-language representation for image-text retrieval.
Revised: A structure-preserving constraint [39] is proposed to preserve some intra-modal properties whenlearning vision-language representation for image-text retrieval.

Original: Referring video object segmentation.
Revised: Referring video object segmentation.

Original: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.
Revised: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.

Original: URVOS [39] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.
Revised: URVOS [36] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.

Original: ClawCraneNet [21] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.
Revised: ClawCraneNet [16] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.

Original: ReferFormer [44] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.
Revised: ReferFormer [41] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.

Original: ReferFormer [44] employs a linguistic prior to the transformerdecoder to focus on the referred object.
Revised: ReferFormer [41] employs a linguistic prior to the transformerdecoder to focus on the referred object.

Original: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.
Revised: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.

Original: Different from other vision-language tasks, e.g., image-textretrieval [25, 26, 32] and video question answering [18, 40], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.
Revised: Different from other vision-language tasks, e.g., image-textretrieval [20, 22, 29] and video question answering [12, 37], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.

Original:  R 2 -VOS3.1 Task Definition
Revised:  R 2 -VOS3.1 Task Definition

Original: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .
Revised: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .

Original: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .
Revised: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .

Original: We define a video V and an expression E o to have semanticconsensus if the object
Revised: We define a video V and an expression E o to have semanticconsensus if the object

Original: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .
Revised: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .

Original: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.
Revised: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.

Original: 3.2 Problem Analysis
Revised: 3.2 Problem Analysis

Original: [Figure]
Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: We enable the joint optimization for cycle consistency with a cross-modal medium f m defined between all single-modal operations (i.e., Π encv , Π ence , Π decv and Π dece ).

Revised: Point-wise consistency is not suitable in R 2 -VOS because the mapping between E and E ′ are not necessarily bijective because an object can be referred to by various textual expressions.

Revised: (d) Instead, we apply a relational consistency to preserve distances and angles.

Original: Primary and dual problems for R 2 -VOS.
Revised: Primary and dual problems for R 2 -VOS.

Original: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .
Revised: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .

Original: By applying the Bayes rule, we obtain:
Revised: By applying the Bayes rule, we obtain:

Original: [Equation]
Revised: [Equation]

Original: As the prior p ( M o | V ) is not affected by the expression E o , we consider maximizing p ( E o | V, M o )as a dual problem of the referring segmentation (primary problem), which is to reconstruct the textexpression given the video and object masks.

Original: We note that for negative videos, p (E o | V, M o  )isundefined because the mask M o is empty. Thus, we only investigate the dual problem for positivevideos. The primary problem and the dual problem can be connected in a cycle path, i.e., from theoriginal expression E o to the reconstructed expression E ′ o
Revised: As the prior p ( M o | V ) is not affected

Original: through positive video queries, as shownin Figure 2 (a).

Original: We believe that the cycle constraint benefits to optimize the primary problem byenhancing the semantic consensus.

Original: Video 𝑉 = 𝐈 ! !"
Revised: Video 𝑉 = 𝐈 ! !"

Original: #$
Revised: #$

Original: Text exp.
Revised: Text exp.

Original: 𝐸 %
Revised: 𝐸 %

Original: Instance emb.
Revised: Instance emb.

Original: 𝐳 & &"#$
Revised: 𝐳 & &"#$

Original: Box 𝐁 ! !"
Revised: Box 𝐁 ! !"

Original: #$ Score 𝐒 ! !"
Revised: #$ Score 𝐒 ! !"

Original: #$
Revised: #$

Original: Visual Encoder
Revised: [Figure]

Original: Textual Encoder
Revised: Encoder

Original: Visual Decoder
Revised: Visual Decoder

Original: Seg.
Revised: Seg.

Original: 𝐌 ! !"
Revised: 𝐌 ! !"

Original: #$ a cat laying by an owl
Revised: #$ a cat laying by an owl

Original: Skip connections
Revised: Skip connections

Original: Pos. / Neg.
Revised: Pos. / Neg.

Original: R
Revised: R

Original: Dot Product
Revised: Dot Product

Original: Repeat R
Revised: Repeat R

Original: Concatenation C
Revised: Concatenation C

Original: Reconstructed text emb. 𝐞′
Revised: Reconstructed [Figure] emb

Revised: [Figure]

Original:  Relational CycleConsistency
Revised: 𝐞′ Relational Cycle

Revised: [Figure]

Original: Multi-scale feat.
Revised: Multi [Figure] scale feat.

Original: 𝐅 𝐅
Revised: 𝐅 𝐅

Original: Instance query 𝐳 ’
Revised: Instance query 𝐳 ’

Original: Sentence-level text emb.
Revised: Sentence- [Figure] emb.

Original: 𝐞
Revised: 𝐞

Original: C
Revised: C

Original: Conv Grounded feat.
Revised: Conv Grounded feat.

Original: 𝐟 ()*+,
Revised: 𝐟 ()*+,

Original: Dynamic kernel Θ 𝐠𝐟 - Video feat.
Revised: Dynamic kernel Θ 𝐠𝐟 - Video feat.

Original: 𝐟
Revised: 𝐟

Original: Word-level feat.
Revised: Word-level feat.

Original: 𝐠
Revised: 𝐠

Original: Multimodal Attention
Revised: [Figure]

Original: Transformer Decoder 𝒟 !
Revised: Transformer Decoder 𝒟 !

Revised: [Figure]

Original: Text query 𝐞 ’
Revised: query 𝐞 ’

Original: Early Grounding
Revised: Early Grounding

Original: V-T Projection Pool
Revised: V

Original: [Figure]
Revised: [Figure]

Original: Transformer Decoder
Revised: Projection

Original: [Figure]
Revised: [Figure]

Revised: ℋ because the is

Original: ℋ
Revised: , we only investigate the problem for The problem and the can be connected in a i. the o to the reconstructed E o through positive video queries as shown in Figure 2 (a). We that the cycle constraint benefits to optimize the primary problem by

Original: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textual embedding space E ′ induced by positive videos.
Revised: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textural embedding space E ′ induced by positive videos.

Original: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.
Revised: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.

Original: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal intermediate feature f m to convey information of boththeinput of the  primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).
Revised: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal medium f m to convey information of both the input ofthe primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).

Original: f m isdefined between the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv ,Π dece , to only focus on the multi-modal interaction.
Revised: f m is definedbetween the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv , Π dece , toonly focus on the multi-modal interaction.

Original: Relational cycle consistency.
Revised: Relational cycle consistency.

Original: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.
Revised: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.

Original: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.
Revised: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.

Original: Instead, we take inspiration from relational knowledgedistillation [33], and introduce relational cycle consistency for E and E ′ .
Revised: Instead, we take inspiration from relational knowledgedistillation [31], and introduce relational cycle consistency for E and E ′ .

Original: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).
Revised: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).

Original: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e
Revised: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e

Original: ′ ∈ E ′ :
Revised: ′ ∈ E ′ :

Original: [Equation]
Revised: [Equation]

Original:  where | · | and ∠ ( · ) denote distance and angle metrics.
Revised: ) where | · | and ∠ ( · ) denote distance and angle metrics.

Original:  Method
Revised:  Method

Original: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.
Revised: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.

Original: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .
Revised: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .

Original: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .
Revised: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .

Original: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.
Revised: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.

Original: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instanceembedding { z t } Tt =1 .
Revised: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instance embedding { z t } Tt =1 .

Original: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f
Revised: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f

Original: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .
Revised: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .

Original: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .
Revised: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .

Original: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .
Revised: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .

Original: We add relational constraint between e ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.
Revised: We add relational constraint betweene ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.

Original: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.
Revised: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.

Original: 4.1 Single-modal Feature Extraction
Revised: 4.1 Single-modal Feature Extraction

Original: [Figure]

Original: Visual encoder.
Revised: Visual encoder.

Original: Following previous methods [1, 44, 43], we build the visual encoder with a visualbackbone and a deformable transformer encoder [52] on top of it.
Revised: Following previous methods [1, 41, 40], we build the visual encoder with a visualbackbone and a deformable transformer encoder [46] on top of it.

Original: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [12], andthen fed into a deformable transformer encoder [52] similar to the previous method [44].
Revised: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [9], and thenfed into a deformable transformer encoder [46] similar to the previous method [41].

Original: We denotethe multi-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.
Revised: We denote themulti-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.

Original: Textual encoder.
Revised: Textual encoder.

Original: We leverage a pre-trained linguistic model RoBERTa
Revised: We leverage a pre-trained linguistic model RoBERTa

Original: [27] to map the input textualexpression E o to a textual embedding space.
Revised: [23] to map the input textualexpression E o to a textual embedding space.

Original: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively. 
Revised: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively. 4.2 Early Grounding Frame

Revised: CAM of 𝐟 𝐞𝐚𝐫𝐥𝐲 a black bear standing on a rock in

Original: [Figure]
Revised: [Figure]

Original: We propose an early grounding module to coarsely
Revised: We propose an early grounding module to coarsely

Original: [Figure]
Revised: [Figure]

Original: locate the referred object o and filter out irrelevant fea-
Revised: locate the referred object o and filter out irrelevant fea-

Original: [Figure]
Revised: [Figure]

Original: tures.
Revised: tures.

Original: Then the grounded feature f early encoding in-
Revised: Then the grounded feature f early encoding in-

Original: [Figure]
Revised: [Figure]

Original: formation of o can not only be utilized for the primary [Figure] segmentation problem, but also for the dual expression [Figure]
Revised: formation of o can not only be utilized for the primarysegmentation problem, but also for the dual expressionreconstruction task, which serves as a medium connecting the two problems.

Original: reconstruction task, which serves as a proxy connecting

Original: [Figure]

Original: the two problems.

Original: Figure 4 shows a visualization of

Original: [Figure]

Original: f early . Specifically, we utilize the power of dynamic[Figure] convolution [4] to discriminate visual features in theearly stage.
Revised: Specifically, we utilize the powerof dynamic convolution [4] to discriminate visual features in the early stage.

Original: As shown in the blue part of Figure 3, we first enable the multimodal interaction betweenvideo and text features, then apply the dynamic convolution with kernels learned from text featureto discriminate the object-level semantics.
Revised: As shown in the blue part of Figure 3, we first enable the multimodal interaction between video and text features, then apply thedynamic convolution with kernels learned from text feature to discriminate the object-level semantics.

Original: In particular, multi-head cross-attention (MCA) [41] isleveraged to conduct the multimodal interaction:
Revised: In particular, multi-head cross-attention (MCA) [38] is leveraged to conduct the multimodalinteraction:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where MCA( X , Y ) =
Revised: where MCA( X , Y ) =

Original: Attention( W Q X , W K Y , W V Y ) .
Revised: Attention( W Q X , W K Y , W V Y ) .

Original: W represents learnable weight.
Revised: W represents learnable weight.

Original: LNand FFN denote layer normalization and feed-forward network respectively.
Revised: LNand FFN denote layer normalization and feed-forward network respectively.

Original: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =
Revised: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =

Original: { θ i } Ki =1 .
Revised: { θ i } Ki =1 .

Original: K is the kernel number and θ i ∈ R C × 1 .
Revised: K is the kernel number and θ i ∈ R C × 1 .

Original: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW
Revised: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW

Original: [Equation]
Revised: [Equation]

Original: 6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.
Revised: (6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.

Original: BN denotes batch normalization.
Revised: BN denotes batch normalization.

Original: 4.3 Text Reconstruction

Original: V-T projection.
Revised: V-T projection.

Original: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.
Revised: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.

Original: As shown in Figure 3, a learnable textquery e
Revised: As shown in Figure 3, a learnable textquery e

Original: 0 ∈ R C
Revised: 0 ∈ R C

Original: e × 1 is employed to query the f
Revised: e × 1 is employed to query the f

Original: early .
Revised: early .

Original: The output of the transformer decoder is thereconstructed text embedding e ′ =
Revised: The output of the transformer decoder is thereconstructed text embedding e ′ =

Original: D E ( f early, e 0 )
Revised: D E ( f early e 0 )

Original: ∈ R C e × 1 .
Revised: R C e × 1 .

Original: 4.4 Referring Segmentation
Revised: 4.4 Referring Segmentation

Original: Mask segmentation.
Revised: Mask segmentation.

Original: Similar to previous methods [44, 1, 11], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.
Revised: Similar to previous methods [41, 1, 8], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.

Original: As shown in Figure 3, we first fuse the reconstructed text embedding e ′ to text embedding e .
Revised: As shown in Figure 3, we first fusethe reconstructed text embedding e ′ to text embedding e .

Original: The fused text embedding e is then repeated
Revised: The fused text embedding e is then repeated

Original: N times to form the instance query [43] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.
Revised: N times to form the instance query [40] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.

Original: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .
Revised: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .

Original: F t is the multiscale visual feature from visual encoder at time t .
Revised: F t is the multiscale visual feature from visual encoder at time t .

Original: A dynamic kernel w t is furtherlearned from the instance embedding z t .
Revised: A dynamic kernel w t is furtherlearned from the instance embedding z t .

Original: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [23] in the visual decoder.
Revised: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [18] in the visual decoder.

Original: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .
Revised: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .

Original: Auxiliary heads.
Revised: Auxiliary heads.

Original: We build a set of auxiliary heads to obtain the final object masks across frames.
Revised: We build a set of auxiliary heads to obtain the final object masks across frames.

Original: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .
Revised: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .

Original: The box and scoring head are two fully-connected layers upon theinstance embedding e t .
Revised: The box and scoring head are two fully-connected layers upon theinstance embedding e t .

Original: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .
Revised: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .

Original: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.
Revised: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.

Original: 4.5 Loss Function
Revised: 4.5 Loss Function

Original: The loss function of our method can be boiled down to three parts:[Equation]
Revised: The loss function of our method can be boiled down to three parts:

Original: [Equation]
Revised: [Equation]

Original: ) where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.A ground-truth semantic alignment ˆ A
Revised: where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.

Original: = { 0 , 1 } is utilizedto discriminate positive and negative pairs.

Original: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .
Revised: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .

Original: The other two terms are computed as follows:
Revised: The other two terms are computed as follows:

Original: Loss for text reconstruction.
Revised: Loss for text reconstruction.

Original: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .
Revised: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .

Original: We calculate theloss by
Revised: We calculate theloss by

Original: [Equation]
Revised: [Equation]

Original: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss
Revised: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss

Original: L angle .
Revised: L angle .

Original: We elaborate these two losses according to the relational cycle consistency Equation 2.
Revised: We elaborate these two losses according to the relational cycle consistency Equation 2.

Original: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )
Revised: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )

Original: | x ∈ X n , x ′ ∈
Revised: | x ∈ X n , x ′ ∈

Original: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and
Revised: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and

Original: X ′ .
Revised: X ′ .

Original: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ 

Revised: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ

Original: = { ( x , x ′ )
Revised: = { ( x , x ′ )

Original: | x = ( e i , e j ) , x ′ =
Revised: | x = ( e i , e j ) , x ′ =

Original: ( e ′ i , e ′ j ) , i ̸
Revised: ( e ′ i , e ′ j ) , i ̸

Original: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =
Revised: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =

Original: ( e ′ i , e ′ j , e ′ k ) , i ̸
Revised: ( e ′ i , e ′ j , e ′ k ) , i ̸

Original: =
Revised: =

Original: j ̸ = k } .
Revised: j ̸ = k } .

Original: Then the losses are given by:
Revised: Then the losses are given by:

Original: [Equation]
Revised: [Equation]

Original: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )
Revised: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )

Original: =
Revised: =

Original: 12 ( x − x ′ ) 

Revised: 12 ( x − x ′ )

Original: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .
Revised: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .

Original: Loss for referring segmentation.
Revised: Loss for referring segmentation.

Original: Given a set of predictions y
Revised: Given a set of predictions y

Original: =
Revised: =

Original: { y i } Ni =1 and ground-truth ˆy ,where y i =
Revised: { y i } Ni =1 and ground-truth ˆy ,where y i =

Original: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).
Revised: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).

Original: The similarity can be computed as
Revised: The similarity can be computed as

Original: [Equation]
Revised: [Equation]

Original: where λ box , λ conf , and λ mask are weights to balance losses.
Revised: where λ box , λ conf , and λ mask are weights to balance losses.

Original: Following previous works [6, 43], weleverage a combination of Dice [20] and BCE loss as L mask , focal loss [24] as L conf , and GIoU[38] and L1 loss as L box .
Revised: Following previous works [5, 40], weleverage a combination of Dice [15] and BCE loss as L mask , focal loss [19] as L conf , and GIoU [35] and L1 loss as L box .

Original: The best assignment ˆ σ is solved by Hungarian algorithm [16].
Revised: The best assignment ˆ σ is solved by Hungarian algorithm [11].

Original: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as
Revised: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as

Original: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .
Revised: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .

Original: 4.6 Inference
Revised: 4.6 Inference

Original: During inference, we select the candidate with the highest confidence to predict the final masks:
Revised: During inference, we select the candidate with the highest confidence to predict the final masks:

Original: [Equation]
Revised: [Equation]

Original:  where { ¯M t } Tt =1 is the masks of referred object.
Revised: ) where { ¯M t } Tt =1 is the masks of referred object.

Original: S i,t and M i,t represent the i -th candidate in S t and
Revised: S i,t and M i,t represent the i -th candidate in S t and

Original: M t respectively.
Revised: M t respectively.

Original: ¯s is the slot with the highest confidence to be the target object.
Revised: ¯s is the slot with the highest confidence to be the target object.

Original: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm.[-1 ( A ) = 
-]
Revised: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm.[-1 ( A ) = 
-]

Original: if A > 0 . 5 , else 0.

Original:  Experiment
Revised:  Experiment5.1 Dataset and Metrics

Original: 5.1 Dataset and Metrics

Original: Dataset.
Revised: Dataset.

Original: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.
Revised: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.

Original: Ref-Youtube-VOS [39] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.
Revised: Ref-Youtube-VOS [36] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.

Original: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.
Revised: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.

Original: Ref-DAVIS-17 [13] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.
Revised: Ref-DAVIS-17 [10] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.

Original: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.
Revised: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.

Original: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.
Revised: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.

Original: Metrics.
Revised: Metrics.

Original: We employ commonly-used region similarity J and contour accuracy F [36] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 − [Equation] [Figure]
Revised: We employ commonly-used region similarity J and contour accuracy F [34] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.

Original: [Equation] [Figure] [Equation] | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively.
Revised: For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 − (cid:80) M ∈M neg | M | (cid:80) M ∈M pos | M | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively.

Original: | M | denotes the foreground area of mask M .
Revised: | M | denotes the foreground area of mask M .

Original: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.
Revised: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.

Original: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .
Revised: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .

Original: 5.2 Implementation Details
Revised: 5.2 Implementation Details

Original: Following previous methods [6, 44], our model is first pre-trained on Ref-COCO/+/g dataset [49, 31]and then finetuned on Ref-Youtube-VOS.
Revised: Following previous methods [5, 41], our model is first pre-trained on Ref-COCO/+/g dataset [45, 27]and then finetuned on Ref-Youtube-VOS.

Original: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.
Revised: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.

Original: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.
Revised: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.

Original: We adopt a batchsize of 8 and an AdamW [29] optimizerwith weight decay 1 × 10 − 4 .
Revised: We adopt a batchsize of 8 and an AdamW [25] optimizerwith weight decay 1 × 10 − 4 .

Original: Following convention [44, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.
Revised: Following convention [41, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.

Original: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels during  evaluation.
Revised: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels during training and evaluation.

Original: The window size isset to 5 for all backbones.
Revised: Thewindow size is set to 5.

Original: We create negative pairs by shuffling positive pairs in each batch.

Original: Ourmethod is implemented with PyTorch [34].
Revised: Our method is implemented with PyTorch [32].

Original: 5.3 Main Results
Revised: 5.3 Main Results

Original: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.
Revised: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.

Original: Comparison on Ref-Youtube-VOS.
Revised: Comparison on Ref-Youtube-VOS.

Original: In Table 1, we first compare our method on Ref-Youtube-VOS.
Revised: In Table 1, we first compare our method on Ref-Youtube-VOS.

Original: For results of ResNet [9] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [44] by 1.7 J & F .
Revised: For results of ResNet [7] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [41] by 1.7 J & F .

Original: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).
Revised: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).

Original: For 
Revised: For results of Swin-Transformer [24, 24] backbones, our method achieves 60.2 J & F and 61.3 J & Fwith Swin-Tiny and Video-Swin-Tiny backbones respectively, which outperforms ReferFormer [41]and MTTR [1] by a clear margin.

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Comparison on Ref-DAVIS-17.

Revised: Our

Revised: 59.7 J & outperforms ReferFormer

Original: Comparison on R 2 -VOS.
Revised: Comparison on R 2 -VOS.

Original: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.
Revised: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.

Original: Compared to the severe false alarm of previous
Revised: Compared to the severe false alarm of previous

Original: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.
Revised: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.

Original: [Figure]
Revised: [Figure]

Original: Qualitative results.
Revised: Qualitative results.

Original: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.
Revised: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.

Original: For positive videos : The result indicates that our method predicts accurate andtemporally-consistent results, while ReferFormer [44] and MTTR [1] fail to locatethe correct object.
Revised: For positive videos : The result indicates that our method predicts accurate andtemporal-consistent results, while ReferFormer [41] and MTTR [1] fail to locatethe correct object.

Original: For negative videos :
Revised: For negative videos :

Original: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.
Revised: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.

Original: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.
Revised: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.

Original: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.
Revised: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.

Original: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.
Revised: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.

Original: 5.4 Ablation Study
Revised: 5.4 Ablation Study

Original: Module effectiveness.
Revised: Module effectiveness.

Original: To investigate the effectiveness of different components in our method,we conduct experiments with the ResNet-50 backbone on R 2 -Youtube-VOS dataset.
Revised: To investigate the effectiveness of different components in our method, weconduct experiments with the ResNet-50 backbone on Ref-Youtube-VOS validation set.

Original: We build atransformer-based baseline model and equip our proposed components step-by-step.
Revised: We build atransformer-based baseline model and equip our proposed components step-by-step.

Original: As shown in
Revised: As shown in

Original: Table 3, the baseline model achieves 52.4 J & F .
Revised: Table 3, the baseline model achieves 52.4 J & F .

Original: Then, we add our proposed components step-bystep to demonstrate the module effectiveness.

Original: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.
Revised: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.

Original: Since the reconstructed text embedding is generated with visual features injected, we consider it can
Revised: Since the reconstructed text embedding is generated with visual features injected, we consider it can

Original: F r a m e
Revised: F r a m e

Original: M TT R
Revised: M TT R

Original: R e f e r F o r m e r
Revised: R e f e r F o r m e r

Original: O u rs
Revised: O u r

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: encode some visual information, thus augmenting the original text embedding.
Revised: encode some visual information, thus augmenting the original text embedding.

Original: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .
Revised: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .

Original: Consistency constraint.
Revised: Consistency constraint.

Original: We conduct experiments to ablate the influence of cycle-consistencyconstraints.
Revised: We conduct experiments to ablate the influence of cycle-consistencyconstraints.

Original: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.
Revised: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.

Original: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.
Revised: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.

Original: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.
Revised: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.

Original: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.
Revised: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.

Original: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.
Revised: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.

Original: Instance query number.
Revised: Instance query number.

Original: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.
Revised: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.

Original: Table 5indicates that a query number of 5 brings the best result.
Revised: Table 5indicates that a query number of 5 brings the best result.

Original: Frame number.
Revised: Frame number.

Original: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporally-consistent results.
Revised: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporal-consistent results.

Original: We ablate on the bestwindow size of input videos during training.
Revised: We ablate on the bestwindow size of input videos during training.

Original: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .
Revised: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we investigate the semantic misalignment problem in R-VOS task.
Revised: In this paper, we investigate the semantic misalignment problem in R-VOS task.

Original: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.
Revised: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.

Original: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.
Revised: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.

Original: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.
Revised: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.

Original: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.
Revised: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.

Original: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation.

Original: A Additional AppendixA.1 More Quantitative Result Analysis

Original: Under the same ResNet-50 backbone, our method achieves 57.3 J & F , 94.1 R and 30 FPS comparedto the 55.6 J & F , 30.6 R and 22 FPS of ReferFormer.

Original: We will then point-to-point analyze reasons ofimprovements on J & F (for positive video), R (for negative videos) and FPS (for inference speed).

Original: • J & F : (1) We introduce the early-grounding module which employs both pixel-wise andchannel-wise attention to enable multimodal interaction.

Original: Different from the CM-FPN used inReferFormer that solely fuses features from text to video in pixel-level, our early-groundingmodule first enables pixel-level bi-directional fusion and then generates dynamic kernels

Original: using the fused text feature g ′ to modulate the video feature f ′ .

Original: The dynamic convolution(channel-wise attention) is commonly used to decode dense masks from visual features and

Original: is suitable to suppress irrelevant features.

Original: By equipping text-guided dynamic convolution in

Original: early-stage, the pixel decoder can be more focused on the target object (as shown in Figure 4).

Original: (2) Our method leverages relational cycle consistency to constraint the intermediate feature

Original: f early to contain correct object-level information to recover some properties of original textembedding.

Original: By applying this constraint, our method can better avoid interference and easierlocate the correct object.

Original: Our instance query is composed of both original sentenceembedding and the reconstructed one.

Original: Different from ReferFormer that only utilizes original

Original: sentence embedding as queries, the reconstructed embedding can encode visual information

Original: to facilitate the instance query decode the objects from visual features.

Original: • R : The newly introduced metric R aims to measure the robustness of the model against

Original: unpaired inputs.

Original: Text-video pairs with(object-level) semantic consensus can be assumed as in-distribution for RVOS problem where semantic consensus can be kind of easily modeled. In contrast, unpaired text-video is much more difficult to tackle because there can 
Revised: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation.

Original: be unlimited

Original: out-of-distribution (OOD) scenarios for the text-video pairs.

Original: In our method,instead of directly detect the OOD of input pairs, we convert the problem to find semanticalignment between the input text embedding and reconstructed embedding and constraintthe property of reconstructed space by introducing the cycle consistency.

Original: In this way,the comparison is conducted in the constraint original and reconstructed text spaces.

Original: For

Original: ReferFormer, it models the alignment of text to video by querying the visual features by textin the transformer decoder.

Original: In this way, the comparison is conducted in unconstrained textand video spaces thus results in a inferior performance.•

Original: FPS

Original: : The speed improvement of our method mainly comes from our efficient multimodalfusion.

Original: Compared to the multi-scale CM-FPN, our early-grounding module is only conductat the high-level.

Original: In addition, our bi-direction multimodal fusion (Equ 4 & 5) only leveragescross-attention to avoid computational expensive video-to-video operations.

Original: A.2 Limitations

Original: An important challenge for video segmentation is that target object disappearance due to occlusion,which can results in false positives on a per-frame level.

Original: In our method, we predict the video-levelsemantic alignment to handle the false positive in video-level resulted from unpaired text-video pairs.

Original: However, since only video-level object expression is available in RVOS task, our method can notaddress the frame-level false positives resulted from occlusion.

Original: A.3 Additional Experiment on Negative Videos without Positive Text

Original: [Table]

Original: As shown in Table A, we test the robustness of our model on two settings.

Original: We generate negativevideos from Ref-Youtube-VOS and a combination of Ref-Youtube-VOS and Ref-DAVIS dataset.

Original: In both settings, all videos in the validation set are leveraged.

Original: The results indicates that source ofnegative videos has minor impact on the robustness of our model.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (b) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you describe the limitations of your work?
Revised: (d) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: We are not including theoretical results.
Revised: We are not including theoretical results.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] The code are proprietary; most of our used dataset are public available; we plan to release code and
Revised: [No] The code areproprietary; most of our used dataset are public available; we plan to release code and

Original: novel dataset upon acceptance.
Revised: novel dataset upon acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?

Original: [No] We conduct several experiments multiple times and find the results are very close.
Revised: [No] We conduct several experiments multiple times and findthe results are very close.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] The data we used are public available datasets for academic
Revised: [Yes] The data we used are public available datasets for academic

Original: purposes.
Revised: purposes.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No] The dataset that we used does not contain such
Revised: [No] The dataset that we used does not contain such

Original: content.
Revised: content.

Original: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.
Revised: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.