Original: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency
Revised: R 2 - VOS : Robust Referring Video Object Segmentation via Relational Cycle Consistency

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.
Revised: Referring video object segmentation (R-VOS) aims to segment the object masks in avideo given a referring linguistic expression to the object.

Original: It is a recently introducedtask attracting growing research attention.
Revised: It is a recently introducedtask attracting growing research attention.

Original: However, all existing works make astrong assumption:
Revised: However, all existing works make astrong assumption:

Original: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.
Revised: The object depicted by the expression must exist in the video,namely, the expression and video must have an object-level semantic consensus.

Original: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.
Revised: This is often violated in real-world applications where an expression can be queriedto false videos, and existing methods always fail in such false queries due to abusingthe assumption.

Original: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.
Revised: In this work, we emphasize that studying semantic consensus isnecessary to improve the robustness of R-VOS.

Original: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).
Revised: Accordingly, we pose an extendedtask from R-VOS without the semantic consensus assumption, named Robust RVOS ( R 2 -VOS).

Original: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).
Revised: The R 2 -VOS task is essentially related to the joint modeling of theprimary R-VOS task and its dual problem (text reconstruction).

Original: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.
Revised: We embrace theobservation that the embedding spaces have relational consistency through the cycleof text-video-text transformation, which connects the primary and dual problems.

Original: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.
Revised: We leverage the cycle consistency to discriminate the semantic consensus, thusadvancing the primary task.

Original: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.
Revised: Parallel optimization of the primary and dual problemsare enabled by introducing an early grounding medium.

Original: A new evaluation dataset,
Revised: A new evaluation dataset,

Original: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.
Revised: R 2 -Youtube-VOS, is collected to measure the robustness of R-VOS models againstunpaired videos and expressions.

Original: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.
Revised: Extensive experiments demonstrate that ourmethod not only identifies negative pairs of unrelated expressions and videos,but also improves the segmentation accuracy for positive pairs with a superiordisambiguating ability.

Original: Our model achieves the state-of-the-art performance on
Revised: Our model achieves the state-of-the-art performance on

Original: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.
Revised: Ref-DAVIS17, Ref-Youtube-VOS, and the novel R 2 -Youtube-VOS dataset.

Original:  Introduction
Revised:  Introduction

Original: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.
Revised: Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequencegiven a linguistic expression.

Original: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.
Revised: R-VOS has witnessed growing interest thanks to its promising potentialin human-computer interaction applications such as video editing and augmented reality.

Original: Unlikeother video segmentation tasks [45, 36, 35, 46] that only rely on visual cues, R-VOS [13] pairs atarget video with a linguistic expression referring to an object.
Revised: Unlikeother video segmentation tasks [45, 36, 35, 46] that only rely on visual cues, R-VOS [13] pairs atarget video with a linguistic expression referring to an object.

Original: Previous works [1, 44] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.
Revised: Previous works [1, 44] tackle the R-VOS problem with a strong assumption that the referred objectexists in the video, i.e., there is an object-level semantic consensus between the expression and thevideo.

Original: However, this assumption does not always hold in practice.
Revised: However, this assumption does not always hold in practice.

Original: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.
Revised: As shown in Figure 1, we notice asevere false-alarm problem experienced by previous methods when the semantic consensus does notexist, which may prevent those methods from being useful in various applications that cannot provideaccurate vision-language pairs.

Original: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.
Revised: We argue that the current R-VOS task is not completely defined withthe assumption that the referred object always exists in the video.

Original: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.
Revised: Even when semantic consensus exists in the given video-language pairs, it is still challenging to locatethe correct object in the video due to the multimodal nature of the R-VOS task.

Original: Recently, MTTR [1]
Revised: Recently, MTTR [1]

Original: Query expression: “a kangaroo in the middle of two” others
Revised: Query expression: “a kangaroo in the middle of two” others

Original: [Figure]
Revised: [Figure]

Original: P o s iti v e
Revised: P o s iti v e

Original: [Figure]
Revised: [Figure]

Original: employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.
Revised: employs a multimodal transformer encoder to learn a joint representation of the linguistic expressionand video, and then obtains the referred object by ranking all objects in the video.

Original: ReferFormer[44] follows the image-level method, ReTR [19], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.
Revised: ReferFormer[44] follows the image-level method, ReTR [19], to adopt the linguistic expression as a query tothe transformer decoder to avoid redundant ranking of all objects.

Original: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.
Revised: However, these latest methodssuffer from semantic misalignment of the segmented object and the linguistic expression, even withsophisticated components employed.

Original: As shown in Figure 1, the segmented objects by MTTR and
Revised: As shown in Figure 1, the segmented objects by MTTR and

Original: ReferFormer are often not the object referred to by the linguistic expression.
Revised: ReferFormer are often not the object referred to by the linguistic expression.

Original: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.
Revised: In this paper, we seek to investigate the semantic alignment problem between visual and linguisticmodalities in referring video segmentation.

Original: We extend the current task definition of R-VOS [13] toaccept both paired and unpaired video and language inputs.
Revised: We extend the current task definition of R-VOS [13] toaccept both paired and unpaired video and language inputs.

Original: This new task, which we term Robust
Revised: This new task, which we term Robust

Original: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.
Revised: R-VOS ( R 2 -VOS), overcomes the current limitation of the R-VOS task by additionally consideringthe semantic alignment of input video to referring expression.

Original: We reveal that this task is essentiallyrelated to two problems that are interrelated [31]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.
Revised: We reveal that this task is essentiallyrelated to two problems that are interrelated [31]: the R-VOS problem as the primary problem ofsegmenting mask sequences from videos with referring texts, and its dual problem of reconstructingtext expressions from videos with object masks.

Original: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.
Revised: By linking the primary and dual problems, weintroduce a text-video-text cycle and a corresponding relational consistency constraint, which canenforce the semantic consensus between the text query and segmented mask to improve the primarytask.

Original: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.
Revised: In practice, naively conducting cyclic training of the text-video-text cycle will lead to a twostage regime and significantly increasing costs.

Original: We address this problem by incorporating an earlygrounding scheme, serving as a proxy, to efficiently model the two tasks in a parallel manner.
Revised: We address this problem by incorporating an earlygrounding scheme, serving as a proxy, to efficiently model the two tasks in a parallel manner.

Original: Inaddition, we discriminate the semantic misalignment between the video and text by assessing the cycleconsistency between the original and reconstructed texts, thus alleviating the false-alarm problem.
Revised: Inaddition, we discriminate the semantic misalignment between the video and text by assessing the cycleconsistency between the original and reconstructed texts, thus alleviating the false-alarm problem.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.
Revised: • We notice a severe false-alarm problem faced by previous R-VOS methods with unpairedinputs.

Original: To investigate the robustness of current referring segmentation models, we introduce
Revised: To investigate the robustness of current referring segmentation models, we introduce

Original: the R 2 -VOS task that accepts unpaired video and text as inputs.•
Revised: the R 2 -VOS task that accepts unpaired video and text as inputs.•

Original: We propose a pipeline that jointly optimizes the primary referring segmentation and dualexpression reconstruction task and introduces a relational cycle consistency constraint toenhance the semantic alignment between visual and textual modalities.•
Revised: We propose a pipeline that jointly optimizes the primary referring segmentation and dualexpression reconstruction task and introduces a relational cycle consistency constraint toenhance the semantic alignment between visual and textual modalities.•

Original: Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS,
Revised: Our method surpasses previous state-of-the-art methods on Ref-Youtube-VOS, Ref-DAVIS,

Original: and R 2 -Youtube-VOS dataset in terms of both performance and speed.
Revised: and R 2 -Youtube-VOS dataset in terms of both performance and speed.

Original:  Related Works
Revised:  Related Works

Original: Vision and language representation learning.
Revised: Vision and language representation learning.

Original: There have been a long line of studies on how tolearn better vision-language representation, e.g., multimodal attention [30, 50, 8, 3], fusion scheme[7, 14, 15, 51], multi-step reasoning [47, 10] and pretraining [37, 5, 17].
Revised: There have been a long line of studies on how tolearn better vision-language representation, e.g., multimodal attention [30, 50, 8, 3], fusion scheme[7, 14, 15, 51], multi-step reasoning [47, 10] and pretraining [37, 5, 17].

Original: KAC Net [2] leverages knowledge-aided consistency constraints to enhance semantic alignment for weakly supervised phrasegrounding.
Revised: KAC Net [2] leverages knowledge-aided consistency constraints to enhance semantic alignment for weakly supervised phrasegrounding.

Original: A structure-preserving constraint [42] is proposed to preserve some intra-modal propertieswhen learning vision-language representation for image-text retrieval.
Revised: A structure-preserving constraint [42] is proposed to preserve some intra-modal propertieswhen learning vision-language representation for image-text retrieval.

Original: Referring video object segmentation.
Revised: Referring video object segmentation.

Original: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.
Revised: R-VOS is a novel task that aims to segment an object acrossframes given a linguistic description.

Original: URVOS [39] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.
Revised: URVOS [39] is the first unified R-VOS framework with across-modal attention and a memory attention module, which largely improves R-VOS performance.

Original: ClawCraneNet [21] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.
Revised: ClawCraneNet [21] leverages cross-modal attention to bridge the semantic correlation between textualand visual modalities.

Original: ReferFormer [44] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.
Revised: ReferFormer [44] and MTTR [1] are two latest works that utilize transformersto decode or fuse multimodal features.

Original: ReferFormer [44] employs a linguistic prior to the transformerdecoder to focus on the referred object.
Revised: ReferFormer [44] employs a linguistic prior to the transformerdecoder to focus on the referred object.

Original: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.
Revised: MTTR [1] leverages a multimodal transformer encoderto fuse linguistic and visual features.

Original: Different from other vision-language tasks, e.g., image-textretrieval [25, 26, 32] and video question answering [18, 40], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.
Revised: Different from other vision-language tasks, e.g., image-textretrieval [25, 26, 32] and video question answering [18, 40], R-VOS needs to construct object-levelmultimodal semantic consensus in a dense visual representation.

Original:  R 2 -VOS3.1 Task Definition
Revised:  R 2 -VOS3.1 Task Definition

Original: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .
Revised: We introduce a novel task, robust referring video segmentation ( R 2 -VOS), which aims to predictmask sequences { M o } for an unconstrained video set { V } given a language expression E o of anobject o .

Original: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .
Revised: Different from the previous R-VOS setup, the queried video V is not required to containthe referred object by expression E o .

Original: We define a video V and an expression E o to have semanticconsensus if the object
Revised: We define a video V and an expression E o to have semanticconsensus if the object

Original: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .
Revised: o appears in V , and the video is positive with respect to E o , otherwise it isnegative .

Original: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.
Revised: The R 2 -VOS task is extended to discriminate positive and negative videos, and predictmasks M o of object o for positive videos and treat all frames in the negative videos as background.

Original: 3.2 Problem Analysis
Revised: 3.2 Problem Analysis

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Primary and dual problems for R 2 -VOS.
Revised: Primary and dual problems for R 2 -VOS.

Original: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .
Revised: The referring segmentation can be formulated as themaximum a posteriori estimation problem of p ( M o | V, E o ) .

Original: By applying the Bayes rule, we obtain:
Revised: By applying the Bayes rule, we obtain:

Original: [Equation]
Revised: [Equation]

Original: As the prior p ( M o | V ) is not affected by the expression E o , we consider maximizing p ( E o | V, M o )as a dual problem of the referring segmentation (primary problem), which is to reconstruct the textexpression given the video and object masks.
Revised: As the prior p ( M o | V ) is not affected by the expression E o , we consider maximizing p ( E o | V, M o )as a dual problem of the referring segmentation (primary problem), which is to reconstruct the textexpression given the video and object masks.

Original: We note that for negative videos, p ( E o | V, M o ) isundefined because the mask M o is empty.
Revised: We note that for negative videos, p ( E o | V, M o ) isundefined because the mask M o is empty.

Original: Thus, we only investigate the dual problem for positivevideos.
Revised: Thus, we only investigate the dual problem for positivevideos.

Original: The primary problem and the dual problem can be connected in a cycle path, i.e., from theoriginal expression E o to the reconstructed expression E ′
Revised: The primary problem and the dual problem can be connected in a cycle path, i.e., from theoriginal expression E o to the reconstructed expression E ′

Original: o
Revised: o

Original: through positive video queries, as shownin Figure 2 (a).
Revised: through positive video queries, as shownin Figure 2 (a).

Original: We believe that the cycle constraint benefits to optimize the primary problem byenhancing the semantic consensus.
Revised: We believe that the cycle constraint benefits to optimize the primary problem byenhancing the semantic consensus.

Original: Video 𝑉 = 𝐈 ! !"
Revised: Video 𝑉 = 𝐈 ! !"

Original: #$
Revised: #$

Original: Text exp.
Revised: Text exp.

Original: 𝐸 %
Revised: 𝐸 %

Original: Instance emb.
Revised: Instance emb.

Original: 𝐳 & &"#$
Revised: 𝐳 & &"#$

Original: Box 𝐁 ! !"
Revised: Box 𝐁 ! !"

Original: #$ Score 𝐒 ! !"
Revised: #$ Score 𝐒 ! !"

Original: #$
Revised: #$

Original: Visual Encoder
Revised: Visual Encoder

Original: Textual Encoder
Revised: Textual Encoder

Original: Visual Decoder
Revised: Visual Decoder

Original: Seg.
Revised: Seg.

Original: 𝐌 ! !"
Revised: 𝐌 ! !"

Original: #$ a cat laying by an owl
Revised: #$ a cat laying by an owl

Original: Skip connections
Revised: Skip connections

Original: Pos. / Neg.
Revised: Pos. / Neg.

Original: R
Revised: R

Original: Dot Product
Revised: Dot Product

Original: Repeat R
Revised: Repeat R

Original: Concatenation C
Revised: Concatenation C

Original: Reconstructed text emb. 𝐞′
Revised: Reconstructed text emb. 𝐞′

Original: Relational Cycle Consistency
Revised: Relational Cycle Consistency

Original: Multi-scale feat.
Revised: Multi-scale feat.

Original: 𝐅 𝐅
Revised: 𝐅 𝐅

Original: Instance query 𝐳 ’
Revised: Instance query 𝐳 ’

Original: Sentence-level text emb.
Revised: Sentence-level text emb.

Original: 𝐞
Revised: 𝐞

Original: C
Revised: C

Original: Conv Grounded feat.
Revised: Conv Grounded feat.

Original: 𝐟 ()*+,
Revised: 𝐟 ()*+,

Original: Dynamic kernel Θ 𝐠𝐟 - Video feat.
Revised: Dynamic kernel Θ 𝐠𝐟 - Video feat.

Original: 𝐟
Revised: 𝐟

Original: Word-level feat.
Revised: Word-level feat.

Original: 𝐠
Revised: 𝐠

Original: Multimodal Attention
Revised: Multimodal Attention

Original: Transformer Decoder 𝒟 !
Revised: Transformer Decoder 𝒟 !

Original: Text query 𝐞 ’
Revised: Text query 𝐞 ’

Original: Early Grounding
Revised: Early Grounding

Original: V-T Projection
Revised: V-T Projection

Original: Pool
Revised: Pool

Original: [Figure]
Revised: [Figure]

Original: Transformer Decoder
Revised: Transformer Decoder

Original: [Figure]
Revised: [Figure]

Original: ℋ
Revised: ℋ

Original: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textual embedding space E ′ induced by positive videos.
Revised: In practice, we study the cycle consistency between the original textual embedding space E and thetransformed textual embedding space E ′ induced by positive videos.

Original: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.
Revised: By definition, the path from theoriginal text embedding e o to the reconstructed text embedding e ′ o is modulated with cross-modalinteractions between video and text.

Original: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal intermediate feature f m to convey information of both theinput of the primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).
Revised: Thus, to link the primary and dual problem and enable the jointoptimization, we introduce a cross-modal intermediate feature f m to convey information of both theinput of the primary problem ( V, E o ) and the dual problem ( V, M o ) , as shown in Figure 2 (b).

Original: f m isdefined between the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv ,Π dece , to only focus on the multi-modal interaction.
Revised: f m isdefined between the encoder and decoder stages of single-modal operations, i.e., Π encv , Π ence , Π decv ,Π dece , to only focus on the multi-modal interaction.

Original: Relational cycle consistency.
Revised: Relational cycle consistency.

Original: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.
Revised: A key observation for cycle consistency between E and E ′ is that themapping between them is not necessarily bijective, as there could be multiple textual descriptions forthe same object.

Original: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.
Revised: Thus, naively adding point-wise consistency, i.e., e o = e ′ o , ∀ e o ∈ E will collapsethe feature space to a sub-optimal solution.

Original: Instead, we take inspiration from relational knowledgedistillation [33], and introduce relational cycle consistency for E and E ′ .
Revised: Instead, we take inspiration from relational knowledgedistillation [33], and introduce relational cycle consistency for E and E ′ .

Original: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).
Revised: The relational cycleconsistency is to preserve the structure of the feature space rather than exact point-wise consistency,as illustrated in Figure 2 (c) and (d).

Original: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e
Revised: Mathematically, the structure-preserving property is defined asisometric and conformal constraints to preserve pair-wise distance and angles for e ∈ E and e

Original: ′ ∈ E ′ :
Revised: ′ ∈ E ′ :

Original: [Equation]
Revised: [Equation]

Original: where | · | and ∠ ( · ) denote distance and angle metrics.
Revised: where | · | and ∠ ( · ) denote distance and angle metrics.

Original:  Method
Revised:  Method

Original: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.
Revised: In this section, we elaborate our R 2 -VOS framework with the relational consistency, which mainlyconsists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection fortext reconstruction, and mask decoding for final segmentation, as shown in Figure 3.

Original: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .
Revised: We first extractthe video feature f , word-level text feature g , and sentence-level text embedding e .

Original: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .
Revised: On the one hand,to model the primary segmentation problem of maximizing p ( M o | V, E o ) , we enable the multimodalinteraction in the early grounding module to generate the grounded feature f early .

Original: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.
Revised: f early coarselylocates the referred object o and filters out irrelevant features, which serves as a medium linkingthe primary segmentation and dual text reconstruction problem.

Original: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instanceembedding { z t } Tt =1 .
Revised: The final mask M o is obtained bydynamic convolution [4] on the decoded visual feature maps, with kernels learned from instanceembedding { z t } Tt =1 .

Original: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f
Revised: On the other hand, to model the dual text reconstruction problem of maximizingp ( E o | V, M o ) , we utilize the grounded video feature f

Original: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .
Revised: early as the alternative of V and M o , sincef early conveys contextual video clues of object o .

Original: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .
Revised: In this way, we enable the parallel optimization ofthe primary and dual problem by relating them to f early .

Original: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .
Revised: Specifically, we employ a V-T projectionmodule to project f early onto a reconstructed text embedding e ′ .

Original: We add relational constraint between e ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.
Revised: We add relational constraint between e ′ and e to enforce the semantic alignment between the segmented mask and expression for positivevideos.

Original: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.
Revised: In addition, we introduce a semantic consensus discrimination head H ( e , e ′ ) to assess theconsistency between original and reconstructed text embeddings, discriminating the alignment ofmultimodal semantics and identifying negative videos.

Original: 4.1 Single-modal Feature Extraction
Revised: 4.1 Single-modal Feature Extraction

Original: [Figure]
Revised: [Figure]

Original: Visual encoder.
Revised: Visual encoder.

Original: Following previous methods [1, 44, 43], we build the visual encoder with a visualbackbone and a deformable transformer encoder [52] on top of it.
Revised: Following previous methods [1, 44, 43], we build the visual encoder with a visualbackbone and a deformable transformer encoder [52] on top of it.

Original: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [12], andthen fed into a deformable transformer encoder [52] similar to the previous method [44].
Revised: The extracted features from thebackbone are flattened, projected to a lower dimension, added with positional encoding [12], andthen fed into a deformable transformer encoder [52] similar to the previous method [44].

Original: We denotethe multi-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.
Revised: We denotethe multi-scale output of the transformer encoder as F and the low-resolution visual feature map fromthe backbone as f , where f ∈ R T × C v × H 32 × W 32 , C v is the feature channel, T is the video length and Hand W are the original image size.

Original: Textual encoder.
Revised: Textual encoder.

Original: We leverage a pre-trained linguistic model RoBERTa
Revised: We leverage a pre-trained linguistic model RoBERTa

Original: [27] to map the input textualexpression E o to a textual embedding space.
Revised: [27] to map the input textualexpression E o to a textual embedding space.

Original: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively.
Revised: The textual encoder extracts a sequence of word-leveltext feature g ∈ R C e × L and a sentence-level text embedding e ∈ R C e × 1 , where C e and L are thedimension of linguistic embedding space and the expression length respectively.

Original: [Figure]
Revised: [Figure]

Original: We propose an early grounding module to coarsely
Revised: We propose an early grounding module to coarsely

Original: [Figure]
Revised: [Figure]

Original: locate the referred object o and filter out irrelevant fea-
Revised: locate the referred object o and filter out irrelevant fea-

Original: [Figure]
Revised: [Figure]

Original: tures.
Revised: tures.

Original: Then the grounded feature f early encoding in-
Revised: Then the grounded feature f early encoding in-

Original: [Figure]
Revised: [Figure]

Original: formation of o can not only be utilized for the primary
Revised: formation of o can not only be utilized for the primary

Original: [Figure]
Revised: [Figure]

Original: segmentation problem, but also for the dual expression
Revised: segmentation problem, but also for the dual expression

Original: [Figure]
Revised: [Figure]

Original: reconstruction task, which serves as a proxy connecting
Revised: reconstruction task, which serves as a proxy connecting

Original: [Figure]
Revised: [Figure]

Original: the two problems.
Revised: the two problems.

Original: Figure 4 shows a visualization of
Revised: Figure 4 shows a visualization of

Original: [Figure]
Revised: [Figure]

Original: f early .
Revised: f early .

Original: Specifically, we utilize the power of dynamic
Revised: Specifically, we utilize the power of dynamic

Original: [Figure]
Revised: [Figure]

Original: convolution [4] to discriminate visual features in theearly stage.
Revised: convolution [4] to discriminate visual features in theearly stage.

Original: As shown in the blue part of Figure 3, we first enable the multimodal interaction betweenvideo and text features, then apply the dynamic convolution with kernels learned from text featureto discriminate the object-level semantics.
Revised: As shown in the blue part of Figure 3, we first enable the multimodal interaction betweenvideo and text features, then apply the dynamic convolution with kernels learned from text featureto discriminate the object-level semantics.

Original: In particular, multi-head cross-attention (MCA) [41] isleveraged to conduct the multimodal interaction:
Revised: In particular, multi-head cross-attention (MCA) [41] isleveraged to conduct the multimodal interaction:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where MCA( X , Y ) =
Revised: where MCA( X , Y ) =

Original: Attention( W Q X , W K Y , W V Y ) .
Revised: Attention( W Q X , W K Y , W V Y ) .

Original: W represents learnable weight.
Revised: W represents learnable weight.

Original: LNand FFN denote layer normalization and feed-forward network respectively.
Revised: LNand FFN denote layer normalization and feed-forward network respectively.

Original: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =
Revised: The text feature g ′ isfurther pooled to a fixed length, and followed by a fully-connected layer to form the dynamic kernelsΘ =

Original: { θ i } Ki =1 .
Revised: { θ i } Ki =1 .

Original: K is the kernel number and θ i ∈ R C × 1 .
Revised: K is the kernel number and θ i ∈ R C × 1 .

Original: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW
Revised: The dynamic kernels are applied separatelyto video feature f ′ ∈ R C × THW to form the f early ∈ R C × THW

Original: [Equation]
Revised: [Equation]

Original: 6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.
Revised: 6) where ⊕ is the concatenation in channel dimension and φ ( · ) is a convolution to reduce the featuredimension.

Original: BN denotes batch normalization.
Revised: BN denotes batch normalization.

Original: 4.3 Text Reconstruction
Revised: 4.3 Text Reconstruction

Original: V-T projection.
Revised: V-T projection.

Original: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.
Revised: We leverage a transformer decoder D E as textual decoder to transform the visualrepresentation of the referred object into the textual space.

Original: As shown in Figure 3, a learnable textquery e
Revised: As shown in Figure 3, a learnable textquery e

Original: 0 ∈ R C
Revised: 0 ∈ R C

Original: e × 1 is employed to query the f
Revised: e × 1 is employed to query the f

Original: early .
Revised: early .

Original: The output of the transformer decoder is thereconstructed text embedding e ′ =
Revised: The output of the transformer decoder is thereconstructed text embedding e ′ =

Original: D E ( f early , e 0 )
Revised: D E ( f early , e 0 )

Original: ∈ R C
Revised: ∈ R C

Original: e × 1 .
Revised: e × 1 .

Original: 4.4 Referring Segmentation
Revised: 4.4 Referring Segmentation

Original: Mask segmentation.
Revised: Mask segmentation.

Original: Similar to previous methods [44, 1, 11], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.
Revised: Similar to previous methods [44, 1, 11], we leverage deformable transformerdecoders with dynamic convolution to segment the object masks.

Original: As shown in Figure 3, we first fuse the reconstructed text embedding e ′ to text embedding e .
Revised: As shown in Figure 3, we first fuse the reconstructed text embedding e ′ to text embedding e .

Original: The fused text embedding e is then repeated
Revised: The fused text embedding e is then repeated

Original: N times to form the instance query [43] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.
Revised: N times to form the instance query [43] z 0 ∈ R C q × N , where C q is the dimension of instance queryand N is the instance query number.

Original: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .
Revised: We then use T × deformable transformer decoders D V withshared weights to decode the instance embeddings z t ∈ R C q × N for each frame, i.e., z t = D V ( F t , z 0 ) .

Original: F t is the multiscale visual feature from visual encoder at time t .
Revised: F t is the multiscale visual feature from visual encoder at time t .

Original: A dynamic kernel w t is furtherlearned from the instance embedding z t .
Revised: A dynamic kernel w t is furtherlearned from the instance embedding z t .

Original: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [23] in the visual decoder.
Revised: The final feature map f out ,t ∈ R C × H × W is obtained byfusing low-level features from the feature pyramid network [23] in the visual decoder.

Original: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .
Revised: The maskprediction M t ∈ R N × H × W can be computed by M t = w T t f out , t .

Original: Auxiliary heads.
Revised: Auxiliary heads.

Original: We build a set of auxiliary heads to obtain the final object masks across frames.
Revised: We build a set of auxiliary heads to obtain the final object masks across frames.

Original: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .
Revised: Inparticular, a box head, a scoring head and a semantic consensus discrimination head are leveraged topredict the bounding boxes B t ∈ R N × 4 , confidence scores S t ∈ R N × 1 and the alignment degree ofmultimodal semantics A ∈ R .

Original: The box and scoring head are two fully-connected layers upon theinstance embedding e t .
Revised: The box and scoring head are two fully-connected layers upon theinstance embedding e t .

Original: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .
Revised: The semantic consensus discrimination head H ( e , e ′ ) consists of two fullyconnected layers upon the text embeddings e ⊕ e ′ .

Original: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.
Revised: Note that A represents the semantic alignment inthe entire video rather a single frame, since the expression is a video-level description.

Original: 4.5 Loss Function
Revised: 4.5 Loss Function

Original: The loss function of our method can be boiled down to three parts:
Revised: The loss function of our method can be boiled down to three parts:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.
Revised: ) where L text , L segm , and L align are losses for text reconstruction, referring segmentation and semanticconsensus discrimination respectively.

Original: A ground-truth semantic alignment ˆ A
Revised: A ground-truth semantic alignment ˆ A

Original: = { 0 , 1 } is utilizedto discriminate positive and negative pairs.
Revised: = { 0 , 1 } is utilizedto discriminate positive and negative pairs.

Original: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .
Revised: The L align is simply a cross-entropy loss between thepredicted alignment A and ground-truth ˆ A .

Original: The other two terms are computed as follows:
Revised: The other two terms are computed as follows:

Original: Loss for text reconstruction.
Revised: Loss for text reconstruction.

Original: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .
Revised: Given the text embedding e and reconstructed text embedding e ′ , weemploy a relational constraint to impose the cycle consistency between e and e ′ .

Original: We calculate theloss by
Revised: We calculate theloss by

Original: [Equation]
Revised: [Equation]

Original: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss
Revised: where the indicator function 1 ( ˆ A ) = 1 if the alignment indicates the referred object exists in thevideo, otherwise 0 , λ angle is a hyperparameter balancing the distance loss L dist and angle loss

Original: L angle .
Revised: L angle .

Original: We elaborate these two losses according to the relational cycle consistency Equation 2.
Revised: We elaborate these two losses according to the relational cycle consistency Equation 2.

Original: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )
Revised: Let X n = { ( x 1 , ..., x n ) | x i ∈ X} denote a set of n -tuples, Φ n = { ( x , x ′ )

Original: | x ∈ X n , x ′ ∈
Revised: | x ∈ X n , x ′ ∈

Original: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and
Revised: X ′ n }denote a set of pairs consisting of two n -tuples of distinct elements from two different sets X and

Original: X ′ .
Revised: X ′ .

Original: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ 

Revised: Specifically, the distance-based and angle-based relations relate text embeddings of 2-tuple and3-tuple respectively, i.e., Φ

Original: = { ( x , x ′ )
Revised: = { ( x , x ′ )

Original: | x = ( e i , e j ) , x ′ =
Revised: | x = ( e i , e j ) , x ′ =

Original: ( e ′ i , e ′ j ) , i ̸
Revised: ( e ′ i , e ′ j ) , i ̸

Original: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =
Revised: = j } and Φ 3 = { ( x , x ′ ) | x =( e i , e j , e k ) , x ′ =

Original: ( e ′ i , e ′ j , e ′ k ) , i ̸
Revised: ( e ′ i , e ′ j , e ′ k ) , i ̸

Original: =
Revised: =

Original: j ̸ = k } .
Revised: j ̸ = k } .

Original: Then the losses are given by:
Revised: Then the losses are given by:

Original: [Equation]
Revised: [Equation]

Original: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )
Revised: where µ ( x ) = (cid:80) x =( x 1 ,x 2 ) ∈X 2 || x 1 − x 2 || 2 |X 2 | is the average distance function, and the Huber lossl δ ( x, x ′ )

Original: =
Revised: =

Original: 12 ( x − x ′ ) 

Revised: 12 ( x − x ′ )

Original: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .
Revised: if | x − x ′ | ≤ 1 , otherwise | x − x ′ | − 12 .

Original: Loss for referring segmentation.
Revised: Loss for referring segmentation.

Original: Given a set of predictions y
Revised: Given a set of predictions y

Original: =
Revised: =

Original: { y i } Ni =1 and ground-truth ˆy ,where y i =
Revised: { y i } Ni =1 and ground-truth ˆy ,where y i =

Original: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).
Revised: { B i,t , S i,t , M i,t } Tt =1 and ˆy = { ˆB t , ˆS t , ˆM t } Tt =1 , we search for an assignment σ ∈ P Nwith the highest similarity where P N is a set of permutations of N elements ( ˆy is padded with ∅ ).

Original: The similarity can be computed as
Revised: The similarity can be computed as

Original: [Equation]
Revised: [Equation]

Original: where λ box , λ conf , and λ mask are weights to balance losses.
Revised: where λ box , λ conf , and λ mask are weights to balance losses.

Original: Following previous works [6, 43], weleverage a combination of Dice [20] and BCE loss as L mask , focal loss [24] as L conf , and GIoU[38] and L1 loss as L box .
Revised: Following previous works [6, 43], weleverage a combination of Dice [20] and BCE loss as L mask , focal loss [24] as L conf , and GIoU[38] and L1 loss as L box .

Original: The best assignment ˆ σ is solved by Hungarian algorithm [16].
Revised: The best assignment ˆ σ is solved by Hungarian algorithm [16].

Original: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as
Revised: Giventhe best assignment ˆ σ , the segmentation loss between ground-truth and predictions is defined as

Original: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .
Revised: L segm = 1 ( ˆ A ) · L match ( y , ˆy ˆ σ ( i ) ) .

Original: 4.6 Inference
Revised: 4.6 Inference

Original: During inference, we select the candidate with the highest confidence to predict the final masks:
Revised: During inference, we select the candidate with the highest confidence to predict the final masks:

Original: [Equation]
Revised: [Equation]

Original: where { ¯M t } Tt =1 is the masks of referred object.
Revised: where { ¯M t } Tt =1 is the masks of referred object.

Original: S i,t and M i,t represent the i -th candidate in S t and
Revised: S i,t and M i,t represent the i -th candidate in S t and

Original: M t respectively.
Revised: M t respectively.

Original: ¯s is the slot with the highest confidence to be the target object.
Revised: ¯s is the slot with the highest confidence to be the target object.

Original: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm.
Revised: We use 1 ( A ) to filterout predictions in negative videos to mitigate false alarm.

Original: 1 ( A )
Revised: 1 ( A )

Original: = 

Revised: =

Original: if A > 0 . 5 , else 0.
Revised: if A > 0 . 5 , else 0.

Original:  Experiment
Revised:  Experiment

Original: 5.1 Dataset and Metrics
Revised: 5.1 Dataset and Metrics

Original: Dataset.
Revised: Dataset.

Original: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.
Revised: We conduct experiments on three datasets: Ref-Youtube-VOS, Ref-DAVIS and R 2Youtube-VOS.

Original: Ref-Youtube-VOS [39] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.
Revised: Ref-Youtube-VOS [39] is a large-scale benchmark that has 3,978 videos with about15 k language descriptions.

Original: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.
Revised: There are 3,471 videos with 12,913 expressions in the training set and 507videos with 2,096 expressions in the validation set.

Original: Ref-DAVIS-17 [13] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.
Revised: Ref-DAVIS-17 [13] contains 90 videos with 1,544expressions, including 60 and 30 videos for training and validation respectively.

Original: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.
Revised: R 2 -Youtube-VOSis our newly proposed evaluation dataset: it extends the Ref-Youtube-VOS validation set with eachlinguistic expression to query a positive video (the same one as Ref-Youtube-VOS) and a negativevideo.

Original: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.
Revised: To make each video can be picked as a negative video, we randomly shuffle the original videoset and constrain all negative text-video pair unrelated.

Original: Metrics.
Revised: Metrics.

Original: We employ commonly-used region similarity J and contour accuracy F [36] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.
Revised: We employ commonly-used region similarity J and contour accuracy F [36] for conventional Ref-Youtube-VOS and Ref-DAVIS-17 benchmarks.

Original: For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 −
Revised: For the proposed R 2 -Youtube-VOStask, we additionally introduce a new metric R = 1 −

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively.
Revised: | to evaluate the degree of objectfalse alarm in negative videos, where M neg and M pos are the sets containing segmented masks innegative and positive videos respectively.

Original: | M | denotes the foreground area of mask M .
Revised: | M | denotes the foreground area of mask M .

Original: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.
Revised: The totalforeground area of positive videos (cid:80) M ∈M pos | M | serves as a normalization term.

Original: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .
Revised: Ideally, a modelshould treat all the negative videos as backgrounds with R = 1 .

Original: 5.2 Implementation Details
Revised: 5.2 Implementation Details

Original: Following previous methods [6, 44], our model is first pre-trained on Ref-COCO/+/g dataset [49, 31]and then finetuned on Ref-Youtube-VOS.
Revised: Following previous methods [6, 44], our model is first pre-trained on Ref-COCO/+/g dataset [49, 31]and then finetuned on Ref-Youtube-VOS.

Original: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.
Revised: The model is trained for 6 epochs with a learning ratemultiplier of 0.1 at the 3rd and the 5th epoch.

Original: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.
Revised: The initial learning rate is 1e-4 and a learning ratemultiplier of 0.5 is applied to the backbone.

Original: We adopt a batchsize of 8 and an AdamW [29] optimizerwith weight decay 1 × 10 − 4 .
Revised: We adopt a batchsize of 8 and an AdamW [29] optimizerwith weight decay 1 × 10 − 4 .

Original: Following convention [44, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.
Revised: Following convention [44, 1], the evaluation on Ref-DAVIS directlyuses models trained on Ref-Youtube-VOS without re-training.

Original: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels during evaluation.
Revised: All images are cropped to have thelongest side of 640 pixels and the shortest side of 360 pixels during evaluation.

Original: The window size isset to 5 for all backbones.
Revised: The window size isset to 5 for all backbones.

Original: We create negative pairs by shuffling positive pairs in each batch.
Revised: We create negative pairs by shuffling positive pairs in each batch.

Original: Ourmethod is implemented with PyTorch [34].
Revised: Ourmethod is implemented with PyTorch [34].

Original: 5.3 Main Results
Revised: 5.3 Main Results

Original: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.
Revised: We compare our method with state-of-the-art R-VOS methods on Ref-Youtube-VOS and Ref-DAVIS17 in Table 1, and R 2 -VOS task in Table 2.

Original: Comparison on Ref-Youtube-VOS.
Revised: Comparison on Ref-Youtube-VOS.

Original: In Table 1, we first compare our method on Ref-Youtube-VOS.
Revised: In Table 1, we first compare our method on Ref-Youtube-VOS.

Original: For results of ResNet [9] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [44] by 1.7 J & F .
Revised: For results of ResNet [9] backbone, our method achieves 57.3 J & F which outperforms the latestmethod ReferFormer [44] by 1.7 J & F .

Original: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).
Revised: In addition, our method runs at 30 FPS compared to 22 FPSof state-of-the-art ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1 ).

Original: For
Revised: For

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison on R 2 -VOS.
Revised: Comparison on R 2 -VOS.

Original: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.
Revised: As shown in Table 2, the state-of-the-art R-VOS methods, ReferFormerand MTTR suffer from a low R metric which measures the false-alarm problem when the semanticconsensus of the input text-video pair does not hold.

Original: Compared to the severe false alarm of previous
Revised: Compared to the severe false alarm of previous

Original: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.
Revised: R-VOS methods, our model successfully mitigates the false alarm of the model, thanks to the proposedmultimodal cycle consistency constraint and semantic consensus discrimination head.

Original: [Figure]
Revised: [Figure]

Original: Qualitative results.
Revised: Qualitative results.

Original: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.
Revised: We compare thequalitative results of our method againststate-of-the-art methods in Figure 6 on R 2VOS.

Original: For positive videos : The result indicates that our method predicts accurate andtemporally-consistent results, while ReferFormer [44] and MTTR [1] fail to locatethe correct object.
Revised: For positive videos : The result indicates that our method predicts accurate andtemporally-consistent results, while ReferFormer [44] and MTTR [1] fail to locatethe correct object.

Original: For negative videos :
Revised: For negative videos :

Original: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.
Revised: Both ReferFormer and MTTR suffer froma severe false-alarm problem when the referred object does not exist in the video.

Original: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.
Revised: Incontrast, with multi-modal cycle constraintand consensus discrimination, our methodsuccessfully filters out negative videos andmitigates the false alarm.

Original: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.
Revised: To further explore how the consensus discriminationworks, we visualize the text embedding andreconstructed text embedding spaces for both positive and negative videos.

Original: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.
Revised: As shown in Figure 5, wenotice that, for embeddings of positive videos, they preserve relative relations well, while for negativevideos, the reconstructed embeddings have a random pattern in the space.

Original: 5.4 Ablation Study
Revised: 5.4 Ablation Study

Original: Module effectiveness.
Revised: Module effectiveness.

Original: To investigate the effectiveness of different components in our method,we conduct experiments with the ResNet-50 backbone on R 2 -Youtube-VOS dataset.
Revised: To investigate the effectiveness of different components in our method,we conduct experiments with the ResNet-50 backbone on R 2 -Youtube-VOS dataset.

Original: We build atransformer-based baseline model and equip our proposed components step-by-step.
Revised: We build atransformer-based baseline model and equip our proposed components step-by-step.

Original: As shown in
Revised: As shown in

Original: Table 3, the baseline model achieves 52.4 J & F .
Revised: Table 3, the baseline model achieves 52.4 J & F .

Original: Then, we add our proposed components step-bystep to demonstrate the module effectiveness.
Revised: Then, we add our proposed components step-bystep to demonstrate the module effectiveness.

Original: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.
Revised: After employing the early grounding module, theperformance boosts to 55.5 J & F and the cycle-consistency constraint brings another 1.4 J & F gain.

Original: Since the reconstructed text embedding is generated with visual features injected, we consider it can
Revised: Since the reconstructed text embedding is generated with visual features injected, we consider it can

Original: F r a m e
Revised: F r a m e

Original: M TT R
Revised: M TT R

Original: R e f e r F o r m e r
Revised: R e f e r F o r m e r

Original: O u r s
Revised: O u r s

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: encode some visual information, thus augmenting the original text embedding.
Revised: encode some visual information, thus augmenting the original text embedding.

Original: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .
Revised: By using the fusedtext embedding as instance query, we achieve our best performance of 57.3 J & F .

Original: Consistency constraint.
Revised: Consistency constraint.

Original: We conduct experiments to ablate the influence of cycle-consistencyconstraints.
Revised: We conduct experiments to ablate the influence of cycle-consistencyconstraints.

Original: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.
Revised: As shown in Table 4, utilizing point-wise consistency constraint will lead to a performancedrop compared to the setting without cycle constraint.

Original: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.
Revised: We consider the point-wise constraint mayforce an injective mapping from the textual domain to the visual domain.

Original: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.
Revised: However, the mapping canbe a many-to-one function for R-VOS, i.e., each object corresponds to multiple textual descriptions.

Original: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.
Revised: In addition, since the early grounding leverages the text feature to locate the referred object, if we usethe direct point-wise constraint to form reconstructed text embedding, it will encourage the networkto memorize the text feature in the f early and result in a collapse for text reconstruction.

Original: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.
Revised: Table 4shows that sole relational angle constraint can bring 1.2 J & F gain, and it can be slightly improvedwith 1.4 J & F gain by jointly using relational angle and distance constraint.

Original: Instance query number.
Revised: Instance query number.

Original: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.
Revised: Although only one referral is involved for each frame in R-VOS task,to help the network optimization, we employ more than one instance query to each video.

Original: Table 5indicates that a query number of 5 brings the best result.
Revised: Table 5indicates that a query number of 5 brings the best result.

Original: Frame number.
Revised: Frame number.

Original: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporally-consistent results.
Revised: Since R-VOS gives a text that describes an object over a period of time, temporalinformation is vital to segment accurate and temporally-consistent results.

Original: We ablate on the bestwindow size of input videos during training.
Revised: We ablate on the bestwindow size of input videos during training.

Original: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .
Revised: As shown in Table 6, we notice that the performanceimproves as the window size increases and a window size of 5 brings the best result of 57.3 J & F .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we investigate the semantic misalignment problem in R-VOS task.
Revised: In this paper, we investigate the semantic misalignment problem in R-VOS task.

Original: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.
Revised: A pipeline jointlymodels the referring segmentation and text reconstruction problem, equipped with a relational cycleconsistency constraint, is introduced to discriminate and enhance the semantic consensus betweenvisual and textual modalities.

Original: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.
Revised: To evaluate the model robustness, we extend the R-VOS task toaccept unpaired inputs and collect a corresponding R 2 -Youtube-VOS dataset.

Original: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.
Revised: We observe a severefalse-alarm problem suffered from previous methods on R 2 -Youtube-VOS while ours accuratelydiscriminates unpaired inputs and segments high-quality masks for paired inputs.

Original: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.
Revised: Our methodachieves state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and R 2 -VOS dataset.

Original: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation. 
Revised: Webelieve that, with unpaired inputs, R 2 -VOS is a more general setting of referring video segmentation,which can shed light on a new direction to investigate the robustness of referring segmentation. A Additional AppendixA.1 More Quantitative Result Analysis

Original: Checklist
Revised: Under the same ResNet-50 backbone, our method achieves 57.3 J & F , 94.1 R and 30 FPS comparedto the 55.6 J & F , 30.6 R and 22 FPS of ReferFormer. We will then point-to-point analyze reasons ofimprovements on J & F (for positive video), R (for negative videos) and FPS (for inference speed). • J & F : (1) We introduce the early-grounding module which employs both pixel-wise andchannel-wise attention to enable multimodal interaction. Different from the CM-FPN used inReferFormer that solely fuses features from text to video in pixel-level, our early-groundingmodule first enables pixel-level bi-directional fusion and then generates dynamic kernels

Revised: using the fused text feature g ′ to modulate the video feature f ′ .

Revised: The dynamic convolution(channel-wise attention) is commonly used to decode dense masks from visual features and

Revised: is suitable to suppress irrelevant features.

Revised: By equipping text-guided dynamic convolution in

Revised: early-stage, the pixel decoder can be more focused on the target object (as shown in Figure 4).

Revised: (2) Our method leverages relational cycle consistency to constraint the intermediate feature

Revised: f early to contain correct object-level information to recover some properties of original textembedding.

Revised: By applying this constraint, our method can better avoid interference and easierlocate the correct object.

Revised: Our instance query is composed of both original sentenceembedding and the reconstructed one.

Revised: Different from ReferFormer that only utilizes original

Revised: sentence embedding as queries, the reconstructed embedding can encode visual information

Revised: to facilitate the instance query decode the objects from visual features.

Revised: • R : The newly introduced metric R aims to measure the robustness of the model against

Revised: unpaired inputs.

Revised: Text-video pairs with (object-level) semantic consensus can be assumed

Revised: as in-distribution for RVOS problem where semantic consensus can be kind of easily

Revised: modeled.

Revised: In contrast, unpaired text-video is much more difficult to tackle because there can

Revised: be unlimited

Revised: out-of-distribution (OOD) scenarios for the text-video pairs.

Revised: In our method,instead of directly detect the OOD of input pairs, we convert the problem to find semanticalignment between the input text embedding and reconstructed embedding and constraintthe property of reconstructed space by introducing the cycle consistency.

Revised: In this way,the comparison is conducted in the constraint original and reconstructed text spaces.

Revised: For

Revised: ReferFormer, it models the alignment of text to video by querying the visual features by textin the transformer decoder.

Revised: In this way, the comparison is conducted in unconstrained textand video spaces thus results in a inferior performance.•

Revised: FPS

Revised: : The speed improvement of our method mainly comes from our efficient multimodalfusion.

Revised: Compared to the multi-scale CM-FPN, our early-grounding module is only conductat the high-level.

Revised: In addition, our bi-direction multimodal fusion (Equ 4 & 5) only leveragescross-attention to avoid computational expensive video-to-video operations.

Revised: A.2 Limitations

Revised: An important challenge for video segmentation is that target object disappearance due to occlusion,which can results in false positives on a per-frame level.

Revised: In our method, we predict the video-levelsemantic alignment to handle the false positive in video-level resulted from unpaired text-video pairs.

Revised: However, since only video-level object expression is available in RVOS task, our method can notaddress the frame-level false positives resulted from occlusion.

Revised: A.3 Additional Experiment on Negative Videos without Positive Text

Revised: [Table]

Revised: As shown in Table A, we test the robustness of our model on two settings.

Revised: We generate negativevideos from Ref-Youtube-VOS and a combination of Ref-Youtube-VOS and Ref-DAVIS dataset.

Revised: In both settings, all videos in the validation set are leveraged.

Revised: The results indicates that source ofnegative videos has minor impact on the robustness of our model.

Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (b) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you describe the limitations of your work?
Revised: (d) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: We are not including theoretical results.
Revised: We are not including theoretical results.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] The code are
Revised: [No] The code are

Original: proprietary; most of our used dataset are public available; we plan to release code and
Revised: proprietary; most of our used dataset are public available; we plan to release code and

Original: novel dataset upon acceptance.
Revised: novel dataset upon acceptance.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We conduct several experiments multiple times and find
Revised: [No] We conduct several experiments multiple times and find

Original: the results are very close.
Revised: the results are very close.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] The data we used are public available datasets for academic
Revised: [Yes] The data we used are public available datasets for academic

Original: purposes.
Revised: purposes.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] The dataset that we used does not contain such
Revised: [No] The dataset that we used does not contain such

Original: content.
Revised: content.

Original: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.
Revised: If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.