Original: M OTION - INDUCTIVE S ELF - SUPERVISED O BJECT D ISCOVERY IN V IDEOS
Revised: M OTION - INDUCTIVE S ELF - SUPERVISED O BJECT D ISCOVERY IN V IDEOS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we consider the task of unsupervised object discovery in videos.
Revised: In this paper, we consider the task of unsupervised object discovery in videos.

Original: Previous works have shown promising results via processing optical flows to segment objects.
Revised: Previous works have shown promising results via processing optical flows to segment objects.

Original: However, taking flow as input brings about two drawbacks.
Revised: However, taking flow as input brings about two drawbacks.

Original: First, flow cannot capture sufficient cues when objects remain static or partially occluded.
Revised: First, flow cannot capture sufficient cues when objects remain static or partially occluded.

Original: Second, it is challenging to establish temporal coherency from flow-only input, due to the missing texture information.
Revised: Second, it is challenging to establish temporal coherency from flow-only input, due to the missing texture information.

Original: To tackle these limitations, we propose a model for directly processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation, with the opacity channels being treated as the segmentation.
Revised: To tackle these limitations, we propose a model for directly processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation, with the opacity channels being treated as the segmentation.

Original: Additionally, to enforce object permanence, we apply temporal consistency loss on the inferred masks from randomly-paired frames, which refer to the motions at different paces, and encourage the model to segment the objects even if they may not move at the current time point.
Revised: Additionally, to enforce object permanence, we apply temporal consistency loss on the inferred masks from randomly-paired frames, which refer to the motions at different paces, and encourage the model to segment the objects even if they may not move at the current time point.

Original: Experimentally, we demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets (DAVIS2016,
Revised: Experimentally, we demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets (DAVIS2016,

Original: SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow as input.
Revised: SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow as input.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Representing the visual scene with objects as the basic elements has long been considered a fundamental cognitive ability of the intelligent agent, for it enables understanding and interaction with the world more efficiently, for example, combinatorial generalization in novel settings (Tenenbaum et al., 2011).
Revised: Representing the visual scene with objects as the basic elements has long been considered a fundamental cognitive ability of the intelligent agent, for it enables understanding and interaction with the world more efficiently, for example, combinatorial generalization in novel settings (Tenenbaum et al., 2011).

Original: Although it remains somewhat obscure at the level of neurophysiology on exactly how humans discover the objects in a visual scene in the first place, it is a consensus that motion seems to play an indispensable role in defining and discovering the objects from the scene.
Revised: Although it remains somewhat obscure at the level of neurophysiology on exactly how humans discover the objects in a visual scene in the first place, it is a consensus that motion seems to play an indispensable role in defining and discovering the objects from the scene.

Original: For example, in 1923, Wertheimer introduced the common fate principle that elements moving together tends to be perceived as a group (Wertheimer, 1923); while later Gibson claimed the independent motion has even been treated as one attribute to define an object visually (Gibson & Carmichael, 1966).
Revised: For example, in 1923, Wertheimer introduced the common fate principle that elements moving together tends to be perceived as a group (Wertheimer, 1923); while later Gibson claimed the independent motion has even been treated as one attribute to define an object visually (Gibson & Carmichael, 1966).

Original: Grounded on the above assumptions, the recent literature has witnessed numerous works with different models proposed for segmenting the moving objects via unsupervised learning (Yang et al., 2019;
Revised: Grounded on the above assumptions, the recent literature has witnessed numerous works with different models proposed for segmenting the moving objects via unsupervised learning (Yang et al., 2019;

Original: 2021b;a; Liu et al., 2021).
Revised: 2021b;a; Liu et al., 2021).

Original: Exploiting optical flows for object discovery naturally incurs two critical limitations: First , objects in videos may stop moving or be partially occluded at any time point, leaving no effective cues for their existence in the flow field; Second , computing optical flow from a pair of frames refers to a lossy encoding procedure, that poses a significant challenge for establishing temporal coherence, due to the lack of effective texture information.
Revised: Exploiting optical flows for object discovery naturally incurs two critical limitations: First , objects in videos may stop moving or be partially occluded at any time point, leaving no effective cues for their existence in the flow field; Second , computing optical flow from a pair of frames refers to a lossy encoding procedure, that poses a significant challenge for establishing temporal coherence, due to the lack of effective texture information.

Original: In contrast, adopting RGB frame sequences poses a few clear advantages.
Revised: In contrast, adopting RGB frame sequences poses a few clear advantages.

Original: The most obvious one is that, while objects do not necessarily move all the time, the property of temporal coherence in RGB space naturally guarantees a preliminary understanding of object permanence; Additionally, the rich textures in the appearance stream give more distinctive patterns than those in motion, allowing to better identify and distinguish the different objects.
Revised: The most obvious one is that, while objects do not necessarily move all the time, the property of temporal coherence in RGB space naturally guarantees a preliminary understanding of object permanence; Additionally, the rich textures in the appearance stream give more distinctive patterns than those in motion, allowing to better identify and distinguish the different objects.

Original: Last but not least, processing RGB streams still enables a faster processing speed than using optical flow.
Revised: Last but not least, processing RGB streams still enables a faster processing speed than using optical flow.

Original: In this paper, our goal is to train a video segmentation model that can discover the moving objects within a sequence of RGB frames, in the form of segmentation.
Revised: In this paper, our goal is to train a video segmentation model that can discover the moving objects within a sequence of RGB frames, in the form of segmentation.

Original: In specific, our proposed model first encodes consecutive frames independently, into a set of frame-wise visual features, that is followed by a temporal fusion with a Transformer encoder.
Revised: In specific, our proposed model first encodes consecutive frames independently, into a set of frame-wise visual features, that is followed by a temporal fusion with a Transformer encoder.

Original: To localise the moving objects, we randomly
Revised: To localise the moving objects, we randomly

Original: [Figure]
Revised: [Figure]

Original: T i
Revised: T i

Original: [Figure]
Revised: [Figure]

Original: f l o w pair the visual features from two frames and pass them into a frame comparator module, effectively establishing the relative motion between frames.
Revised: f l o w pair the visual features from two frames and pass them into a frame comparator module, effectively establishing the relative motion between frames.

Original: Inspired by Yang et al.
Revised: Inspired by Yang et al.

Original: (2021a), we decode the motion features into optical flows with a dual-layered representation, with the opacity weight of each layer treated as the segmentation mask.
Revised: (2021a), we decode the motion features into optical flows with a dual-layered representation, with the opacity weight of each layer treated as the segmentation mask.

Original: At training time, we exploit an off-the-shelf optical flow estimator, e.g. , RAFT (Teed & Deng, 2020), as the induction for flow reconstruction.
Revised: At training time, we exploit an off-the-shelf optical flow estimator, e.g. , RAFT (Teed & Deng, 2020), as the induction for flow reconstruction.

Original: To develop the property of object permanence, we enforce a temporal consistency on the inferred segmentation masks, which encourages the model to mine effective texture information from the RGB sequence and keep track of the objects even if they may be static at the current time point.
Revised: To develop the property of object permanence, we enforce a temporal consistency on the inferred segmentation masks, which encourages the model to mine effective texture information from the RGB sequence and keep track of the objects even if they may be static at the current time point.

Original: In short, we summarize the contributions in this paper:
Revised: In short, we summarize the contributions in this paper:

Original: First , we introduce the M otion-inductive
Revised: First , we introduce the M otion-inductive

Original: O bject D iscovery ( MOD ) model, a simple architecture for discovering the moving objects in videos, by directly processing a set of consecutive RGB frames.
Revised: O bject D iscovery ( MOD ) model, a simple architecture for discovering the moving objects in videos, by directly processing a set of consecutive RGB frames.

Original: Second , we propose a self-supervised proxy task that is used to train the architecture without relying upon any manual annotation.
Revised: Second , we propose a self-supervised proxy task that is used to train the architecture without relying upon any manual annotation.

Original: To overcome the challenge from flow-based methods, i.e. , objects may stay static or move slowly, we adopt a random-paired policy and restrain the temporal consistency.
Revised: To overcome the challenge from flow-based methods, i.e. , objects may stay static or move slowly, we adopt a random-paired policy and restrain the temporal consistency.

Original: Third , we conduct a series of ablation studies to validate each key component of our method, such as the temporal consistency of randompaired flow.
Revised: Third , we conduct a series of ablation studies to validate each key component of our method, such as the temporal consistency of randompaired flow.

Original: While evaluating three public benchmarks, we demonstrate superior performance over existing approaches on DAVIS2016 (Perazzi et al., 2016), SegTrackv

Revised: While evaluating three public benchmarks, we demonstrate superior performance over existing approaches on DAVIS2016 (Perazzi et al., 2016), SegTrackv

Original: (Li et al., 2013), and FBMS59 (Ochs et al., 2013), with considerable speed-up during the inference procedure.
Revised: (Li et al., 2013), and FBMS59 (Ochs et al., 2013), with considerable speed-up during the inference procedure.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Video Object Segmentation.
Revised: Video Object Segmentation.

Original: How to segment objects coherently in one video sequence has extended the topic of instance segmentation in the image.
Revised: How to segment objects coherently in one video sequence has extended the topic of instance segmentation in the image.

Original: There is a great amount of work about video object segmentation (VOS) in recent decades (Caelles et al., 2017; Hu et al., 2017; Fan et al., 2019; Dutt Jain et al., 2017; Lai & Xie, 2019; Maninis et al., 2018;
Revised: There is a great amount of work about video object segmentation (VOS) in recent decades (Caelles et al., 2017; Hu et al., 2017; Fan et al., 2019; Dutt Jain et al., 2017; Lai & Xie, 2019; Maninis et al., 2018;

Original: Oh et al., 2019; Voigtlaender et al., 2019; Caelles et al., 2017; Perazzi et al., 2017; Hu et al., 2018; Li & Loy, 2018; Bao et al., 2018; Voigtlaender et al., 2019; Johnander et al., 2019).
Revised: Oh et al., 2019; Voigtlaender et al., 2019; Caelles et al., 2017; Perazzi et al., 2017; Hu et al., 2018; Li & Loy, 2018; Bao et al., 2018; Voigtlaender et al., 2019; Johnander et al., 2019).

Original: Recently, the research on getting rid of the dense annotation and designing more effective self-supervised algorithms has attracted more and more interest in the computer vision community including VOS (Xu & Wang, 2021;
Revised: Recently, the research on getting rid of the dense annotation and designing more effective self-supervised algorithms has attracted more and more interest in the computer vision community including VOS (Xu & Wang, 2021;

Original: Jabri et al., 2020; Lai et al., 2020; Li et al., 2019; Vondrick et al., 2018; Lu et al., 2020; Wang et al., 2019;
Revised: Jabri et al., 2020; Lai et al., 2020; Li et al., 2019; Vondrick et al., 2018; Lu et al., 2020; Wang et al., 2019;

Original: Kipf et al., 2022).
Revised: Kipf et al., 2022).

Original: For VOS, there are two mainstream protocols to evaluate the learned model.
Revised: For VOS, there are two mainstream protocols to evaluate the learned model.

Original: One is semi-supervised video object segmentation, the other is unsupervised video object segmentation.
Revised: One is semi-supervised video object segmentation, the other is unsupervised video object segmentation.

Original: Given the first-frame mask of the objects of interest, semi-supervised VOS tracks those objects in subsequent frames, while unsupervised VOS directly segments the most salient objects from the background without any reference.
Revised: Given the first-frame mask of the objects of interest, semi-supervised VOS tracks those objects in subsequent frames, while unsupervised VOS directly segments the most salient objects from the background without any reference.

Original: These two protocols are defined in the inference phase, meaning methods could leverage ground truth annotations in the training stage.
Revised: These two protocols are defined in the inference phase, meaning methods could leverage ground truth annotations in the training stage.

Original: In this paper, we don’t use any kinds of manual annotations for either training or evaluation.
Revised: In this paper, we don’t use any kinds of manual annotations for either training or evaluation.

Original: Motion Segmentation.
Revised: Motion Segmentation.

Original: As the name suggests, the aim of motion segmentation is to discover moving objects.
Revised: As the name suggests, the aim of motion segmentation is to discover moving objects.

Original: One line of the work (Brox & Malik, 2010; Fragkiadaki et al., 2012; Ochs & Brox, 2012; Ochs et al., 2013; Lezama et al., 2011; Keuper et al., 2015) formulates motion as the point trajectory to take advantage of long-range temporal information so that segmentation results can be acquired by grouping the trajectories.
Revised: One line of the work (Brox & Malik, 2010; Fragkiadaki et al., 2012; Ochs & Brox, 2012; Ochs et al., 2013; Lezama et al., 2011; Keuper et al., 2015) formulates motion as the point trajectory to take advantage of long-range temporal information so that segmentation results can be acquired by grouping the trajectories.

Original: Later, deep learning methods take over the area (Tokmakov et al., 2017b;a; Xie et al., 2019; Yang et al., 2019; 2021a; Choudhury et al., 2022; Yang et al., 2021b;
Revised: Later, deep learning methods take over the area (Tokmakov et al., 2017b;a; Xie et al., 2019; Yang et al., 2019; 2021a; Choudhury et al., 2022; Yang et al., 2021b;

Original: Ye et al., 2022a; Wang et al., 2022).
Revised: Ye et al., 2022a; Wang et al., 2022).

Original: Tokmakov et al.
Revised: Tokmakov et al.

Original: (2017b) adopts a two-stream network that ingests both RGB and optical flow.
Revised: (2017b) adopts a two-stream network that ingests both RGB and optical flow.

Original: Then they realize a memory mechanism by the convolutional recurrent unit to enhance the visual cues.
Revised: Then they realize a memory mechanism by the convolutional recurrent unit to enhance the visual cues.

Original: CIS (Yang et al., 2019) achieves fully unsupervised motion segmentation which discards the supervision of annotated masks during training.
Revised: CIS (Yang et al., 2019) achieves fully unsupervised motion segmentation which discards the supervision of annotated masks during training.

Original: By formulating a min-max game of mutual information, the generator is asked to create foreground segments that are as unrelated as possible to the background.
Revised: By formulating a min-max game of mutual information, the generator is asked to create foreground segments that are as unrelated as possible to the background.

Original: AMD (Liu et al., 2021) minimizes the warp synthesis error to train appearance and motion pathways without any supervision.
Revised: AMD (Liu et al., 2021) minimizes the warp synthesis error to train appearance and motion pathways without any supervision.

Original: The most similar work to ours is MG (Yang et al., 2021a), which solely leverages the optical flow to separate the pixels via cross attention mechanism (Locatello et al., 2020).
Revised: The most similar work to ours is MG (Yang et al., 2021a), which solely leverages the optical flow to separate the pixels via cross attention mechanism (Locatello et al., 2020).

Original: Compared to MG (Yang et al., 2021a), we keep reserved on the module design and the training recipe to demonstrate the improvement brought by our method is purely from taking consecutive RGB frames instead of optical flow.
Revised: Compared to MG (Yang et al., 2021a), we keep reserved on the module design and the training recipe to demonstrate the improvement brought by our method is purely from taking consecutive RGB frames instead of optical flow.

Original: Recent work GWM (Choudhury et al., 2022) also utilizes RGB images and adopt the supervision from optical flow but their model ingests a single image to segment the foreground and fails to consider temporal coherency.
Revised: Recent work GWM (Choudhury et al., 2022) also utilizes RGB images and adopt the supervision from optical flow but their model ingests a single image to segment the foreground and fails to consider temporal coherency.

Original: Object Discovery.
Revised: Object Discovery.

Original: There is rich literature on identifying salient objects without explicit supervision, known as object discovery.
Revised: There is rich literature on identifying salient objects without explicit supervision, known as object discovery.

Original: There exist a series of works that aim to learn object-centric representations in images (Locatello et al., 2020; Lin et al., 2020; Jiang & Ahn, 2020; Greff et al., 2019;
Revised: There exist a series of works that aim to learn object-centric representations in images (Locatello et al., 2020; Lin et al., 2020; Jiang & Ahn, 2020; Greff et al., 2019;

Original: Emami et al., 2021; Burgess et al., 2019; Crawford & Pineau, 2019; Engelcke et al., 2019; 2021).
Revised: Emami et al., 2021; Burgess et al., 2019; Crawford & Pineau, 2019; Engelcke et al., 2019; 2021).

Original: Typically, IODINE (Greff et al., 2019) develops iterative variational inference to separate different objects.
Revised: Typically, IODINE (Greff et al., 2019) develops iterative variational inference to separate different objects.

Original: Locatello et al.
Revised: Locatello et al.

Original: (2020) proposes slot attention to iteratively update latent object representations.
Revised: (2020) proposes slot attention to iteratively update latent object representations.

Original: Further, a line of works (Zablotskaia et al., 2020; Min et al., 2021; Kosiorek et al., 2018; Kipf et al., 2022; Jiang et al., 2019; Kabra et al., 2021; Crawford & Pineau, 2020; Besbinar & Frossard, 2021; Bear et al., 2020; Bao et al., 2022; Ye et al., 2022b;
Revised: Further, a line of works (Zablotskaia et al., 2020; Min et al., 2021; Kosiorek et al., 2018; Kipf et al., 2022; Jiang et al., 2019; Kabra et al., 2021; Crawford & Pineau, 2020; Besbinar & Frossard, 2021; Bear et al., 2020; Bao et al., 2022; Ye et al., 2022b;

Original: Yang et al., 2021a) extend object-centric learning to video domain.
Revised: Yang et al., 2021a) extend object-centric learning to video domain.

Original: Most of these approaches incorporate motion cues into the reconstruction task and perform well in moving object segmentation but perform poorly in processing static objects.
Revised: Most of these approaches incorporate motion cues into the reconstruction task and perform well in moving object segmentation but perform poorly in processing static objects.

Original: While in this work, we adopt slot attention to form optical flow reconstruction bottleneck and perceive both dynamic and static instances through RGB clip input in a temporally consistent way.
Revised: While in this work, we adopt slot attention to form optical flow reconstruction bottleneck and perceive both dynamic and static instances through RGB clip input in a temporally consistent way.

Original:  M OTION - INDUCTIVE O BJECT D ISCOVERY (MOD) M ODEL
Revised:  M OTION - INDUCTIVE O BJECT D ISCOVERY (MOD) M ODEL

Original: In this section, we detail our MOD model, which processes a set of consecutive RGB frames and automatically discovers the moving objects in the form of segmentation.
Revised: In this section, we detail our MOD model, which processes a set of consecutive RGB frames and automatically discovers the moving objects in the form of segmentation.

Original: An overview of the training procedure can be seen in Figure 2, where the visual features computed from individual frames are temporally fused, and randomly paired together, for decoding the optical flows between two corresponding frames.
Revised: An overview of the training procedure can be seen in Figure 2, where the visual features computed from individual frames are temporally fused, and randomly paired together, for decoding the optical flows between two corresponding frames.

Original: Taking inspiration from the motion grouping (Yang et al., 2021a), we also adopt a dual-layered representation for the output flow, with the foreground and background flows being reconstructed separately, and later composited with the inferred opacity masks (soft segmentation).
Revised: Taking inspiration from the motion grouping (Yang et al., 2021a), we also adopt a dual-layered representation for the output flow, with the foreground and background flows being reconstructed separately, and later composited with the inferred opacity masks (soft segmentation).

Original: In the following section, we will detail each key component in our proposed architecture.
Revised: In the following section, we will detail each key component in our proposed architecture.

Original: 3. S PATIAL - TEMPORAL V ISUAL E NCODER
Revised: 3. S PATIAL - TEMPORAL V ISUAL E NCODER

Original: To start with, our model takes a short video clip as input, i.e. , v = { x 1 , · · · , x T } , v ∈ R T × H × W × 3 , consisting of a set of RGB frames, the frame-wise visual representations are computed with a shared visual encoder Φ enc .
Revised: To start with, our model takes a short video clip as input, i.e. , v = { x 1 , · · · , x T } , v ∈ R T × H × W × 3 , consisting of a set of RGB frames, the frame-wise visual representations are computed with a shared visual encoder Φ enc .

Original: Formally, the output feature map o t at timestamp t (1 ≤ t ≤ T ) can be obtained:
Revised: Formally, the output feature map o t at timestamp t (1 ≤ t ≤ T ) can be obtained:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where h, w, d denotes the dimension of the height, width, and channel, respectively.
Revised: where h, w, d denotes the dimension of the height, width, and channel, respectively.

Original: Till this point, to build the temporal dependency between multiple visual frames, a global fusion module is introduced, via a standard Transformer Encoder layer:
Revised: Till this point, to build the temporal dependency between multiple visual frames, a global fusion module is introduced, via a standard Transformer Encoder layer:

Original: [Equation]
Revised: [Equation]

Original: where pe refers to the learnable spatial-temporal positional encodings, and the output ˜ o t ∈ R h × w ×
Revised: where pe refers to the learnable spatial-temporal positional encodings, and the output ˜ o t ∈ R h × w ×

Original: d remains the same dimension as input.
Revised: d remains the same dimension as input.

Original: By taking the multiple RGB frames as input, our proposed visual encoder can explicitly consider the temporal coherence within the video clip.
Revised: By taking the multiple RGB frames as input, our proposed visual encoder can explicitly consider the temporal coherence within the video clip.

Original: Note that , such seemingly simple design poses two critical differences from previous work on motion-driven object discovery (Yang et al., 2019; 2021a), where only frame-wise optical flow is adopted: First , using RGB frames as input can drastically reduce the computation latency at inference time.
Revised: Note that , such seemingly simple design poses two critical differences from previous work on motion-driven object discovery (Yang et al., 2019; 2021a), where only frame-wise optical flow is adopted: First , using RGB frames as input can drastically reduce the computation latency at inference time.

Original: The throughput of our model without computing dense optical flow reaches round 100 fps on a standard 32GB Tesla V100 GPU while prior works need to calculate flow at first.
Revised: The throughput of our model without computing dense optical flow reaches round 100 fps on a standard 32GB Tesla V100 GPU while prior works need to calculate flow at first.

Original: RGB provides more semantic information than flows for the model to exploit, i.e. , including not only the object’s shape but also its texture.
Revised: RGB provides more semantic information than flows for the model to exploit, i.e. , including not only the object’s shape but also its texture.

Original: Second , processing multiple frames contributes to the development of a sense of object permanence within the video clip, i.e. , the understanding that items or people still exist even when they cannot be perceived explicitly.
Revised: Second , processing multiple frames contributes to the development of a sense of object permanence within the video clip, i.e. , the understanding that items or people still exist even when they cannot be perceived explicitly.

Original: Therefore, even though the objects in videos may stop moving or be partially occluded at any time point, they can still be effectively segmented with the temporal cues.
Revised: Therefore, even though the objects in videos may stop moving or be partially occluded at any time point, they can still be effectively segmented with the temporal cues.

Original: 3. R ANDOM - PAIRED F RAME C OMPARATOR
Revised: 3. R ANDOM - PAIRED F RAME C OMPARATOR

Original: Till this point, we consider building the relative motion between any two visual frames within the video clip, from reference frame i to target frame j , i.e. , f i → j .
Revised: Till this point, we consider building the relative motion between any two visual frames within the video clip, from reference frame i to target frame j , i.e. , f i → j .

Original: In specific, we select the visual representation of the two corresponding frames, i.e. , ˜ o
Revised: In specific, we select the visual representation of the two corresponding frames, i.e. , ˜ o

Original: i and ˜ o j , concatenate them along the feature dimension, and feed it into a comparator module:
Revised: i and ˜ o j , concatenate them along the feature dimension, and feed it into a comparator module:

Original: [Equation]
Revised: [Equation]

Original: where Φ comp : R 2 d → R d consists of multiple deformable convolutional layers (Zhu et al., 2019) followed by a series of Transformer Encoders, that dynamically construct the feature representation for later estimating relative motion between frames while reducing the feature dimensions at the same time.
Revised: where Φ comp : R 2 d → R d consists of multiple deformable convolutional layers (Zhu et al., 2019) followed by a series of Transformer Encoders, that dynamically construct the feature representation for later estimating relative motion between frames while reducing the feature dimensions at the same time.

Original: Discussion.
Revised: Discussion.

Original: With such a design of random-paired frame comparator, our MOD is capable of modeling the relative motion between any two randomly sampled frames out of the video clip ( T frames), accounting for a total of T 2 frame pairs for forward, backward, single, or multi-step motions.
Revised: With such a design of random-paired frame comparator, our MOD is capable of modeling the relative motion between any two randomly sampled frames out of the video clip ( T frames), accounting for a total of T 2 frame pairs for forward, backward, single, or multi-step motions.

Original: Note that, we do not distinguish the case of i ̸
Revised: Note that, we do not distinguish the case of i ̸

Original: =
Revised: =

Original: j and i = j .
Revised: j and i = j .

Original: Specifically, the former encourages the model to discover the relative motion between two different frames, while the latter refers to an extreme case that neither objects nor camera is moving, and no motion cues are available, thus enforcing the model to discover objects via temporal coherence, i.e. , objects that moves in any frame along the video should also be discovered in static frames.
Revised: Specifically, the former encourages the model to discover the relative motion between two different frames, while the latter refers to an extreme case that neither objects nor camera is moving, and no motion cues are available, thus enforcing the model to discover objects via temporal coherence, i.e. , objects that moves in any frame along the video should also be discovered in static frames.

Original: 3. D UAL - LAYERED F LOW D ECODER
Revised: 3. D UAL - LAYERED F LOW D ECODER

Original: To decode the features into the form of optical flow, we adopt several slot attention layers (Locatello et al., 2020) with two learnable queries, i.e. , termed as slot vectors, iteratively attending the output visual features from the comparator, and decoded into the optical flow between any two frames with a dual-layered representation.
Revised: To decode the features into the form of optical flow, we adopt several slot attention layers (Locatello et al., 2020) with two learnable queries, i.e. , termed as slot vectors, iteratively attending the output visual features from the comparator, and decoded into the optical flow between any two frames with a dual-layered representation.

Original: In detail, a slot attention module acts similarly to a Transformer
Revised: In detail, a slot attention module acts similarly to a Transformer

Original: Decoder, with the only exception being that the normalisation is computed along the slot side, thus each slot competes to take over the pixels.
Revised: Decoder, with the only exception being that the normalisation is computed along the slot side, thus each slot competes to take over the pixels.

Original: In each iteration, given two slot vectors as S ∈ R 2 × d and visual feature maps ˜ o t ∈ R hw × d , we use three linear projections to compute the query , key and value , i.e. , Q ∈ R 2 × d , K, V ∈ R hw × d .
Revised: In each iteration, given two slot vectors as S ∈ R 2 × d and visual feature maps ˜ o t ∈ R hw × d , we use three linear projections to compute the query , key and value , i.e. , Q ∈ R 2 × d , K, V ∈ R hw × d .

Original: Thereafter, we can obtain the weights matrix W ∈ R 2 × hw and normalise along the slot dimension, i.e. ,
Revised: Thereafter, we can obtain the weights matrix W ∈ R 2 × hw and normalise along the slot dimension, i.e. ,

Original: [Equation]
Revised: [Equation]

Original: Then, we gain the next iteration’s slot vectors by aggregating the values V and passing them into a
Revised: Then, we gain the next iteration’s slot vectors by aggregating the values V and passing them into a

Original: Gated Recurrent Unit (GRU) (Cho et al., 2014), i.e. ,
Revised: Gated Recurrent Unit (GRU) (Cho et al., 2014), i.e. ,

Original: [Equation]
Revised: [Equation]

Original: GRU
Revised: GRU

Original: [Equation]
Revised: [Equation]

Original: inputs
Revised: inputs

Original: [Equation]
Revised: [Equation]

Original: AV
Revised: AV

Original: [Equation]
Revised: [Equation]

Original: states = S ) , where
Revised: states = S ) , where

Original: [Equation]
Revised: [Equation]

Original: We iterate the whole routing process for N times.
Revised: We iterate the whole routing process for N times.

Original: In this way, the entities of similar RGB and flow patterns are grouped together and distinct pixels are separated by two slots.
Revised: In this way, the entities of similar RGB and flow patterns are grouped together and distinct pixels are separated by two slots.

Original: Eventually, we broadcast the final outputted slots into G = { G s ∈ R h × w × d } 2 s =1 added with learnable spatial positional embeddings to construct the optical flow.
Revised: Eventually, we broadcast the final outputted slots into G = { G s ∈ R h × w × d } 2 s =1 added with learnable spatial positional embeddings to construct the optical flow.

Original: A flow decoder Φ dec consisting Transformer encoders and up-sampling layers takes the slot grids as input, and outputs dual layers of optical flows { (cid:101) I s ∈
Revised: A flow decoder Φ dec consisting Transformer encoders and up-sampling layers takes the slot grids as input, and outputs dual layers of optical flows { (cid:101) I s ∈

Original: R
Revised: R

Original: H
Revised: H

Original: × W × 3 } 2 s =1 1 and their opacity weights {
Revised: × W × 3 } 2 s =1 1 and their opacity weights {

Original: α s ∈ R H
Revised: α s ∈ R H

Original: × W × 1 } 2 s =1 :
Revised: × W × 1 } 2 s =1 :

Original: [Equation]
Revised: [Equation]

Original: where α s ∈ [0 , 1] H × W × 1 is normalized across two slots via softmax function.
Revised: where α s ∈ [0 , 1] H × W × 1 is normalized across two slots via softmax function.

Original: Noted that thanks to the softmax function, the alpha value for foreground could be close to zero when the foreground object is missing in part of the video.
Revised: Noted that thanks to the softmax function, the alpha value for foreground could be close to zero when the foreground object is missing in part of the video.

Original: For a given frame pair ( i, j ) , their relative flow (cid:101) I i
Revised: For a given frame pair ( i, j ) , their relative flow (cid:101) I i

Original: → j can be computed via:
Revised: → j can be computed via:

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes the element-wise multiplication.
Revised: where ⊗ denotes the element-wise multiplication.

Original: At inference time, we adopt the binarized opacity weights α s as the object segmentation masks.
Revised: At inference time, we adopt the binarized opacity weights α s as the object segmentation masks.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: In this section, we describe the training procedure for the proposed model, on the raw videos without using manual annotations for the object segmentations.
Revised: In this section, we describe the training procedure for the proposed model, on the raw videos without using manual annotations for the object segmentations.

Original: In general, the training loss is composed of three components, namely, flow reconstruction, temporal consistency, and entropy minimisation.
Revised: In general, the training loss is composed of three components, namely, flow reconstruction, temporal consistency, and entropy minimisation.

Original: Flow Reconstruction.
Revised: Flow Reconstruction.

Original: As the main objective for optimisation, we use the flow reconstruction, where we adopt an off-the-shelf optical flow estimator, for example, RAFT (Teed & Deng, 2020), to estimate the flow between any two frames in the video.
Revised: As the main objective for optimisation, we use the flow reconstruction, where we adopt an off-the-shelf optical flow estimator, for example, RAFT (Teed & Deng, 2020), to estimate the flow between any two frames in the video.

Original: We minimise the discrepancy between the dual-layer flow reconstruction and the output from the existing flow estimator:
Revised: We minimise the discrepancy between the dual-layer flow reconstruction and the output from the existing flow estimator:

Original: [Equation]
Revised: [Equation]

Original: where Ω = { 1 , · · · , H }×{ 1 , · · · , W } represents the spatial lattice, and a |·| 2 denotes the L2 norm.
Revised: where Ω = { 1 , · · · , H }×{ 1 , · · · , W } represents the spatial lattice, and a |·| 2 denotes the L2 norm.

Original: In practice, the reconstruction constraint for zero flow when i =
Revised: In practice, the reconstruction constraint for zero flow when i =

Original: j is dropped due to the distribution gap between dynamic flow and static flow, resulting in the divergence of the training.
Revised: j is dropped due to the distribution gap between dynamic flow and static flow, resulting in the divergence of the training.

Original: Note that , flow estimator is only used for model training, at inference time, our proposed architecture directly processes the RGB video clips.
Revised: Note that , flow estimator is only used for model training, at inference time, our proposed architecture directly processes the RGB video clips.

Original: Temporal Consistency.
Revised: Temporal Consistency.

Original: In order to build up temporal consistency within the input video, the pair of motion embeddings f i → j , f i → k that starts from the same reference frame i , are passed through the flow decoder to reconstruct the optical flow between two corresponding frames.
Revised: In order to build up temporal consistency within the input video, the pair of motion embeddings f i → j , f i → k that starts from the same reference frame i , are passed through the flow decoder to reconstruct the optical flow between two corresponding frames.

Original: Note that, as 1 ≤ j, k ≤ T are randomly sampled at every training iteration, the output flow will refer to the motion at a different pace.
Revised: Note that, as 1 ≤ j, k ≤ T are randomly sampled at every training iteration, the output flow will refer to the motion at a different pace.

Original: However, the predicted alpha weights for flow composition denote the soft segmentation for the same objects in the i -th frame, thus remaining consistent.
Revised: However, the predicted alpha weights for flow composition denote the soft segmentation for the same objects in the i -th frame, thus remaining consistent.

Original: In specific, the two inferred masks { α si → j } 2 s =1 , { α si → k } 2 s =1 are enforced to pull closer by minimising mean-squared error L cons , i.e. ,
Revised: In specific, the two inferred masks { α si → j } 2 s =1 , { α si → k } 2 s =1 are enforced to pull closer by minimising mean-squared error L cons , i.e. ,

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: where L i cons =
Revised: where L i cons =

Original: [Equation]
Revised: [Equation]

Original: | Ω |
Revised: | Ω |

Original: [Equation]
Revised: [Equation]

Original: Entropy Minimisation.
Revised: Entropy Minimisation.

Original: Lastly, we impose a pixel-wise entropy regularisation on inferred masks, that is zero if the alpha channels are one-hot, and maximum when they are of equal probability.
Revised: Lastly, we impose a pixel-wise entropy regularisation on inferred masks, that is zero if the alpha channels are one-hot, and maximum when they are of equal probability.

Original: Intuitively, this helps encourage the masks to be binary, which aligns with our goal in obtaining segmentation masks:
Revised: Intuitively, this helps encourage the masks to be binary, which aligns with our goal in obtaining segmentation masks:

Original: [Equation]
Revised: [Equation]

Original: Total Loss.
Revised: Total Loss.

Original: Accordingly, we rewrite aforementioned reconstruction loss L recon and entropy regulatization L entro in summed version:
Revised: Accordingly, we rewrite aforementioned reconstruction loss L recon and entropy regulatization L entro in summed version:

Original: [Equation]
Revised: [Equation]

Original: The total loss for training our model can thus be computed as:
Revised: The total loss for training our model can thus be computed as:

Original: [Equation]
Revised: [Equation]

Original: where we set λ r = 100 , λ e = λ c = 0 .
Revised: where we set λ r = 100 , λ e = λ c = 0 .

Original: at the beginning of the training.
Revised: at the beginning of the training.

Original: We notice the model to be fairly robust to these hyper-parameters.
Revised: We notice the model to be fairly robust to these hyper-parameters.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: In the experiments setup, we first introduce the benchmarks and then elaborate on implementation details.
Revised: In the experiments setup, we first introduce the benchmarks and then elaborate on implementation details.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We benchmark on three popular datasets designed for video object segmentation.
Revised: We benchmark on three popular datasets designed for video object segmentation.

Original: DAVIS2016 (Perazzi et al., 2016) consists of 50 high quality videos, 3455 frames in total.
Revised: DAVIS2016 (Perazzi et al., 2016) consists of 50 high quality videos, 3455 frames in total.

Original: Every frame is annotated with a pixel-accurate segmentation mask.
Revised: Every frame is annotated with a pixel-accurate segmentation mask.

Original: SegTrackv2 (Li et al., 2013) contains 14 sequences and 947 fully-annotated frames.
Revised: SegTrackv2 (Li et al., 2013) contains 14 sequences and 947 fully-annotated frames.

Original: Each sequence involves 1-6 moving objects and presents challenges including motion blur, appearance change, complex deformation, occlusion, slow motion, and interacting objects.
Revised: Each sequence involves 1-6 moving objects and presents challenges including motion blur, appearance change, complex deformation, occlusion, slow motion, and interacting objects.

Original: FBMS-59 (Ochs et al., 2013) has 59 sequences with greatly varied resolution and annotates every 20th frame.
Revised: FBMS-59 (Ochs et al., 2013) has 59 sequences with greatly varied resolution and annotates every 20th frame.

Original: Many sequences contain multiple moving objects.
Revised: Many sequences contain multiple moving objects.

Original: Following previous evaluation metric (Yang et al., 2019; Xie et al., 2022), we merge objects of SegTrackv2 and FBMS59 into one single object for video object segmentation.
Revised: Following previous evaluation metric (Yang et al., 2019; Xie et al., 2022), we merge objects of SegTrackv2 and FBMS59 into one single object for video object segmentation.

Original: We evaluate the pixel-wise segmentation through Jaccard index J , also called Intersection over Union (IoU).
Revised: We evaluate the pixel-wise segmentation through Jaccard index J , also called Intersection over Union (IoU).

Original: Following prior arts (Yang et al., 2019; 2021a), we compute the mean per frame over the test set and merge multi-object annotation into single unified segmentation.
Revised: Following prior arts (Yang et al., 2019; 2021a), we compute the mean per frame over the test set and merge multi-object annotation into single unified segmentation.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data input, we sample T = 7 consecutive frames as the input clip.
Revised: For data input, we sample T = 7 consecutive frames as the input clip.

Original: Each frame is resized to 192 × 384 and the estimated optical flow is computed by RAFT (Teed & Deng, 2020), which is pre-trained on the synthetic dataset (Mayer et al., 2016).
Revised: Each frame is resized to 192 × 384 and the estimated optical flow is computed by RAFT (Teed & Deng, 2020), which is pre-trained on the synthetic dataset (Mayer et al., 2016).

Original: To compute spatial-temporal visual representation, we adopt the first three stages of a SwinV2T as the frame encoder Φ enc , which is then followed by a standard Transformer Encoders withheads (Vaswani et al., 2017) as temporal fusion module Φ temp .
Revised: To compute spatial-temporal visual representation, we adopt the first three stages of a SwinV2T as the frame encoder Φ enc , which is then followed by a standard Transformer Encoders withheads (Vaswani et al., 2017) as temporal fusion module Φ temp .

Original: For the frame comparator Φ comp , we use two deformable convolutional layers in the company with three standard Transformer Encoders with 8 heads to process the pixel transformation.
Revised: For the frame comparator Φ comp , we use two deformable convolutional layers in the company with three standard Transformer Encoders with 8 heads to process the pixel transformation.

Original: Then, we choose N = 5 iteration in total for the iterative routing in slot attention.
Revised: Then, we choose N = 5 iteration in total for the iterative routing in slot attention.

Original: Lastly, we utilize three stages of SwinV2 blocks with the linear patch expanding layers as the flow decoder Φ dec (Cao et al., 2021).
Revised: Lastly, we utilize three stages of SwinV2 blocks with the linear patch expanding layers as the flow decoder Φ dec (Cao et al., 2021).

Original: As for training, we adopt AdamW optimizer (Loshchilov & Hutter, 2018) with learning rate 4 × 10 − 5 .
Revised: As for training, we adopt AdamW optimizer (Loshchilov & Hutter, 2018) with learning rate 4 × 10 − 5 .

Original: The model is trained from scratch without any pretrained weights, for a total of 300k iterations.
Revised: The model is trained from scratch without any pretrained weights, for a total of 300k iterations.

Original: At inference time, we adopt the overlapping temporal sliding window to ensemble the segmentation masks.
Revised: At inference time, we adopt the overlapping temporal sliding window to ensemble the segmentation masks.

Original: We average the resultant masks obtained by the whole temporal segments.
Revised: We average the resultant masks obtained by the whole temporal segments.

Original: We propose two protocols to evaluate our results.
Revised: We propose two protocols to evaluate our results.

Original: Besides measuring the masks without any post-processing, we also apply test-time adaptation with the help of the self-supervised DINO-pretrained ViT (Caron et al., 2021).
Revised: Besides measuring the masks without any post-processing, we also apply test-time adaptation with the help of the self-supervised DINO-pretrained ViT (Caron et al., 2021).

Original: Without any fine-tuning, the pretrained ViT can propagate the masks as noisy annotations to the whole frames in the same manner as CRW (Jabri et al., 2020).
Revised: Without any fine-tuning, the pretrained ViT can propagate the masks as noisy annotations to the whole frames in the same manner as CRW (Jabri et al., 2020).

Original: We refine the masks further with CRF (Lafferty et al., 2001).
Revised: We refine the masks further with CRF (Lafferty et al., 2001).

Original: For more detailed technical information, please refer to the supplementary materials.
Revised: For more detailed technical information, please refer to the supplementary materials.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we compare primarily with several top-performing approaches trained without human annotations, for example, OCLR (Xie et al., 2022),
Revised: In this section, we compare primarily with several top-performing approaches trained without human annotations, for example, OCLR (Xie et al., 2022),

Original: MG (Yang et al., 2021a), CIS (Yang et al., 2019), etc.
Revised: MG (Yang et al., 2021a), CIS (Yang et al., 2019), etc .

Original: However, as the architecture, modality, input resolution, and post-processing protocols are all different, we try our best to conduct the comparison as fairly as possible.
Revised: However, as the architecture, modality, input resolution, and post-processing protocols are all different, we try our best to conduct the comparison as fairly as possible.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: We conduct all ablation studies on DAVIS2016 and vary one variable each time, as shown in Table 1.
Revised: We conduct all ablation studies on DAVIS2016 and vary one variable each time, as shown in Table 1.

Original: Temporal Fusion Φ temp and Frame Comparator Φ comp .
Revised: Temporal Fusion Φ temp and Frame Comparator Φ comp .

Original: As shown by Ours-A and Our-C, the performance degrades significantly without temporal fusion, demonstrating the importance of building up global temporal dependency.
Revised: As shown by Ours-A and Our-C, the performance degrades significantly without temporal fusion, demonstrating the importance of building up global temporal dependency.

Original: Also, indicated by Ours-B, we find the model fails to converge when removing the component of frame comparator Φ comp .
Revised: Also, indicated by Ours-B, we find the model fails to converge when removing the component of frame comparator Φ comp .

Original: It meets expectations because frame comparator Φ comp is the sole module in charge of relative motion estimation.
Revised: It meets expectations because frame comparator Φ comp is the sole module in charge of relative motion estimation.

Original: Without it, the model cannot reconstruct optical flow thus leading to divergence.
Revised: Without it, the model cannot reconstruct optical flow thus leading to divergence.

Original: Number of Frames T .
Revised: Number of Frames T .

Original: While comparing Ours-A, Ours-D, and Ours-E, there is a clear trend that increasing the frame number boosts the segmentation quality, which coincides with our intuition that incorporating a wider temporal receptive field can enhance the sense of temporal coherence and object permanence.
Revised: While comparing Ours-A, Ours-D, and Ours-E, there is a clear trend that increasing the frame number boosts the segmentation quality, which coincides with our intuition that incorporating a wider temporal receptive field can enhance the sense of temporal coherence and object permanence.

Original: Due to the limited computational memory, we only set T = 7 as the maximum frame number in the paper.
Revised: Due to the limited computational memory, we only set T = 7 as the maximum frame number in the paper.

Original: A promising performance is expected when inputting more frames.
Revised: A promising performance is expected when inputting more frames.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Temporal Consistency L cons and Entropy Regularisation L entro .
Revised: Temporal Consistency L cons and Entropy Regularisation L entro .

Original: Lastly, comparing Ours-G and
Revised: Lastly, comparing Ours-G and

Original: Ours-A, we observe that the performance increases considerably with temporal consistency.
Revised: Ours-A, we observe that the performance increases considerably with temporal consistency.

Original: It manifests the validity of our design motivation, temporal consistency conduces to persistently tracking the object.
Revised: It manifests the validity of our design motivation, temporal consistency conduces to persistently tracking the object.

Original: The entropy regularisation is also indispensable shown by Ours-H and Ours-A.
Revised: The entropy regularisation is also indispensable shown by Ours-H and Ours-A.

Original: 5. C OMPARISON WITH S TATE - OF - THE - ART
Revised: 5. C OMPARISON WITH S TATE - OF - THE - ART

Original: We show the comparison with state-of-the-art in Table 2.
Revised: We show the comparison with state-of-the-art in Table 2.

Original: On DAVIS2016, MOD achieves 73.9% mIOU without any post-processing, exceeding MG
Revised: On DAVIS2016, MOD achieves 73.9% mIOU without any post-processing, exceeding MG

Original: (Yang et al., 2021a) by a large margin (+5.8%).
Revised: (Yang et al., 2021a) by a large margin (+5.8%).

Original: Compared to the latest method OCLR (Xie et al., 2022) which fabricates a synthesized dataset to train its model, our method still surpasses it only using the information of the DAVIS dataset itself.
Revised: Compared to the latest method OCLR (Xie et al., 2022) which fabricates a synthesized dataset to train its model, our method still surpasses it only using the information of the DAVIS dataset itself.

Original: On another two benchmarks SegTrackv2 and FBMS-59, MOD also beats MG (Yang et al., 2021a), which only leverages single-step flows to decompose foreground and background, by +3.6% and +8.2%, respectively.
Revised: On another two benchmarks SegTrackv2 and FBMS-59, MOD also beats MG (Yang et al., 2021a), which only leverages single-step flows to decompose foreground and background, by +3.6% and +8.2%, respectively.

Original: The superior experimental results demonstrate that our methodology of multiframe reasoning benefits moving object discovery.
Revised: The superior experimental results demonstrate that our methodology of multiframe reasoning benefits moving object discovery.

Original: Furthermore, equipped with DINO-pretrained
Revised: Furthermore, equipped with DINO-pretrained

Original: ViT, a further performance gain is observed on all three benchmarks, which is even more competitive with current supervised approaches.
Revised: ViT, a further performance gain is observed on all three benchmarks, which is even more competitive with current supervised approaches.

Original: 5. Q UALITATIVE R ESULTS
Revised: 5. Q UALITATIVE R ESULTS

Original: In Figure 3, we present several qualitative illustrations of the model.
Revised: In Figure 3, we present several qualitative illustrations of the model.

Original: It can be seen that our results are robust to the noticeable background flow signal (drift-chicane sequence in the second column) and estimate more accurate boundaries when single-step foreground flow cannot represent exact object shape (breakdance and dance-twirl sequences in middle) compared to MG (Yang et al., 2021a).
Revised: It can be seen that our results are robust to the noticeable background flow signal (drift-chicane sequence in the second column) and estimate more accurate boundaries when single-step foreground flow cannot represent exact object shape (breakdance and dance-twirl sequences in middle) compared to MG (Yang et al., 2021a).

Original: It demonstrates inferring the masks by associating a bunch of RGB features well resolves the limitation of the usage of flow.
Revised: It demonstrates inferring the masks by associating a bunch of RGB features well resolves the limitation of the usage of flow.

Original: Moreover, in virtue of temporally consistent cues, our model handles occlusion well shown in the libby sequence at the rightmost column, which could be hard for the flow-only method to maintain the object shape constantly.
Revised: Moreover, in virtue of temporally consistent cues, our model handles occlusion well shown in the libby sequence at the rightmost column, which could be hard for the flow-only method to maintain the object shape constantly.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: / o p o
Revised: / o p o

Original: [Figure]
Revised: [Figure]

Original: t p r o c
Revised: t p r o c

Original: [Figure]
Revised: [Figure]

Original: ( t e s t - t
Revised: ( t e s t - t

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e a d a p
Revised: e a d a p

Original: [Figure]
Revised: [Figure]

Original: 5. L IMITATIONS
Revised: 5. L IMITATIONS

Original: Though we demonstrate that associating multiple frames stimulates the comprehensive sense of objectness, which is proved by the superior experimental performance across the prior arts, there still exist limitations and room for improvement.
Revised: Though we demonstrate that associating multiple frames stimulates the comprehensive sense of objectness, which is proved by the superior experimental performance across the prior arts, there still exist limitations and room for improvement.

Original: First , our method uses a number of consecutive frames, which challenges computational memory.
Revised: First , our method uses a number of consecutive frames, which challenges computational memory.

Original: How to utilize pretrained features for reducing training expenses would be meaningful.
Revised: How to utilize pretrained features for reducing training expenses would be meaningful.

Original: Second , how to segment multiple objects remains unresolved.
Revised: Second , how to segment multiple objects remains unresolved.

Original: We display a preliminary result in supplementary material.
Revised: We display a preliminary result in supplementary material.

Original: It will be promising to exploit the semantic information from RGB to discriminate different foreground objects.
Revised: It will be promising to exploit the semantic information from RGB to discriminate different foreground objects.

Original: We leave them as the feature work.
Revised: We leave them as the feature work.

Original: Despite these limitations, the approach has convincingly manifested the value of considering textural information and processing RGB frames as a whole.
Revised: Despite these limitations, the approach has convincingly manifested the value of considering textural information and processing RGB frames as a whole.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a self-supervised model for video object discovery.
Revised: In this paper, we propose a self-supervised model for video object discovery.

Original: The model takes a set of consecutive RGB frames as input and generates the segmentation mask for the moving objects in the video.
Revised: The model takes a set of consecutive RGB frames as input and generates the segmentation mask for the moving objects in the video.

Original: At training time, the model is tasked to reconstruct the optical flow between any pair of frames, through a layered representation with the opacity channels being treated as the segmentation.
Revised: At training time, the model is tasked to reconstruct the optical flow between any pair of frames, through a layered representation with the opacity channels being treated as the segmentation.

Original: To encourage the model to capture the objects even when they may be static at a certain time point, a temporal consistency loss is enforced on the inferred masks on the randomlypaired frames.
Revised: To encourage the model to capture the objects even when they may be static at a certain time point, a temporal consistency loss is enforced on the inferred masks on the randomlypaired frames.

Original: As a consequence, we demonstrate superior performance over previous state-of-theart methods on three public video segmentation datasets (DAVIS2016, SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow.
Revised: As a consequence, we demonstrate superior performance over previous state-of-theart methods on three public video segmentation datasets (DAVIS2016, SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow.

Original: A M ORE I MPLEMENTATION D ETAILS
Revised: A M ORE I MPLEMENTATION D ETAILS

Original: In this section, we list the architecture details and training settings.
Revised: In this section, we list the architecture details and training settings.

Original: Codes and models will be released publicly.
Revised: Codes and models will be released publicly.

Original: A. V ISUAL E NCODER
Revised: A. V ISUAL E NCODER

Original: The visual encoder contains the first three stages of Swin-Tiny V2.
Revised: The visual encoder contains the first three stages of Swin-Tiny V2.

Original: We tabulate the workflow in
Revised: We tabulate the workflow in

Original: Table 3.
Revised: Table 3.

Original: [Table]
Revised: [Table]

Original: A. F RAME C OMPARATOR
Revised: A. F RAME C OMPARATOR

Original: The frame comparator possesses two deformable convolutional layers, with ReLU operation in between and three Transformer Encoder blocks.
Revised: The frame comparator possesses two deformable convolutional layers, with ReLU operation in between and three Transformer Encoder blocks.

Original: We tabulate the workflow in Table 4.
Revised: We tabulate the workflow in Table 4.

Original: [Table]
Revised: [Table]

Original: DeformConv
Revised: DeformConv

Original: [Table]
Revised: [Table]

Original: ReLU -
Revised: ReLU -

Original: [Table]
Revised: [Table]

Original: DeformConv
Revised: DeformConv

Original: [Table]
Revised: [Table]

Original: 12 ×
Revised: 12 ×

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: h =
Revised: h =

Original: [Table]
Revised: [Table]

Original: ×
Revised: ×

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. F LOW D ECODER
Revised: A. F LOW D ECODER

Original: The frame comparator consists of three stages of SwinV2 block added with linear expanding layers.
Revised: The frame comparator consists of three stages of SwinV2 block added with linear expanding layers.

Original: We tabulate the workflow in Table 5.
Revised: We tabulate the workflow in Table 5.

Original: A
Revised: A

Original: . T RAINING D ETAILS
Revised: . T RAINING D ETAILS

Original: For all datasets, we train with a batch size of 2.
Revised: For all datasets, we train with a batch size of 2.

Original: To train more efficiently, we sample three flow pairs ( i → j ) for a given frame i ; one is static replication i =
Revised: To train more efficiently, we sample three flow pairs ( i → j ) for a given frame i ; one is static replication i =

Original: j , another two are motion pair i ̸
Revised: j , another two are motion pair i ̸

Original: = j .
Revised: = j .

Original: We apply temporal consistency first on the masks from two motion pairs, then pull the static mask to the average of two dynamic masks closer.
Revised: We apply temporal consistency first on the masks from two motion pairs, then pull the static mask to the average of two dynamic masks closer.

Original: We linearly warm up the learning rate for the first 1k iterations.
Revised: We linearly warm up the learning rate for the first 1k iterations.

Original: Besides, for every 100k iterations, we decay the learning rate by half and increase the scale of temporal consistency λ c and entropy regularisation λ e by the factor of 5 .
Revised: Besides, for every 100k iterations, we decay the learning rate by half and increase the scale of temporal consistency λ c and entropy regularisation λ e by the factor of 5 .

Original: In the default setting, we train for about 3 days on 8 standard Tesla V100 GPUs with 32GB memory each.
Revised: In the default setting, we train for about 3 days on 8 standard Tesla V100 GPUs with 32GB memory each.

Original: [Table]
Revised: [Table]

Original: A. T EST - TIME A DAPTATION
Revised: A. T EST - TIME A DAPTATION

Original: Inspired by OCLR (Xie et al., 2022), we adopt test-time adaptation based on RGB sequence to enhance appearance consistency.
Revised: Inspired by OCLR (Xie et al., 2022), we adopt test-time adaptation based on RGB sequence to enhance appearance consistency.

Original: In detail, we follow existing works on self-supervised tracking (Lai et al., 2020; Jabri et al., 2020) to propagate object masks across time span.
Revised: In detail, we follow existing works on self-supervised tracking (Lai et al., 2020; Jabri et al., 2020) to propagate object masks across time span.

Original: The whole process consists of three steps.
Revised: The whole process consists of three steps.

Original: First, we extract RGB features of each frame with a DINO-pretrained ViT encoder.
Revised: First, we extract RGB features of each frame with a DINO-pretrained ViT encoder.

Original: Then, we select key frames for object mask propagation.
Revised: Then, we select key frames for object mask propagation.

Original: Finally, we calculate the affinity matrix between frames and perform mask propagation.
Revised: Finally, we calculate the affinity matrix between frames and perform mask propagation.

Original: DINO Feature Extraction.
Revised: DINO Feature Extraction.

Original: Given a video sequence v = { x 1 , ..., x T } , v ∈ R T × H × W × 3 , we use
Revised: Given a video sequence v = { x 1 , ..., x T } , v ∈ R T × H × W × 3 , we use

Original: DINO pretrained ViT-Small encoder with patch size 8 × 8 to extract features:
Revised: DINO pretrained ViT-Small encoder with patch size 8 × 8 to extract features:

Original: [Equation]
Revised: [Equation]

Original: (14) where h = H// 8 and w = W// 8 .
Revised: (14) where h = H// 8 and w = W// 8 .

Original: The extracted features will be used in the mask propagation step.
Revised: The extracted features will be used in the mask propagation step.

Original: Key Frame Selection.
Revised: Key Frame Selection.

Original: Given video v , our model predicts object mask of each frame as m = { α 1 , ..., α T } , m ∈ R T × H × W × 1 .
Revised: Given video v , our model predicts object mask of each frame as m = { α 1 , ..., α T } , m ∈ R T × H × W × 1 .

Original: Since the video frames are continuous along the temporal dimension, it is practical to propagate object masks between neighboring frames.
Revised: Since the video frames are continuous along the temporal dimension, it is practical to propagate object masks between neighboring frames.

Original: The propagation operation is the same as Jabri et al.
Revised: The propagation operation is the same as Jabri et al.

Original: (2020), the only difference is that we have no ground-truth mask for reference.
Revised: (2020), the only difference is that we have no ground-truth mask for reference.

Original: Therefore, we need to design a mechanism to select object masks of high confidence.
Revised: Therefore, we need to design a mechanism to select object masks of high confidence.

Original: To do this, we measure the temporal coherence of predicted object masks for key frame selection.
Revised: To do this, we measure the temporal coherence of predicted object masks for key frame selection.

Original: Specifically, for each timestamp t ∈ { 3 , ..., T − 2 } , we can calculate four propagated masks as:
Revised: Specifically, for each timestamp t ∈ { 3 , ..., T − 2 } , we can calculate four propagated masks as:

Original: [Equation]
Revised: [Equation]

Original: where ‘Mask-prop’ denotes the propagation operation.
Revised: where ‘Mask-prop’ denotes the propagation operation.

Original: Then we calculate the average IoU between the original mask and propagated masks as the confidence score:
Revised: Then we calculate the average IoU between the original mask and propagated masks as the confidence score:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The calculated s t measures the coherency between α t and its neighbors.
Revised: The calculated s t measures the coherency between α t and its neighbors.

Original: Our empirical studies show that it serves as a reliable signal for key frame selection.
Revised: Our empirical studies show that it serves as a reliable signal for key frame selection.

Original: Object Mask Propagation.
Revised: Object Mask Propagation.

Original: We select timestamps with Top- k % confidence score as the key reference frames ( k = 15 on DAVIS2016, k = 25 on SegTrackv2, k = 10 on FBMS-59).
Revised: We select timestamps with Top- k % confidence score as the key reference frames ( k = 15 on DAVIS2016, k = 25 on SegTrackv2, k = 10 on FBMS-59).

Original: Then we iteratively propagate the object masks with a neighbor temporal window size n = 7 .
Revised: Then we iteratively propagate the object masks with a neighbor temporal window size n = 7 .

Original: Compared to conventional semi-supervised object segmentation which only relies on the first frame as key frame, we have multiple key frames on different temporal positions to correct the accumulated propagation error.
Revised: Compared to conventional semi-supervised object segmentation which only relies on the first frame as key frame, we have multiple key frames on different temporal positions to correct the accumulated propagation error.

Original: In this way, the propagation enhances object permanence across time and boosts performance.
Revised: In this way, the propagation enhances object permanence across time and boosts performance.

Original: B R ESULTS B REAKDOWN
Revised: B R ESULTS B REAKDOWN

Original: We include a specific result breakdown in this section.
Revised: We include a specific result breakdown in this section.

Original: We show the per-sequence results on
Revised: We show the per-sequence results on

Original: DAVIS2016 in Table 6, SegTrackv2 in Table 7 and FBMS-59 in Table 8.
Revised: DAVIS2016 in Table 6, SegTrackv2 in Table 7 and FBMS-59 in Table 8.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sequence
Revised: Sequence

Original: [Table]
Revised: [Table]

Original: camel
Revised: camel

Original: [Table] [Table]
Revised: [Table] [Table]

Original: cars
Revised: cars

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: cars
Revised: cars

Original: [Table]
Revised: [Table]

Original: cars 81.
Revised: cars 81.

Original: [Table]
Revised: [Table]

Original: cats 80.
Revised: cats 80.

Original: cats 62.
Revised: cats 62.

Original: [Table]
Revised: [Table]

Original: cats 40.
Revised: cats 40.

Original: dogs 70.
Revised: dogs 70.

Original: dogs 62.
Revised: dogs 62.

Original: farm 86.
Revised: farm 86.

Original: giraffes 38.
Revised: giraffes 38.

Original: goats 44.
Revised: goats 44.

Original: horses 64.
Revised: horses 64.

Original: horses 68.
Revised: horses 68.

Original: horses 43.
Revised: horses 43.

Original: lion 60.
Revised: lion 60.

Original: marple 74.
Revised: marple 74.

Original: marple 65.
Revised: marple 65.

Original: marple 79.
Revised: marple 79.

Original: marple 76.
Revised: marple 76.

Original: marple 72.
Revised: marple 72.

Original: marple 87.
Revised: marple 87.

Original: people 76.
Revised: people 76.

Original: people 72.
Revised: people 72.

Original: people 80.
Revised: people 80.

Original: rabbits 50.
Revised: rabbits 50.

Original: rabbits 39.
Revised: rabbits 39.

Original: rabbits 47.
Revised: rabbits 47.

Original: tennis 63.
Revised: tennis 63.

Original: frame avg.
Revised: frame avg.

Original: [Table]
Revised: [Table]

Original: C M ORE Q UANTITATIVE R ESULTS
Revised: C M ORE Q UANTITATIVE R ESULTS

Original: Besides RAFT flow estimator, we consider a fully unsupervised model ARFlow (Liu et al., 2020) shown in Table 9.
Revised: Besides RAFT flow estimator, we consider a fully unsupervised model ARFlow (Liu et al., 2020) shown in Table 9.

Original: The inferior result brought from ARFlow demonstrates that a good optical flow quality lays strong foundation for the success of the object discovery, which aligns with our motivation.
Revised: The inferior result brought from ARFlow demonstrates that a good optical flow quality lays strong foundation for the success of the object discovery, which aligns with our motivation.

Original: D M ORE Q UALITATIVE R ESULTS
Revised: D M ORE Q UALITATIVE R ESULTS

Original: We visualize the segmentation results when initializing 5 slots vectors in Figure 4.
Revised: We visualize the segmentation results when initializing 5 slots vectors in Figure 4.

Original: The results are promising, where each slot groups similar part under the only supervision of motion.
Revised: The results are promising, where each slot groups similar part under the only supervision of motion.

Original: We believe more supervision like semantics (Caron et al., 2021) and depth (Ranftl et al., 2021) would lead to better segmentation.
Revised: We believe more supervision like semantics (Caron et al., 2021) and depth (Ranftl et al., 2021) would lead to better segmentation.

Original: We show more qualitative results of SegTrackv2 and FBMS-59 in Figure 5 and
Revised: We show more qualitative results of SegTrackv2 and FBMS-59 in Figure 5 and

Original: Figure 6.
Revised: Figure 6.

Original: Besides,
Revised: Besides,

Original: [Figure]
Revised: [Figure]

Original: ( w / o p o s t p r o c . )
Revised: ( w / o p o s t p r o c . )

Original: [Figure]
Revised: [Figure]

Original: ( t e s t - t i m e a d a p . )
Revised: ( t e s t - t i m e a d a p . )

Original: [Figure]
Revised: [Figure]