Segmenting Moving Objects via an Object-Centric Layered Representation
Anonymous Author(s)
AffiliationAddress email
Abstract
The objective of this paper is a model that is able to discover, track and segmentmultiple moving objects in a video.
We make four contributions: First, we introducean object-centric segmentation model with a depth-ordered layer representation.
This is implemented using a variant of the transformer architecture that ingestsoptical flow, where each query vector specifies an object and its layer for theentire video.
The model can effectively discover multiple moving objects andhandle mutual occlusions; Second, we introduce a scalable pipeline for generatingmulti-object synthetic training data via layer compositions, which is used to trainour proposed model, significantly reducing the requirements for labour-intensiveannotations, and supporting Sim2Real generalisation; Third, we conduct thoroughablation studies, showing that the model is able to learn object permanence andtemporal shape consistency, and is able to predict amodal segmentation masks;
Fourth, our synthetic-supervised model is evaluated on standard video segmentation benchmarks, DAVIS, MoCA, SegTrack, FBMS-59, achieving state-of-the-artperformance among existing methods that do not rely on any manual annotations.
With test-time adaptation, we observe further performance boosts.
Introduction
Humans have the ability to discover and segment moving objects in videos.
This ability is presenteven in young infants together with notions of object permanence through occlusions and temporalshape constancy [4, 5, 47, 51].
Achieving this ability by machine has long been a goal of the field [7–9, 14, 25, 32, 42, 54, 55, 60, 62, 63] and modern methods are now able to reliably discover andsegment single moving objects using both appearance (RGB) and motion (optical flow) streams ofthe video.
However, success is limited in the case of multiple moving objects and occlusion.
Our goal in this paper is to develop and study a model that can discover multiple moving objectsand also handle their mutual occlusions.
To this end, we propose a layered object-centric model thatpredicts amodal segmentation masks [67] for each object throughout the video.
Note that, givenamodal segmentations, the depth ordering of the object layers can be determined from occlusions inthe observed (modal) frames when the objects overlap.
See Figure 1.
Our innovation is to build thismechanism into the representation: when objects overlap in a frame, the model has to maintain theamodal shapes through temporal consistency, and can thus infer depth ordering.
To achieve this goal we use only optical flow as our primary information stream.
There are benefitsand drawbacks to this choice.
The benefits are two fold: first, the information that we are after,the segmentation masks, are directly available from discontinuities in the flow field; second, a keyadvantage of using solely optical flow is that the domain adaptation problem is largely avoided [15, 31]– and this means that the model can be trained on synthetic sequences and applied directly to realsequences without a significant Sim2Real disparity.
However, the drawbacks are also two fold: first,
[Figure]
unlike the appearance stream, objects can be invisible in flow when they are not moving, therebyrequiring the model to build a concept of object permanence; and second, but more subtly, flow is notlike appearance in that we cannot expect constancy even when the object is moving (for example, dueto camera shake that affects the flow but not the appearance), and thus the model cannot strongly useflow coherence as a cue but must instead learn to model the segmentation mask shape.
We make four contributions: (i) we introduce an O bject- C entric L ayered R epresentation ( OCLR )model for discovering and segmenting multiple moving objects and inferring their mutual occlusionsfrom optical flow alone.
The architecture of the model is based on a transformer, with DETR likelearnable queries, where each query determines the amodal segmentation mask for an object and itslayer order throughout the video.
(ii) we introduce a layer-based synthetic pipeline to generate multiobject training sequences that cover the variability of the target real sequences in terms of: multipledeforming objects, overlapping objects in some frames, moving cameras, and non-static backgrounds.(iii)
We demonstrate that the model trained on flow from the synthetic videos is able to learn objectpermanence and temporal shape constancy – even if an object is partially occluded in many frames,the model is still able to predict an amodal mask.
(iv) Finally, we evaluate the model trained onlyon synthetic sequences on multiple real video segmentation benchmarks, DAVIS, MoCA, SegTrack,and FBMS-59; and demonstrate strong Sim2Real generalisation, achieving superior segmentationperformance compared to other human-label-free methods.
The addition of a test-time adaptationto include appearance from a self-supervised DINO model boosts the performance further, evenoutperforming some supervised approaches that are finetuned on real data sequences.
The Object-Centric Layered Representation (OCLR) Model
This section describes our object-centric layered representation model that ingests optical flow for theframes of a video, and outputs the segmentation masks and tracks of the moving objects it contains.
The model also determines the number of moving objects and their depth order.
Specifically, themodel predicts amodal ( i.e. complete) segmentation masks and associates each object with a layer.
The layers are composed to form the modal segmentations in the observed optical flow.
The model istrained on synthetic sequences to learn temporal coherence and maintain the object shape even underocclusion, and as a consequence, develops a notion of object permanence.
2.1 Layered representation
For a given video with T frames, we can represent its motion by optical flows, V flow =
{ F 1 , F 2 , . . .
, F T } , we aim to segment the moving objects in videos by adopting a layered model,where each object is segmented and consistently associated to one layer throughout the video:
[Equation]
[Figure]
where ˆ A i ∈ R H × W × K refers to the binary amodal segmentation of the i -th frame, together with apredicted layer depth ˆ r i ∈ R K .
H, W refer to the input height and width, and K denotes the numberof object layers.
Note that, the layer depth value here does not have a true meaning, but only indicatesthe layer ordering.
By definition, at all time steps, the same layer would remain empty or always bindto the same object throughout the video, i.e. objects can be effectively tracked by the consideredlayered representation.
Modal mask generation.
Given the predicted amodal segmentation and layer depth, we describethe composition procedure to generate the modal segmentation for moving objects.
In specific, wepermute the index of layers ( amodal segmentations) based on their depth values, such that, ˆ A k − 1 isalways in front of ˆ A k .
Note that, pixels in amodal masks defined here are binary-valued, i.e. either at0 or 1.
For the i -th frame, its modal segmentation can thus be computed via an iterative, front-to-backblending procedure :
[Equation]
where α ki = clip ( α k − 1 i + ˆ M k − 1 i , 0 , 1) , denoting an accumulated opacity layer with α
i = [-ˆ M 
 =-]{+0 .2.
+}
Architecture
In order to accommodate the layered representation, we introduce an architecture that takes opticalflows as input, and outputs a set of layers and their depths, see Figure 2.
We adopt a U-Net architecturewith a Transformer-based bottleneck, the queries in transformer decoder play the role as layers in ourproposed representation, each is either empty or encoding the amodal shape of one moving objectalong the video.
CNN backbone.
Given a sequence of optical flow inputs, a U-Net encoder is used to computeframe-wise features, V = { v 1 , . .
. , v T } =
{ Φ enc ( F 1 ) , . . . , Φ enc ( F T ) } , where v i ∈
R h × w × c refersto the feature maps, with height h , width w and number of feature channels c .
Following [62], weuse Instance Normalisation (IN) [56] after each convolution, which encourages the separation offoreground motions from background.
Transformer encoder.
We flatten the outputs from the ConvNets in both temporal and spatialdimensions, and inject the positional information by adding a set of learnable spatio-temporalembeddings.
As a consequence, the video features are converted into a sequence of vectors, andpassed into Transformer Encoder to model the temporal relationship across multiple frames.
Transformer decoder.
We pass the vector sequence to transformer decoder, where K learnableembeddings are used as object queries for individual layers.
We broadcast the queries along the temporal dimension, and supply positional information with learnable temporal embeddings, Q obj ∈
R TK × c .
The queries that only differ in temporal embeddings will be associated to the same movingobject along the video.
As outputs from transformer decoder, we extract the last cross-attention maps and recover their spatialdimensions, ρ ∈ R TK × h × w × nheads , which will be progressively upsampled with skip connectionsfrom the ConvNets encoders, resulting in binary amodal segmentation for the object queries in eachframe, { ˆ A 1 , . . .
, ˆ A T } =
Φ dec ( ρ ) .
Layer ordering.
Apart from amodal segmentations, the model will also infer the depth of the layers.
We assume a global layer ordering that remains fixed across frames.
As shown in Figure 2, outputobject queries from transformer decoder are maxpooled along the temporal dimension and passedthrough two feed-forward layers, to get the output ˆ r ∈ R 1 × K , indicating the depth for each of the Klayers.
Intuitively, this is equivalent to reason the layer depth based on one selected key occlusionframe from the whole sequence.
By defining the lowest depth value as corresponding to the toplayer, we then obtain a depth order prediction between layers.
Note that, for layers that are empty orcontain non-interacting objects, the ordering is not strictly defined, thus any prediction should not bepenalised.
2.3 Training objectives
In this section, we describe the training procedure for the proposed architecture, with groundtruthsupervision on both amodal segmentation and layer ordering.
Amodal segmentation loss.
In addition to the conventional pixelwise binary classification ( L bce )on each of the K layers, we also adopt a pixelwise classification loss defined only on a strip regionaround object boundaries, termed as the boundary loss L bound .
This additional loss further emphasisesthe boundary regions, forcing the model to maintain shape information.
During training, since thepredicted amodal segmentations are permutation invariant, we use Hungarian matching to match theircorresponding ground truth [12], and then compute the amodal loss :
[Equation]
where λ bce and λ bound are loss weights.
Layer ordering loss.
For layers with mutual occlusions, we train the order prediction module withranking losses on all K !
pairs of layers, formally,
[Equation]
where σ is the sigmoid function with a temperature factor τ , and the ground-truth ordering indicatesa relative depth relationship ˆ r i > ˆ r j between layer i and j .
Total loss.
The overall loss for training is a combination of amodal L amodal and ordering loss L order :
[Equation]
where λ order is the weight factor for the layer ordering loss.
Discussion.
As discussed in early sections, our model is trained to infer amodal ( i.e. complete)object segmentations and layer depth, even though the object itself may sometimes be invisible inthe flow field due to being temporarily static, or partially occluded by other objects.
As a result, themodel is forced to always maintain the object shape internally, i.e. a notion of object permanence, aswell as infer mutual occlusions from the visible flows.
In the next section, we will introduce a pipelinefor simulating video sequences, where groundtruth amodal segmentations and layer orderings can begenerated at scale to train our model.
Synthetic dataset generation
We introduce a scalable pipeline to synthesize videos to train the proposed layered object-centricmodel.
The pipeline builds on the method of Lamdouar et al.
[31], but extends it to simulating videoswith multiple objects and complex inter-object occlusions through a layer composition.
[Figure]
As shown in Figure 3, we first simulate independent motions for foreground objects and backgroundswithin each layer.
To construct a frame, we overlay the layers via an iterative binary blendingprocedure, and an occlusion occurs when objects in different layers reside in the same spatial position,similar to the idea discussed in Sect.
The proposed pipeline yields rich information at frame level,including RGB appearances, groundtruth optical flows, and modal and amodal object segmentations.
Additionally, we can also compute optical flows for the RGB sequences with an off-the-shell flowestimator, for example, RAFT [53].
In the following sections, we describe some key procedures forvideo simulation, and leave more technical details to the Supplementary Material.
Foreground objects.
We generate opaque 2D sprites with random shapes and RGB textures asforeground objects.
In detail, we adopt two major shapes, namely polygons, and real object sprites.
The polygon shapes can be either convex or non-convex, with different numbers of sides.
A minimumdistance between vertices is also defined to better support vertex-based deformations.
On the otherhand, the real object masks are directly sourced from the silhouettes in the YouTubeVOS dataset [61].
These generated objects are then applied with textures sampled from the PASS dataset [3], which is alarge-scale image dataset with human identifiable information removed.
Non-rigid object transformations.
To simulate non-rigid motions, we apply both homographytransformations and thin plate splines to the objects.
Thin plate splines define a non-linear coordinatetransformation with a set of control points.
For polygon-shaped objects, we consider their verticesas control points, and spatially perturb a subset of the vertices by distances d i ∈ (0 , D i / 2) , where
D i = min ( | P i − P
j | , ∀ j ) , defines the minimum distance between the perturbed point P i and anyother points P j .
For real object silhouettes, a uniform grid is distributed over the object, and the gridpoints are defined as control points, with D i being the distance between adjacent grid points.
Backgrounds and artificial occluders.
To simulate the camera motions, we randomly sampleimages from the PASS dataset, and apply random homography transformations and color jitterings.
Additionally, we replace backgrounds with real videos cropped from copyright-free sources tosimulate the complex background motions and depth variations.
For cases with objects beingoccluded, we introduce artificial occluders superimposed as the top layer.
These occluders aredesigned to have the same motion as the background.
Summary.
The proposed simulation pipeline enables an arbitrary number of videos to be generated,together with multiple types of groundtruth annotation; for example, amodal segmentation and layerordering.
We prepare 4 k synthetic sequences (around 120 k frames) for training, the video sequencescontain 1 , 2 , or 3 objects in equal proportions.
In the following, we train all models on this syntheticvideo dataset unless otherwise specified.
Experiments
4.1 Datasets
To evaluate our multi-layer model, we benchmark on several popular datasets for video object segmentation tasks.
A brief overview of the datasets is given below, with full details in the Supplementary
Material.
For single object video segmentation, we evaluate the model on DAVIS2016 [26], SegTrackv2 [33],
FBMS-59 [43] and MoCA [32].
Note that, despite the fact that multiple objects may be annotated, thecommunity often treats SegTrackv2 and FBMS-59 as a benchmark for single object segmentation [23,63] by grouping objects in the foreground.
To benchmark motion-based segmentation for multiple objects, we introduce a synthetic validationdataset (Syn-val) and a curated dataset (DAVIS2017-motion).
The former is generated with the sameparameters as our synthetic training set (Sect. 3), containing over 300 multi-object sequences (around10 k frames) with 1 , 2 , 3 objects at equal proportions, and controllable occlusions for evaluating modaland amodal segmentations in the ablation studies.
Moreover, as objects in common motion cannotbe distinguished purely from motion cues, we re-annotate the original DAVIS2017 [48] dataset bygrouping jointly moving objects to form a new DAVIS2017-motion dataset.
We will release ourmodified annotations for future research.
4.2 Evaluation metrics
Following the common practice [46], we adopt region similarity ( J ), aka.
intersection-over-union, insingle object segmentation tasks.
While for multiple object segmentation, we additionally considerthe contour similarity ( F ) [41].
As for MoCA dataset, since only bounding box annotations areprovided, we follow the same metric used in [35, 62], and report a mean object detection success rateaveraged over different IoU thresholds { 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
4.3 Implementation details
Training details.
As for pre-processing, we use RAFT to estimate optical flows at ± 1 frame gaps,and resize the obtained flows to 128 × 224 .
During training, we split the video sequences into 30frames per sample, each input frame is first encoded by a U-Net encoder into a feature map with1 / 16 of its original spatial resolution, and passed to the transformer bottleneck.
We use K = 3learnable object queries, associating to 3 independent foreground layers.
The model is trained by the
Adam optimizer [27] with a learning rate linearly warmed up to 5 × 10 − 5 during 40 k iterations, anddecreased by half every 80 k iterations.
RGB-based test-time adaption.
To alleviate the drawbacks from using purely flow information ( e.g. stationary objects over a long period of time), we investigate the benefits of augmentingwith test-time adaptation of appearance features.
For this we use the self-supervised DINO-pretrainedvision transformer [13] (ViT-S/8, patch sizes 8 × 8 ).
The model is adapted by using the per-frameobject masks obtained from the flow-based model predictions as noisy annotations.
These annotationsare used to finetune the last two layers of the vision transformer by supervised contrastive learning.
The finetuned models for each sequence is then applied for mask propagation in the same manneras [22].
More technical details can be found in the Supplementary Material.
4.4 Ablation study
Here, we present a series of ablation studies on training details and pipeline for data simulationhowever, due to the space limitation, we can only summarise some key findings, we refer the readerto Supplementary Materials for all details.
[Table]
As shown in Table 1, we can make the following observations: First , Instance Normalisation in
CNN encoder is indispensable for training the network, as indicated by Ours-A vs. Ours-C; Second , training on only modal mask degrades model performance on all datasets, especially on Syn-Valwith multiple objects and occlusions, suggesting that explicit amodal supervision are critical forlearning object permanence, as indicated by Ours-B vs. Ours-D; Third , a longer temporal input (OursC vs. Ours-D) tends to result in slightly higher overall performance;
Fourth , applying boundaryloss (Ours-C vs. Ours-E) leads to a noticeable performance boost.
This validates our assumption thatfocusing on object boundaries can help the model to learn the information regarding object
shapesand layer orders from optical flows.
4.5 Single object video segmentation
In this section, we compare our model with state-of-the-art methods on various single object segmentation benchmarks.
Note that, we mainly compare with the self-supervised approaches, as both linesshare the same spirit in the sense that training does not require any manual annotation, nor fine-tuningon the target dataset.
As shown in Table 2, our flow-only model demonstrates superior performanceover all other human-label-free approaches.
Figure 4 provides qualitative illustrations of the model,in comparison with other state-of-the-art methods.
It can be seen that our predictions are temporallyconsistent and not affected by noticeable background distractors in flow signals.
Furthermore, ourmodel is capable of handling complex scenarios including heavy object deformations and occlusions.
The other methods are not able to maintain the object shape consistently, and consequently theirperformance is weaker.
[Table]
The RGB-based test-time adaptation gives a further performance boost on most of the benchmarks,and is even competitive to some supervised approaches that have been finetuned on the target videodata.
Note that the test-time adaptation is actually detrimental to performance on MoCA.
This is notunexpected though, as MoCA has many camouflage sequences where the objects are not visuallydistinguishable in appearance from their background environment [Table] and motion provides crucial cuesfor discovering them.
4.6 Multiple object video segmentation
To the best of our knowledge, no existing unsupervised approaches have reported performancesegmenting multiple objects purely based on optical flow.
Apart from re-running the originalself-supervised Motion Grouping [62] method (with three foreground queries) as a baseline on DAVIS2017-motion, we also train a variant of Motion Grouping model directly supervised by oursynthetic dataset, namely Motion Grouping (Syn.).
We also compare with existing semi-supervisedapproaches, where the first-frame ground-truth segmentation is provided, and the model only needs topropagate the masks through the video, thus an easier task than ours which requires to simultaneouslydiscover the objects and track them.
As shown in Table 3, our proposed model outperforms both Motion Grouping baselines by a significantmargin.
This is because our proposed OCLR exploits a layered representation to maintain the objectshape through the video, which enables to segment the objects that are under occlusion or having
O u r s (f l o w )
G T
M A T N e t
R G B
M o t i on G r oup
i ng
F l o w
No prediction
O u r s (t es t-t i m e a d a p . ) unnoticeable motion. In contrast, Motion Grouping only predicts per-frame segmentations, i.e. notforced to preserve temporal relations between objects. This conjecture can also be confirmed by thequalitative results in Figure 5, for example, despite the person and one dog (4th column) are notvisible in the flow, our model still correctly recovers them.
Moreover, after RGB-based test-timeadaptions, the performance can be further boosted both quantitatively and qualitatively, for example,the boundary of the pig mask in 1st column, the person in the 5th column.
[Table]
Motion Grouping (
[Table]
[Table]
Related Work
[Table] Video object segmentation has been a longstanding task in computer vision, two protocols haveattracted increasing interest in the recent literature [6, 9, 11, 14, 18, 19, 23–25, 29, 30, 40, 42, 44, 45,49, 49, 55, 57, 58, 61, 64], namely, semi-supervised video object segmentation ( semi-supervised
VOS ), and unsupervised video object segmentation ( unsupervised VOS ).
The former aims to relocalize one or multiple targets that are specified in the first frame of a video with pixel-wise masks,and the latter considers automatically segmenting the object of interest (usually the most salient one)from the background in a video sequence.
Despite being called unsupervised VOS , in practice, thepopular methods address such problems by supervised training on large-scale external datasets, thisis in contrast to our proposed approach that does not rely on human annotations whatsoever.
Motion segmentation focuses on discovering the moving objects in videos.
In the literature, [9, 25,42, 60] proposed to cluster the pixels with similar motion patterns; [14, 54, 55] train deep networks
O u r s (f l o w )
O u r s (t es tt i m e a d a p .
G T
R G B
F l o w to map the motions to segmentation masks.
In [63], adversarial training was adopted to leverage theindependent motions between the moving object and its background; In [7, 8, 32], the authors proposeto highlight the independently moving object by compensating for the background motion, either byregistering consecutive frames, or explicitly estimating camera motion.
In [62], a Transformer-likearchitecture is used to reconstruct the input flows, and the segmentation masks can be generated asa side product.
In contrast to the existing approaches that primarily focus on single moving objectsegmentation, we focus on segmenting multiple moving objects, even under mutual occlusions.
Layered representation was originally proposed by Wang and Adelson [59], to represent a videoas a composition of layers with simpler motions.
Recently, the layer decomposition ideas havebeen adopted for novel view synthesis [52, 65], separating reflections and other semi-transparenteffects [1, 2, 20, 37, 38], or foreground/background estimation [20].
Unlike these approaches thatprimarily focus on graphics applications, we propose a layered representation to handle the occlusionproblems in multi-object discovery purely from their motions.
Object-centric representation decomposes the scenes into “objects”, normally, visual frame reconstruction has been widely used as training objective, for example, IODINE [21] uses iterativevariational inference to infer a set of latent variables recurrently, with each representing one object inan image.
Similarly, MONet [10] and GENESIS [16] also adopt multiple encoding-decoding steps,[28, 36] propose a slot attention mechanism, which enables the iterative binding procedure.
In thispaper, we also adopt a Transformer variant, but focus on discovering objects in videos by motions.
Discussion
We have achieved our design goal, in that the model is able to handle multiple objects, and theirocclusions, and to predict depth ordering of their layers.
Also, the model demonstrates superior performance, both quantitatively and qualitatively, over prior methods that rely on zero human annotations.
Nevertheless, there are of course some limitations and room for further improvements: First , thecurrent method may fail in very challenging real-world scenarios such as heavy object deformations,complex mutual interactions, etc.
Probably a more sophisticated data simulation pipeline with highlyarticulated objects or even 3D sprites would help to further reduce this Sim2Real domain gap.
Second ,our test-time adaptation has demonstrated a remarkable performance improvement benefiting fromthe combination between flow inferences and RGB correlations.
This overcomes, to an extent, theproblem of the flow-only model ‘forgetting’ the shape of objects when they are stationary for multipleframes.
A further direction could naturally be to incorporate RGB information into our flow-basednetwork in pre-training.
Third , the model currently uses a global depth ordering for the sequence, soit cannot handle situations such as order altering and mutual occlusions.
Potential future studies couldwork on a more sophisticated layered model by focusing on pair-wise relationships between objects.
Despite these limitations, the approach has convincingly demonstrated both the value of inferringamodal segmentation masks, in order to handle occlusions, and the possibility of training such modelsentirely on synthetic sequences.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] In Abstract and Section 1, we have clearly included the intuitions, motivationsand contributions in our work, and briefly introduced a depth-ordered layered model
for multi-object motion segmentation by following a Sim2Real training procedure.
(b) Did you describe the limitations of your work?
[Yes] In Section 6, we reflect on several limitations and propose possible further
research directions.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In the Supplementary Material, we have included a section discussing if our
work leads to possible negative social consequences.
Overall, we reckon that our work
does not lead to noticeable detrimental effects to human rights, security, environment,
human livelihood, economics, etc.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
Yes
In the Supplementary Material, we provided detailed explanation on how our
work conforms ethics review guidelines, including our effort to eliminate potential bias
and personally identifiable information in the dataset.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] There are no theoretical results introduced in our work.
(b) Did you include complete proofs of all theoretical results?
[N/A] There are no theoretical results introduced in our work.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] In the Supplementary Material, we have provided the details in the curated
DAVIS2017-motion dataset, together with some demonstration videos of our synthetic
dataset.
The full datasets and codes for training and evaluation will be released before
the publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] In Section 4, we have described the dataset, evaluation metrics and implemen-
tation details in our work.
We also give more training details and settings in the
Supplementary Material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] In the Supplementary Material, we have reported the main results with error bars from repeated experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? Yes]
In the Supplementary Material we have reported the amount of computation totrain, test and test-time adapt our model, together with details of the adopted GPU andapproximate time taken.4.
If you are using existing assets (e
g
code
data
models) or
curating
/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Section 4, we have described the datasets we used with references to theoriginal source
(b) Did you mention the license of the assets?
[Yes] In the Supplementary Material, we have mentioned that all dataset set used andcurated follows CC-BY license with references to the original source provided.
Realvideo backgrounds for synthetic data generation are from https://pixabay.com/videos/.
All selected video follows a Pixabay License which states that videos are free for anycommercial and non-commercial uses with no attribution required.
c)
Did you include any new assets either in the supplemental material or as a URL?[Yes]
In the Supplementary Material, we have explained in detail how we curate
DAVIS2017 dataset to a new DAVIS2017-motion dataset, and also provided moreinformation about our synthetic dataset.
All generation code and resultant datasets willbe released.
(d) Did you discuss whether and how consent was obtained from
people whose data you’re
using/curating?
[Yes] In the Supplementary Material, we have discussed that the datasets used forcuration and synthetic data generation are DAVIS2017, YouTubeVOS2018 and PASS,which follow CC-BY license.
We have included references for all these sources.
e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?[Yes]
In the Supplementary Material, there is a discussion on how we eliminatepersonally identifiable
information and offensive content in our curated dataset and
synthetic dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
There are no crowdsourcing or conducted research with human subjects in ourwork.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
There are no crowdsourcing or conducted research with human subjects in our
work.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] There is are crowdsourcing or conducted research with human subjects in our
work.