Original: C ONDITIONAL O BJECT -C ENTRIC L EARNING FROM V IDEO
Revised: C ONDITIONAL O BJECT -C ENTRIC L EARNING FROM V IDEO

Original: Anonymous authors
Revised: Google

Original: Paper under double-blind review
Revised: Research

Original: A BSTRACT
Revised: A BSTRACT

Original: Object-centric representations are a promising path toward more systematic generalization by providing ﬂexible abstractions upon which compositional world models can be built.
Revised: Object-centric representations are a promising path toward more systematic generalization by providing ﬂexible abstractions upon which compositional world models can be built.

Original: Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision.
Revised: Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision.

Original: However, such fully-unsupervised methods still fail to scale to  realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene.
Revised: However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene.

Original: In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical ﬂow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in signiﬁcantly more realistic synthetic data.
Revised: In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical ﬂow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in signiﬁcantly more realistic synthetic data.

Original: We introduce a sequential extension to Slot Attention which we train to predict optical ﬂow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the ﬁrst frame, is sufﬁcient to signiﬁcantly improve instance segmentation.
Revised: We introduce a sequential extension to Slot Attention which we train to predict optical ﬂow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the ﬁrst frame, is sufﬁcient to signiﬁcantly improve instance segmentation.

Original: These beneﬁts generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences.
Revised: These beneﬁts generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences.

Original: We also ﬁnd that such initial-state-conditioning can be used during inference as a ﬂexible interface to query the model for speciﬁc objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.
Revised: We also ﬁnd that such initial-state-conditioning can be used during inference as a ﬂexible interface to query the model for speciﬁc objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.

Revised: Project page: https://slot-attention-video.github.io/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans understand the world in terms of separate objects (Kahneman et al., 1992; Spelke & Kinzler, 2007), which serve as compositional building blocks that can be processed independently and recombined.
Revised: Humans understand the world in terms of separate objects (Kahneman et al., 1992; Spelke & Kinzler, 2007), which serve as compositional building blocks that can be processed independently and recombined.

Original: Such a compositional model of the world forms the foundation for high-level cognitive abilities such as language, causal reasoning, mathematics, planning, etc. and is crucial for generalizing in predictable and systematic ways.
Revised: Such a compositional model of the world forms the foundation for high-level cognitive abilities such as language, causal reasoning, mathematics, planning, etc. and is crucial for generalizing in predictable and systematic ways.

Original: Object-centric representations have the potential to greatly improve sample efﬁciency, robustness, generalization to new tasks, and interpretability of machine learning algorithms (Greff et al., 2020).
Revised: Object-centric representations have the potential to greatly improve sample efﬁciency, robustness, generalization to new tasks, and interpretability of machine learning algorithms (Greff et al., 2020).

Original: In this work, we focus on the aspect of modeling motion of objects from video, because of its synergistic relationship with object-centric representations: On the one hand, objects support learning an efﬁcient dynamics model by factorizing the scene into approximately independent parts with only sparse interactions.
Revised: In this work, we focus on the aspect of modeling motion of objects from video, because of its synergistic relationship with object-centric representations: On the one hand, objects support learning an efﬁcient dynamics model by factorizing the scene into approximately independent parts with only sparse interactions.

Original: Conversely, motion provides a powerful cue for which inputs should be grouped together, and is thus an important tool for learning about objects.
Revised: Conversely, motion provides a powerful cue for which inputs should be grouped together, and is thus an important tool for learning about objects.

Original: Unsupervised multi-object representation learning has recently made signiﬁcant progress both on images (e.g. Burgess et al., 2019; Greff et al., 2019; Lin et al., 2020; Locatello et al., 2020) and on video (e.g. Veerapaneni et al., 2020;
Revised: Unsupervised multi-object representation learning has recently made signiﬁcant progress both on images (e.g. Burgess et al., 2019; Greff et al., 2019; Lin et al., 2020; Locatello et al., 2020) and on video (e.g. Veerapaneni et al., 2020;

Original: Weis et al., 2020; Jiang et al., 2020).
Revised: Weis et al., 2020; Jiang et al., 2020).

Original: By incorporating objectcentric inductive biases, these methods learn to segment and represent objects from the statistical structure of the data alone without the need for supervision.
Revised: By incorporating objectcentric inductive biases, these methods learn to segment and represent objects from the statistical structure of the data alone without the need for supervision.

Original: Despite promising results these methods are currently limited by two important problems: Firstly, they are restricted to toy data like moving 2D sprites or very simple 3D scenes and generally fail at more realistic data with complex textures (Greff et al., 2019;
Revised: Despite promising results these methods are currently limited by two important problems: Firstly, they are restricted to toy data like moving 2D sprites or very simple 3D scenes and generally fail at more realistic data with complex textures (Greff [Figure]

Original:  Harley et al., 2021; Karazija et al., 2021).
Revised: et al., 2019; Harley et al., 2021; Karazija et al., 2021).

Original: And secondly, it is not entirely clear how to interface with these models both during training and inference.
Revised: And secondly, it is not entirely clear how to interface with these models both during training and inference.

Original: The notion of an object is ambiguous and task-dependent, and the segmentation learned by these models does not necessarily align with the tasks of interest.
Revised: The notion of an object is ambiguous and task-dependent, and the segmentation learned by these models does not necessarily align with the tasks of interest.

Original: The model might, for example, over-segment a desired object into distinct parts,[Figure] or alternatively fail to segment into the desired parts.
Revised: The model might, for example, over-segment a desired object into distinct parts, or alternatively fail to segment into the desired parts.

Original: Ideally, we would like to be able to provide the model with hints as to the desired level of granularity during training, and ﬂexibly query the model during inference to request that the model detects and tracks a particular object or a part thereof.
Revised: Ideally, we would like to be able to provide the model with hints as to the desired level of granularity during training, and ﬂexibly query the model during inference to request that the model detects and tracks a particular object or a part thereof.

Original: In this paper we introduce a sequential extension of Slot Attention (Locatello et al., 2020) that we call Slot Attention for Video (SAVi) to tackle the problem of unsupervised / weakly-supervised multiobject segmentation and tracking in video data.
Revised: In this paper we introduce a sequential extension of Slot Attention (Locatello et al., 2020) that we call Slot Attention for Video (SAVi) to tackle the problem of unsupervised / weakly-supervised multiobject segmentation and tracking in video data.

Original: We demonstrate that 1) using optical ﬂow prediction as a self-supervised objective and 2) providing a small set of abstract hints such as the center of mass position for objects as conditional inputs in the ﬁrst frame sufﬁces to direct the decomposition process in complex video scenes without otherwise requiring any priors on the size of objects or on the information content of their representations.
Revised: We demonstrate that 1) using optical ﬂow prediction as a self-supervised objective and 2) providing a small set of abstract hints such as the center of mass position for objects as conditional inputs in the ﬁrst frame sufﬁces to direct the decomposition process in complex video scenes without otherwise requiring any priors on the size of objects or on the information content of their representations.

Original: We show successful segmentation and tracking for synthetic video data with signiﬁcantly higher realism and visual complexity than the datasets used in prior work on unsupervised object representation learning.
Revised: We show successful segmentation and tracking for synthetic video data with signiﬁcantly higher realism and visual complexity than the datasets used in prior work on unsupervised object representation learning.

Original: These results are robust with respect to noise on both the optical ﬂow signal and the conditioning, and that they generalize almost perfectly to longer sequences, novel objects, and novel backgrounds.
Revised: These results are robust with respect to noise on both the optical ﬂow signal and the conditioning, and that they generalize almost perfectly to longer sequences, novel objects, and novel backgrounds.

Original:  S LOT A TTENTION FOR V IDEO (SAV I )
Revised:  S LOT A TTENTION FOR V IDEO (SAV I )

Original: We introduce a sequential extension of the Slot Attention (Locatello et al., 2020) architecture to video data, which we call SAVi.
Revised: We introduce a sequential extension of the Slot Attention (Locatello et al., 2020) architecture to video data, which we call SAVi.

Original: Inspired by predictor-corrector methods for integration of ordinary differential equations, SAVi performs two steps for each observed video frame: a prediction and a correction step.
Revised: Inspired by predictor-corrector methods for integration of ordinary differential equations, SAVi performs two steps for each observed video frame: a prediction and a correction step.

Original: The correction step uses Slot Attention to update (or correct ) the set of slot representations based on slot-normalized cross-attention with the inputs.
Revised: The correction step uses Slot Attention to update (or correct ) the set of slot representations based on slot-normalized cross-attention with the inputs.

Original: The prediction step uses self-attention among the slots to allow for modeling of temporal dynamics and object interactions.
Revised: The prediction step uses self-attention among the slots to allow for modeling of temporal dynamics and object interactions.

Original: The output of the predictor is then used to initialize the corrector at the next time step, thus allowing the model to consistently track objects over time.
Revised: The output of the predictor is then used to initialize the corrector at the next time step, thus allowing the model to consistently track objects over time.

Original: Importantly, both of these steps are permutation equivariant and thus are able to preserve slot symmetry.
Revised: Importantly, both of these steps are permutation equivariant and thus are able to preserve slot symmetry.

Original: See Figure 1 for a schematic overview of the SAVi architecture.
Revised: See Figure 1 for a schematic overview of the SAVi architecture.

Original: Encoder For each time-step t ∈ { 1 . . .
Revised: Encoder For each time-step t ∈ { 1 . . .

Original: T } the corresponding video frame x t is ﬁrst passed through a small convolutional neural network (CNN) encoder (here, a stack of ﬁve convolutional layers with
Revised: T } the corresponding video frame x t is ﬁrst passed through a small convolutional neural network (CNN) encoder (here, a stack of ﬁve convolutional layers with

Original: ReLUs), where we concatenate a linear positional encoding at the second-to-last layer.
Revised: ReLUs), where we concatenate a linear positional encoding at the second-to-last layer.

Original: The resulting grid of visual features is ﬂattened into a set of vectors h t = f enc ( x t ) ∈
Revised: The resulting grid of visual features is ﬂattened into a set of vectors h t = f enc ( x t ) ∈

Original: R N × D enc , where N is the size of the ﬂattened grid (i.e., width*height) and D enc is the dimensionality of the CNN feature maps.
Revised: R N × D enc , where N is the size of the ﬂattened grid (i.e., width*height) and D enc is the dimensionality of the CNN feature maps.

Original: Afterwards, each vector is independently passed through a multi-layer perceptron (MLP).
Revised: Afterwards, each vector is independently passed through a multi-layer perceptron (MLP).

Original: Slot Initialization SAVi maintains K slots, each of which can represent a part of the input such as objects or parts thereof.
Revised: Slot Initialization SAVi maintains K slots, each of which can represent a part of the input such as objects or parts thereof.

Original: We denote the set of slot representations at time t as S t =
Revised: We denote the set of slot representations at time t as S t =

Original: [ s 1 t , . .
Revised: [ s 1 t , . .

Original: , s Kt ] ∈ R K × D , where we use the calligraphic font to indicate that any operation on these sets is equivariant (or invariant) w.r.t.
Revised: , s Kt ] ∈ R K × D , where we use the calligraphic font to indicate that any operation on these sets is equivariant (or invariant) w.r.t.

Original: permutation of their elements.
Revised: permutation of their elements.

Original: In other words, the ordering of the slots carries no information and they can be freely permuted (in a consistent manner across all time steps) without changing the model output.
Revised: In other words, the ordering of the slots carries no information and they can be freely permuted (in a consistent manner across all time steps) without changing the model output.

Original: We consider two types of initializers: conditional and unconditional.
Revised: We consider two types of initializers: conditional and unconditional.

Original: In the conditional case, we encode the conditional input either via a simple MLP (in the case of bounding boxes or center of mass coordinates) or via a CNN (in the case of segmentation masks).
Revised: In the conditional case, we encode the conditional input either via a simple MLP (in the case of bounding boxes or center of mass coordinates) or via a CNN (in the case of segmentation masks).

Original: For slots for which there is no conditioning information available (e.g. if K is larger than the number of objects), we set the conditional input to a ﬁxed value (e.g., ‘ − 1 ’ for bounding box coordinates).
Revised: For slots for which there is no conditioning information available (e.g. if K is larger than the number of objects), we set the conditional input to a ﬁxed value (e.g., ‘ − 1 ’ for bounding box coordinates).

Original: The encoded conditional input forms the initial slot representation for SAVi.
Revised: The encoded conditional input forms the initial slot representation for SAVi.

Original: In the unconditional case, we either randomly initialize slots by sampling from a Gaussian distribution independently for each video (both at training and at test time), or by learning a set of initial slot vectors.
Revised: In the unconditional case, we either randomly initialize slots by sampling from a Gaussian distribution independently for each video (both at training and at test time), or by learning a set of initial slot vectors.

Original: Corrector The task of the corrector is to update the slot representations based on the visual features from the encoder.
Revised: Corrector The task of the corrector is to update the slot representations based on the visual features from the encoder.

Original: In SAVi this is done using the iterative attention mechanism introduced in Slot
Revised: In SAVi this is done using the iterative attention mechanism introduced in Slot

Original: Attention (Locatello et al., 2020).
Revised: Attention (Locatello et al., 2020).

Original: Different from a regular cross-attention mechanism (e.g. Vaswani et al., 2017) which is normalized over the inputs, Slot Attention encourages decomposition of the input into multiple slots via softmax-normalization over the output (i.e. the slots), which makes it an appealing choice for our video decomposition architecture.
Revised: Different from a regular cross-attention mechanism (e.g. Vaswani et al., 2017) which is normalized over the inputs, Slot Attention encourages decomposition of the input into multiple slots via softmax-normalization over the output (i.e. the slots), which makes it an appealing choice for our video decomposition architecture.

Original: When using a single iteration of Slot
Revised: When using a single iteration of Slot

Original: Attention, the corrector update takes the following form:
Revised: Attention, the corrector update takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where Z t = (cid:80) Nn
Revised: where Z t = (cid:80) Nn

Original: =
Revised: =

Original: A t,n and (cid:12) denotes the Hadamard product.
Revised: A t,n and (cid:12) denotes the Hadamard product.

Original: k , q , v are learned linear projections that map to a common dimension D .
Revised: k , q , v are learned linear projections that map to a common dimension D .

Original: We apply LayerNorm (Ba et al., 2016) before each projection.
Revised: We apply LayerNorm (Ba et al., 2016) before each projection.

Original: The slot representations are then individually updated using Gated Recurrent Units (Cho et al., 2014) as ˆ s kt =
Revised: The slot representations are then individually updated using Gated Recurrent Units (Cho et al., 2014) as ˆ s kt =

Original: GRU( u kt , s kt ) .
Revised: GRU( u kt , s kt ) .

Original: Alternatively, the attention step (followed by the GRU update) can be iterated multiple times with shared parameters per frame of the input video.
Revised: Alternatively, the attention step (followed by the GRU update) can be iterated multiple times with shared parameters per frame of the input video.

Original: For added expressiveness, we apply an MLP with residual connection ˆ s kt ← ˆ s kt + MLP(LN( ˆ s kt )) and LayerNorm ( LN ) after the
Revised: For added expressiveness, we apply an MLP with residual connection ˆ s kt ← ˆ s kt + MLP(LN( ˆ s kt )) and LayerNorm ( LN ) after the

Original: GRU when using multiple Slot Attention iterations, following Locatello et al.
Revised: GRU when using multiple Slot Attention iterations, following Locatello et al.

Original: Predictor The predictor takes the role of a transition function to model temporal dynamics, including interactions between slots.
Revised: Predictor The predictor takes the role of a transition function to model temporal dynamics, including interactions between slots.

Original: To preserve permutation equivariance, we use a Transformer encoder (Vaswani et al., 2017).
Revised: To preserve permutation equivariance, we use a Transformer encoder (Vaswani et al., 2017).

Original: It allows for modeling of independent object dynamics as well as information exchange between slots via self-attention, while being more memory efﬁcient than GNN-based models such as the Interaction Network (Battaglia et al., 2016).
Revised: It allows for modeling of independent object dynamics as well as information exchange between slots via self-attention, while being more memory efﬁcient than GNN-based models such as the Interaction Network (Battaglia et al., 2016).

Original: Slots are updated as follows:
Revised: Slots are updated as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: For MultiHeadSelfAttn we use the default multi-head dot-product attention mechanism from Vaswani et al.
Revised: For MultiHeadSelfAttn we use the default multi-head dot-product attention mechanism from Vaswani et al.

Original: We apply LayerNorm ( LN ) after each residual connection.
Revised: We apply LayerNorm ( LN ) after each residual connection.

Original: Decoder The network output should be permutation equivariant (for per-slot outputs) or invariant (for global outputs) with respect to the slots.
Revised: Decoder The network output should be permutation equivariant (for per-slot outputs) or invariant (for global outputs) with respect to the slots.

Original: Slots can be read out either after application of the corrector or after the predictor (transition model).
Revised: Slots can be read out either after application of the corrector or after the predictor (transition model).

Original: We decode slot representations after application of the corrector using a slot-wise Spatial Broadcast Decoder (Watters et al., 2019) to produce per-slot
Revised: We decode slot representations after application of the corrector using a slot-wise Spatial Broadcast Decoder (Watters et al., 2019) to produce per-slot

Original: RGB predictions of the optical ﬂow (or reconstructed frame) and an alpha mask.
Revised: RGB predictions of the optical ﬂow (or reconstructed frame) and an alpha mask.

Original: The alpha mask is normalized across slots via a softmax and used to perform a weighted sum over the slot-wise RGB reconstruction to arrive at a combined reconstructed frame:
Revised: The alpha mask is normalized across slots via a softmax and used to perform a weighted sum over the slot-wise RGB reconstruction to arrive at a combined reconstructed frame:

Original: [Equation]
Revised: [Equation]

Original: Training Our sole prediction target is optical ﬂow for each individual video frame, which we represent as RGB images using the default conversion in the literature (Sun et al., 2018).
Revised: Training Our sole prediction target is optical ﬂow for each individual video frame, which we represent as RGB images using the default conversion in the literature (Sun et al., 2018).

Original: Alternatively, our framework also supports prediction of other image-shaped targets, such as reconstruction of the original input frame.
Revised: Alternatively, our framework also supports prediction of other image-shaped targets, such as reconstruction of the original input frame.

Original: We minimize the pixel-wise squared reconstruction error (averaged over the batch), summed over both the temporal and spatial dimensions:
Revised: We minimize the pixel-wise squared reconstruction error (averaged over the batch), summed over both the temporal and spatial dimensions:

Original: [Equation]
Revised: [Equation]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Object-centric representation learning There is a rich literature on learning object representations from static scenes (Greff et al., 2016; Eslami et al., 2016; Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2020; Crawford & Pineau, 2019; Lin et al., 2020; Locatello et al., 2020) or videos (van Steenkiste et al., 2018; Kosiorek et al., 2018; Stelzner et al., 2019;Kipf et al., 2020;
Revised: Object-centric representation learning There is a rich literature on learning object representations from static scenes (Greff et al., 2016; Eslami et al., 2016; Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2020; Crawford & Pineau, 2019; Lin et al., 2020; Locatello et al., 2020; Du et al., 2021a) or videos (van Steenkiste et al., 2018; Kosiorek et al., 2018; Stelzner et al., 2019;

Original:  Crawford & Pineau, 2020; Creswell et al., 2021) without explicit supervision.
Revised: Kipf et al., 2020; Crawford & Pineau, 2020; Creswell et al., 2021) without explicit supervision.

Original: PSGNet (Bear et al., 2020) learns to decompose static images or individual frames from a video into hierarchical scene graphs using motion information estimated from neighboring video frames.
Revised: PSGNet (Bear et al., 2020) learns to decompose static images or individual frames from a video into hierarchical scene graphs using motion information estimated from neighboring video frames.

Original: Most closely related to our work are sequential object-centric models for videos and dynamic environments, such as OP3 (Veerapaneni et al., 2020), R-SQAIR (Stani´c & Schmidhuber, 2019), ViMON (Weis et al., 2020), and SCALOR (Jiang et al., 2020), which learn an internal motion model for each object.
Revised: Most closely related to our work are sequential object-centric models for videos and dynamic environments, such as OP3 (Veerapaneni et al., 2020), R-SQAIR (Stani´c & Schmidhuber, 2019) , ViMON (Weis et al., 2020), and SCALOR (Jiang et al., 2020), which learn an internal motion model for each object.

Original: SIMONe (Kabra et al., 2021) auto-encodes an entire video in parallel and learns temporally-abstracted representations of objects.
Revised: SIMONe (Kabra et al., 2021) auto-encodes an entire video in parallel and learns temporally-abstracted representations of objects.

Original: OP3 (Veerapaneni et al., 2020) uses the same decoder as SAVi and a related dynamics model, but a less efﬁcient inference process compared to Slot Attention.
Revised: OP3 (Veerapaneni et al., 2020) uses the same decoder as SAVi and a related dynamics model, but a less efﬁcient inference process compared to Slot Attention.

Original: In an attempt to bridge the gap to visually richer and more realistic environments, recent works in object-centric representation learning have explored integration of inductive biases related to 3D scene geometry, both for static scenes (Chen et al., 2020; Stelzner et al., 2021) and for videos (Du et al., 2021; Henderson & Lampert, 2020;
Revised: In an attempt to bridge the gap to visually richer and more realistic environments, recent works in object-centric representation learning have explored integration of inductive biases related to 3D scene geometry, both for static scenes (Chen et al., 2020; Stelzner et al., 2021) and for videos (Du et al., 2021b; Henderson & Lampert, 2020;

Original: Harley et al., 2021).
Revised: Harley et al., 2021).

Original: This is largely orthogonal to our approach of utilizing conditioning and optical ﬂow.
Revised: This is largely orthogonal to our approach of utilizing conditioning and optical ﬂow.

Original: A recent related method, FlowCaps (Sabour et al., 2020), similarly proposed to use optical ﬂow in a multi-object model.
Revised: A recent related method, FlowCaps (Sabour et al., 2020), similarly proposed to use optical ﬂow in a multi-object model.

Original: FlowCaps uses capsules (Sabour et al., 2017; Hinton et al., 2018) instead of a slot-based representation and assumes specialization of individual capsules to objects or parts of a certain appearance, making it unsuitable for environments that contain a large variety of object types.
Revised: FlowCaps uses capsules (Sabour et al., 2017; Hinton et al., 2018) instead of a slot-based representation and assumes specialization of individual capsules to objects or parts of a certain appearance, making it unsuitable for environments that contain a large variety of object types.

Original: Using a slot-based, exchangeable representation of objects allows SAVi to represent a diverse range of objects and generalize to novel objects at test time.
Revised: Using a slot-based, exchangeable representation of objects allows SAVi to represent a diverse range of objects and generalize to novel objects at test time.

Original: We discuss further recent related works on attention-basedmodular architectures  (Santoro et al., 2018; Goyal et al., 2021a;b;c), object-centric models for dynamic visual reasoning (Yi et al., 2020; Ding et al., 2021a; Bar et al., 2021), and supervised attention-based object-centric models (Fuchs et al., 2019; Carion et al., 2020; Kamath et al., 2021; Meinhardt et al., 2021) in Appendix A.1.
Revised: We discuss further recent related works on attention-based architectures operating on sets of latent variables (Santoro et al., 2018; Goyal et al., 2021a;b;c; Jaegle et al., 2021b;a), object-centric models for dynamic visual reasoning (Yi et al., 2020; Ding et al., 2021a; Bar et al., 2021), and supervised attention-based object-centric models (Fuchs et al., 2019; Carion et al., 2020; Kamath et al., 2021; Meinhardt et al., 2021) in Appendix A.1.

Original: Video object segmentation and tracking The conditional tasks we consider in our work are closely related to the computer vision task of semi-supervised video object segmentation (VOS), where segmentation masks are provided for the ﬁrst video frame during evaluation.
Revised: Video object segmentation and tracking The conditional tasks we consider in our work are closely related to the computer vision task of semi-supervised video object segmentation (VOS), where segmentation masks are provided for the ﬁrst video frame during evaluation.

Original: Different from the typical setting, which is addressed by supervised learning on fully annotated videos or related datasets (e.g. Caelles et al., 2017; Luiten et al., 2018), we consider the problem where models do not have access to any supervised information beyond the conditioning information on the ﬁrst frame (e.g. a bounding box for each object).
Revised: Different from the typical setting, which is addressed by supervised learning on fully annotated videos or related datasets (e.g. Caelles et al., 2017; Luiten et al., 2018), we consider the problem where models do not have access to any supervised information beyond the conditioning information on the ﬁrst frame (e.g. a bounding box for each object).

Original: Several recent works have explored pre-training using self-supervision (Li et al., 2019; Jabri et al., 2020; Caron et al., 2021) or image classiﬁcation (Zhang et al., 2020) for the semi-supervised VOS task.
Revised: Several recent works have explored pre-training using self-supervision (Li et al., 2019; Jabri et al., 2020; Caron et al., 2021) or image classiﬁcation (Zhang et al., 2020) for the semi-supervised VOS task.

Original: These models rely on having access to segmentation masks in the ﬁrst frame at evaluation time.
Revised: These models rely on having access to segmentation masks in the ﬁrst frame at evaluation time.

Original: We demonstrate that multi-object segmentation and tracking can emerge even when segmentation labels are absent at both training and test time.
Revised: We demonstrate that multi-object segmentation and tracking can emerge even when segmentation labels are absent at both training and test time.

Original: There is a rich literature on using motion cues to segment objects in the computer vision community.
Revised: There is a rich literature on using motion cues to segment objects in the computer vision community.

Original: These methods use motion information at test time to, for example, cluster trajectories in order to segment independently moving objects (Faktor & Irani, 2014) or estimate multiple fundamental matrices between two views (Isack & Boykov, 2012), to name a few.
Revised: These methods use motion information at test time to, for example, cluster trajectories in order to segment independently moving objects (Faktor & Irani, 2014) or estimate multiple fundamental matrices between two views (Isack & Boykov, 2012), to name a few.

Original: Closest to our work is a contemporary method (Yang et al., 2021) that trains a Slot Attention model on isolated optical ﬂow data for foreground-background segmentation of a single object, independently for individual video frames and without using visual observations.
Revised: Closest to our work is a contemporary method (Yang et al., 2021) that trains a Slot Attention model on isolated optical ﬂow data for foreground-background segmentation of a single object, independently for individual video frames and without using visual observations.

Original: Our method, on the other hand, supports multi-object environments and only relies on motion information as a training signal but otherwise operates directly on textured visual information, which allows it to segment static scenes at test time where optical ﬂow is unavailable and consistently represent and track multiple objects throughout a video.
Revised: Our method, on the other hand, supports multi-object environments and only relies on motion information as a training signal but otherwise operates [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We investigate 1) how SAVi compares to existing unsupervised video decomposition methods, 2) how various forms of hints (e.g., bounding boxes) can facilitate scene decomposition, and 3) how SAVi generalizes to unseen objects, backgrounds, and longer videos at test time.[Figure]
Revised: We investigate 1) how SAVi compares to existing unsupervised video decomposition methods, 2) how various forms of hints (e.g., bounding boxes) can facilitate scene decomposition, and 3) how SAVi generalizes to unseen objects, backgrounds, and longer videos at test time.

Original: Metrics We report two metrics to measure the quality of video decomposition, object segmentation, and tracking: Adjusted Rand Index (ARI) and mean Intersection over Union (mIoU).
Revised: Metrics We report two metrics to measure the quality of video decomposition, object segmentation, and tracking: Adjusted Rand Index (ARI) and mean Intersection over Union (mIoU).

Original: ARI is a clustering similarity metric which we use to measure how well predicted segmentation masks match ground-truth masks in a permutation-invariant fashion, which makes it suitable for unsupervised methods.
Revised: ARI is a clustering similarity metric which we use to measure how well predicted segmentation masks match ground-truth masks in a permutation-invariant fashion, which makes it suitable for unsupervised methods.

Original: Like in prior work (Greff et al., 2019; Locatello et al., 2020; Kabra et al., 2021), we only measure ARI based on foreground objects, which we refer to as FG-ARI.
Revised: Like in prior work (Greff et al., 2019; Locatello et al., 2020; Kabra et al., 2021), we only measure ARI based on foreground objects, which we refer to as FG-ARI.

Original: For video data, one cluster in the computation of ARI corresponds to the segmentation of a single object for the entire video, requiring temporal consistency without object identity switches to perform well on this metric.
Revised: For video data, one cluster in the computation of ARI corresponds to the segmentation of a single object for the entire video, requiring temporal consistency without object identity switches to perform well on this metric.

Original: Training setup During training, we split each video into consecutive sub-sequences of 6 frames each, where we provide the conditioning signal for the ﬁrst frame.
Revised: Training setup During training, we split each video into consecutive sub-sequences of 6 frames each, where we provide the conditioning signal for the ﬁrst frame.

Original: We train for 100k steps (200k for fully unsupervised video decomposition) with a batch size of 64 using Adam (Kingma & Ba, 2015) with a base learning rate of 2 · 10 − 4 .
Revised: We train for 100k steps (200k for fully unsupervised video decomposition) with a batch size of 64 using Adam (Kingma & Ba, 2015) with a base learning rate of 2 · 10 − 4 .

Original: We use a total of 11 slots in SAVi.
Revised: We use a total of 11 slots in SAVi.

Original: For our experiments on fully unsupervised video decomposition we use 2 iterations of Slot Attention per frame and a single iteration otherwise.
Revised: For our experiments on fully unsupervised video decomposition we use 2 iterations of Slot Attention per frame and a single iteration otherwise.

Original: Further architecture details and hyperparameters are provided in the appendix.
Revised: Further architecture details and hyperparameters are provided in the appendix.

Original: On 8x V100 GPUs with 16GB memory each, training SAVi with bounding box conditioning takes approx.
Revised: On 8x V100 GPUs with 16GB memory each, training SAVi with bounding box conditioning takes approx.

Original: 12hrs for videos with 64 × 64 resolution and 30hrs for videos with 128 × 128 resolution.
Revised: 12hrs for videos with 64 × 64 resolution and 30hrs for videos with 128 × 128 resolution.

Original: 4. U NSUPERVISED VIDEO DECOMPOSITION
Revised: 4. U NSUPERVISED VIDEO DECOMPOSITION

Original: We ﬁrst evaluate SAVi in the unconditional setting and with a standard RGB reconstruction objective on the CATER (Girdhar & Ramanan, 2019) dataset 1 .
Revised: We ﬁrst evaluate SAVi in the unconditional setting and with a standard RGB reconstruction objective on the CATER (Girdhar & Ramanan, 2019) dataset 1 .

Original: Dataset examples are shown in Figure 2.
Revised: Dataset examples are shown in Figure 2.

Original: Results of SAVi and four unsupervised object representation learning baselines (taken from Kabra et al.
Revised: Results of SAVi and four unsupervised object representation learning baselines (taken from Kabra et al.

Original: (2021)) are summarized in Table 1a.
Revised: (2021)) are summarized in Table 1a.

Original: The two image-based methods (Slot Attention (Locatello et al., 2020) and MONet (Burgess et al., 2019)) are independently applied to each frame, and thus lack a built-in notion of temporal consistency which is reﬂected in their poor FG-ARI scores.
Revised: The two image-based methods (Slot Attention (Locatello et al., 2020) and MONet (Burgess et al., 2019)) are independently applied to each frame, and thus lack a built-in notion of temporal consistency which is reﬂected in their poor FG-ARI scores.

Original: Unsurprisingly, the two video-based baselines, S-IODINE (Greff et al., 2019) and SIMONe (Kabra et al., 2021), perform better.
Revised: Unsurprisingly, the two video-based baselines, S-IODINE (Greff et al., 2019) and SIMONe (Kabra et al., 2021), perform better.

Original: The (unconditional) SAVi model outperforms these baselines, demonstrating the adequacy of our architecture for the task of unsupervised object representation learning, albeit only for the case of simple synthetic data.
Revised: The (unconditional) SAVi model outperforms these baselines, demonstrating the adequacy of our architecture for the task of unsupervised object representation learning, albeit only for the case of simple synthetic data.

Original: For qualitative results, see Figure 5c and Appendix A.3.
Revised: For qualitative results, see Figure 5c and Appendix A.3.

Original: 4. M ORE REALISTIC DATASETS
Revised: 4. M ORE REALISTIC DATASETS

Original: To test SAVi on more realistic data, we use two video datasets (see Figure 2) introduced in Kubric (Greff et al., 2021), created by simulating rigid body dynamics of up to ten 3D objects rendered in various scenes via raytracing using Blender (Blender Online Community, 2021).
Revised: To test SAVi on more realistic data, we use two video datasets (see Figure 2) introduced in Kubric (Greff et al., 2021), created by simulating rigid body dynamics of up to ten 3D objects rendered in various scenes via raytracing using Blender (Blender Online Community, 2021).

Original: We refer to these as
Revised: We refer to these as

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  Multi-Object Video (MOVi) datasets.
Revised: [Figure] [Table] [Figure] Multi-Object Video (MOVi) datasets.

Original: The ﬁrst dataset, MOVi, uses the same simple 3D shapes as the
Revised: The ﬁrst dataset, MOVi, uses the same simple 3D shapes as the

Original: CATER dataset (both inspired by the CLEVR benchmark (Johnson et al., 2017)), but introduces more complex rigid-body physical dynamics with frequent collisions and occlusions.
Revised: CATER dataset (both inspired by the CLEVR benchmark (Johnson et al., 2017)), but introduces more complex rigid-body physical dynamics with frequent collisions and occlusions.

Original: The second dataset,
Revised: The second dataset,

Original: MOVi++ signiﬁcantly increases the visual complexity compared to datasets used in prior work and uses approx.
Revised: MOVi++ signiﬁcantly increases the visual complexity compared to datasets used in prior work and uses approx.

Original: 380 high resolution HDR photos as backgrounds and a set of 1028 3D scanned everyday objects (Google Research, 2020), such as shoes, toys, or household equipment.
Revised: 380 high resolution HDR photos as backgrounds and a set of 1028 3D scanned everyday objects (Google Research, 2020), such as shoes, toys, or household equipment.

Original: Each dataset contains 9000 training videos and 1000 validation videos with 24 frames at 12 fps each and, unless otherwise mentioned, a resolution of 64 × 64 pixels for MOVi and 128 × 128 pixels for MOVi [Table]
Revised: Each dataset contains 9000 training videos and 1000 validation videos with 24 frames at 12 fps each and, unless otherwise mentioned, a resolution of 64 × 64 pixels for MOVi and 128 × 128 pixels for MOVi++.

Original: [Figure] Due to its high visual complexity the MOVi++ dataset is a signiﬁcantly more challenging testcase for unsupervised and weakly-supervised instance segmentation.
Revised: Due to its high visual complexity the MOVi++ dataset is a signiﬁcantly more challenging testcase for unsupervised and weakly-supervised instance segmentation.

Original: In fact, while SCALOR (Jiang et al., 2020) (using the reference implementation) was able to achieve 81 .
Revised: In fact, while SCALOR (Jiang et al., 2020) (using the reference implementation) was able to achieve 81 .

Original: 2 ± 0 .
Revised: 2 ± 0 .

Original: 4% FG-ARI on
Revised: 4% FG-ARI on

Original: MOVi, it only resulted in 22 .
Revised: MOVi, it only resulted in 22 .

Original: 7 ± 0 .
Revised: 7 ± 0 .

Original: 9% FG-ARI on MOVi++.
Revised: 9% FG-ARI on MOVi++.

Original: The SIMONe model also only achieved around 33% FG-ARI (with one outlier as high as 46% ), using our own re-implementation (which approximately reproduces the above results on CATER).
Revised: The SIMONe model also only achieved around 33% FG-ARI (with one outlier as high as 46% ), using our own re-implementation (which approximately reproduces the above results on CATER).

Original: We observed that both models, as well as
Revised: We observed that both models, as well as

Original: SAVi, converged to a degenerate solution where slots primarily represent ﬁxed regions of the image (often a rectangular patch or a stripe), instead of tracking individual objects.
Revised: SAVi, converged to a degenerate solution where slots primarily represent ﬁxed regions of the image (often a rectangular patch or a stripe), instead of tracking individual objects.

Original: 4. C ONDITIONAL VIDEO DECOMPOSITION
Revised: 4. C ONDITIONAL VIDEO DECOMPOSITION

Original: To tackle these more realistic videos we 1) change the training objective from predicting RGB image to optical ﬂow and 2) condition the latent slots of the SAVi model on hints about objects in the ﬁrst frame of the video, such as their segmentation mask.
Revised: To tackle these more realistic videos we 1) change the training objective from predicting RGB image to optical ﬂow and 2) condition the latent slots of the SAVi model on hints about objects in the ﬁrst frame of the video, such as their segmentation mask.

Original: We showcase typical qualitative results for
Revised: We showcase typical qualitative results for

Original: SAVi in Figure 4, and on especially challenging cases with heavy occlusion or complex textures in
Revised: SAVi in Figure 4, and on especially challenging cases with heavy occlusion or complex textures in

Original: Figures 5a and 5b.
Revised: Figures 5a and 5b.

Original: In all cases, SAVi was trained on six consecutive frames, but we show results for full (24 frame) sequences at test time.
Revised: In all cases, SAVi was trained on six consecutive frames, but we show results for full (24 frame) sequences at test time.

Original: Using detailed segmentation information for the ﬁrst frame is common practice in video object segmentation, where models such as T-VOS (Zhang et al., 2020) or
Revised: Using detailed segmentation information for the ﬁrst frame is common practice in video object segmentation, where models such as T-VOS (Zhang et al., 2020) or

Original: CRW (Jabri et al., 2020) propagate the initial masks through the remainder of the video sequence.
Revised: CRW (Jabri et al., 2020) propagate the initial masks through the remainder of the video sequence.

Original: Quantitative Evaluation On the MOVi and MOVi++ datasets T-VOS achieves 50 .

Original: 4% and 46 .

Original: 4% mIoU respectively, whereas CRW achieves 42 .

Original: 4% and 50 .

Original: 9% mIoU (see Table 1b).

Original: SAVi learns to produce temporally consistent masks that are signiﬁcantly better on MOVi ( 72 . 0% mIoU) and slightly worse than T-VOS and CRW on MOVi++ ( 43 . 0% mIoU) when trained to predict ﬂow and with segmentation masks as conditioning signal in the ﬁrst frame.

Original: However, we hold that this comparison is not entirely fair, since both CRW and T-VOS use a more expressive ResNet (He et al., 2016) backbone and operate on higher-resolution frames (480p).

Original: For T-VOS, this backbone is pre-trained on ImageNet (Deng et al., 2009), whereas for CRW we re-train it on MOVi/MOVi++.

Original: For a fairer comparison, we adapt T-VOS (Zhang et al., 2020) to our setting by using the same CNN encoder architecture at the same resolution as in SAVi and trained using the same supervision signal

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d . t=

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t

Original: G .

Original: T .

Original: P r e d .

Original: [Figure]
Revised: [Figure]

Original: Successfully handled challenging examples.

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: (b) Failure case.

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t

Original: G .
Revised: [Figure]

Original: T .
Revised: [Figure]

Original: P r e d .
Revised: V i

Original: (c) Unsupervised.
Revised: d e o t= G . T . P r e d .

Revised: [Figure]

Revised: Failure

Original: (i.e. optical ﬂow of moving objects).
Revised: [Figure] V i d e o t= G .

Revised: T .

Revised: P r e d .

Revised: (c) Unsupervised.

Revised: Quantitative Evaluation On the MOVi and MOVi++ datasets T-VOS achieves 50 .

Revised: 4% and 46 .

Revised: 4% mIoU respectively, whereas CRW achieves 42 .

Revised: 4% and 50 .

Revised: 9% mIoU (see Table 1b).

Revised: SAVi learns to produce temporally consistent masks that are signiﬁcantly better on MOVi ( 72 . 0% mIoU) and slightly worse than T-VOS and CRW on MOVi++ ( 43 . 0% mIoU) when trained to predict ﬂow and with segmentation masks as conditioning signal in the ﬁrst frame.

Revised: However, we hold that this comparison is not entirely fair, since both CRW and T-VOS use a more expressive ResNet (He et al., 2016) backbone and operate on higher-resolution frames (480p).

Revised: For T-VOS, this backbone is pre-trained on ImageNet (Deng et al., 2009), whereas for CRW we re-train it on MOVi/MOVi++.

Revised: For a fairer comparison, we adapt T-VOS (Zhang et al., 2020) to our setting by using the same CNN encoder architecture at the same resolution as in SAVi and trained using the same supervision signal (i.e. optical ﬂow of moving objects).

Original: While this adapted baseline ( Segmentation Propagation in
Revised: While this adapted baseline ( Segmentation Propagation in

Original: Table 1b) performs similar to T-VOS on MOVi, there is a signiﬁcant drop in performance on MOVi++.
Revised: Table 1b) performs similar to T-VOS on MOVi, there is a signiﬁcant drop in performance on MOVi++.

Original: We verify that SAVi can similarly beneﬁt from using a stronger ResNet backbone on MOVi++ ( SAVi (ResNet) in Table 1b), which closes the gap to the CRW (Jabri et al., 2020) baseline.
Revised: We verify that SAVi can similarly beneﬁt from using a stronger ResNet backbone on MOVi++ ( SAVi (ResNet) in Table 1b), which closes the gap to the CRW (Jabri et al., 2020) baseline.

Original: Conditioning Ablations Surprisingly, we ﬁnd that even simple hints about objects in the ﬁrst video frame such as bounding boxes, or even just a point in the center of a bounding box (termed as center of mass ), sufﬁce for the model to establish correspondence between hint and visual input, and to retain the same quality in object segmentation and tracking.
Revised: Conditioning Ablations Surprisingly, we ﬁnd that even simple hints about objects in the ﬁrst video frame such as bounding boxes, or even just a point in the center of a bounding box (termed as center of mass ), sufﬁce for the model to establish correspondence between hint and visual input, and to retain the same quality in object segmentation and tracking.

Original: Conditioning on such simple signals is not possible with segmentation propagation models (Zhang et al., 2020; Jabri et al., 2020) as they require accurate segmentation masks as input.
Revised: Conditioning on such simple signals is not possible with segmentation propagation models (Zhang et al., 2020; Jabri et al., 2020) as they require accurate segmentation masks as input.

Original: To study whether an even weaker conditioning signal sufﬁces, we add noise — sampled from a Gaussian distribution N (0 , σ ) — to the center-of-mass coordinates provided to the model in the ﬁrst frame, both during training and testing.
Revised: To study whether an even weaker conditioning signal sufﬁces, we add noise — sampled from a Gaussian distribution N (0 , σ ) — to the center-of-mass coordinates provided to the model in the ﬁrst frame, both during training and testing.

Original: We ﬁnd that SAVi remains robust to noise scales σ up to ∼ 20 % of the average object size (Figure 3b), after which point mIoU starts to decay (and faster than FG-ARI), indicating that SAVi starts to lose correspondence between  the conditioning signal and the objects in the scene.
Revised: We ﬁnd that SAVi remains robust to noise scales σ up to ∼ 20 % of the average object size (Figure 3b), after which point mIoU starts to decay (and faster than FG-ARI), indicating that SAVi starts to lose correspondence between [Figure] [Figure] the conditioning signal and the objects in the scene.

Original: Full ablation of the conditioning signal leads to a signiﬁcant drop in performance (Figure 3), which demonstrates that conditional hints are indeed important to SAVi, though even very rough hints sufﬁce.
Revised: Full ablation of the conditioning signal leads to a signiﬁcant drop in performance (Figure 3), which demonstrates that conditional hints are indeed important to SAVi, though even very rough hints sufﬁce.

Original: Optical ﬂow ablations Using ground-truth optical ﬂow as a training target — especially in the absence of camera motion — is a form of weak supervision that won’t be available for real data.
Revised: Optical ﬂow ablations Using ground-truth optical ﬂow as a training target — especially in the absence of camera motion — is a form of weak supervision that won’t be available for real data.

Original: To investigate its beneﬁts and importance, we ﬁrst apply k-means clustering to the optical ﬂow signal (enriched with pixel position information) where we use the center of mass coordinates of objects to initialize the cluster centers.
Revised: To investigate its beneﬁts and importance, we ﬁrst apply k-means clustering to the optical ﬂow signal (enriched with pixel position information) where we use the center of mass coordinates of objects to initialize the cluster centers.

Original: The results (see Table 1b) show that simply clustering the optical ﬂow signal itself is insufﬁcient for accurate segmentation.
Revised: The results (see Table 1b) show that simply clustering the optical ﬂow signal itself is insufﬁcient for accurate segmentation.

Original: Next, we perform the following ablations[Figure] on a SAVi model with bounding boxes as conditional input, summarized in Figure 3: 1) instead of using ground-truth optical ﬂow provided by the simulator/renderer, we use approximate optical ﬂow obtained from a recent unsupervised ﬂow estimation model (Stone et al., 2021) ( estimated ﬂow ); 2) we replace optical ﬂow with RGB pixel values as training targets, i.e. using a reconstruction target ( w/o ﬂow ); 3) we train SAVi without conditioning by initializing slots using ﬁxed parameters that are learned during training ( unconditional ).
Revised: Next, we perform the following ablations on a SAVi model with bounding boxes as conditional input, summarized in Figure 3: 1) instead of using ground-truth optical ﬂow provided by the simulator/renderer, we use approximate optical ﬂow obtained from a recent unsupervised ﬂow estimation model (Stone et al., 2021) ( estimated ﬂow ); 2) we replace optical ﬂow with RGB pixel values as training targets, i.e. using a reconstruction target ( w/o ﬂow ); 3) we train SAVi without conditioning by initializing slots using ﬁxed parameters that are learned during training ( unconditional ).

Original: We ﬁnd, that estimated ﬂow confers virtually the same beneﬁts as ground truth-ﬂow, and that ﬂow is unnecessary for the simpler MOVi dataset.
Revised: We ﬁnd, that estimated ﬂow confers virtually the same beneﬁts as ground truth-ﬂow, and that ﬂow is unnecessary for the simpler MOVi dataset.

Original: For
Revised: For

Original: MOVi++, on the other hand, it is crucial for learning an accurate decomposition of the scene, and without access to optical ﬂow (true or estimated) SAVi fails to learn meaningful object segmentations.
Revised: MOVi++, on the other hand, it is crucial for learning an accurate decomposition of the scene, and without access to optical ﬂow (true or estimated) SAVi fails to learn meaningful object segmentations.

Original: Finally, we apply SAVi to a variation of MOVi++ which incorporates (linear) camera movement.

Original: This effectively leads to colored background in the optical ﬂow images, and weakens the usefulness of optical ﬂow for foreground-background segmentation.

Original: Nonetheless, SAVi with bounding box conditioning achieves approx.

Original: 5 ± 0 .

Original: 5% FG-ARI on this modiﬁed dataset, and we ﬁnd that it is still able to segment and track objects, although at a lower ﬁdelity than in MOVi++.

Original: 4. T EST TIME GENERALIZATION
Revised: 4. T EST TIME GENERALIZATION

Original: Longer sequences For memory efﬁciency, we train SAVi on subsequences with a total of 6 frames, i.e. shorter than the full video length.
Revised: Longer sequences For memory efﬁciency, we train SAVi on subsequences with a total of 6 frames, i.e. shorter than the full video length.

Original: We ﬁnd, however, that even without any additional data augmentation or regularization, SAVi generalizes well to longer sequences at test time, far beyond the setting used in training.
Revised: We ﬁnd, however, that even without any additional data augmentation or regularization, SAVi generalizes well to longer sequences at test time, far beyond the setting used in training.

Original: We ﬁnd that segmentation quality measured in terms of FG-ARI remains largely stable, and surprisingly even increases for unconditional models (Figure 6a).
Revised: We ﬁnd that segmentation quality measured in terms of FG-ARI remains largely stable, and surprisingly even increases for unconditional models (Figure 6a).

Original: Part of the increase can be explained by objects leaving the ﬁeld of view later in the video (see examples in the appendix), but models nonetheless appear to beneﬁt from additional time steps to reliably break symmetry and bind objects to slots.
Revised: Part of the increase can be explained by objects leaving the ﬁeld of view later in the video (see examples in the appendix), but models nonetheless appear to beneﬁt from additional time steps to reliably break symmetry and bind objects to slots.

Original: The opposite effect can be observed for segmentation propagation models: performance typically decays over time as errors accumulate.
Revised: The opposite effect can be observed for segmentation propagation models: performance typically decays over time as errors accumulate.

Original: New objects and backgrounds We ﬁnd that SAVi generalizes surprisingly well to settings with previously unseen objects and with new backgrounds at test time: there is no signiﬁcant difference in both FG-ARI and mIoU metrics when evaluating SAVi with bounding box conditioning on evaluation sets of MOVi++ with entirely new objects or backgrounds.
Revised: New objects and backgrounds We ﬁnd that SAVi generalizes surprisingly well to settings with previously unseen objects and with new backgrounds at test time: there is no signiﬁcant difference in both FG-ARI and mIoU metrics when evaluating SAVi with bounding box conditioning on evaluation sets of MOVi++ with entirely new objects or backgrounds.

Original: On an evaluation set where we provide both new backrounds and new objects at the same time, there is a drop of less than 2% (absolute), see
Revised: On an evaluation set where we provide both new backrounds and new objects at the same time, there is a drop of less than 2% (absolute), see

Original: Figure 6b.
Revised: Figure 6b.

Original: Remarkably, a SAVi model trained on MOVi++ also transfers well to MOVi at test time.
Revised: Remarkably, a SAVi model trained on MOVi++ also transfers well to MOVi at test time.

Original: Part-whole segmentation Our dataset is annotated only at the level of whole objects, yet we ﬁnd anecdotal evidence that by conditioning at the level of parts during inference the model can in fact successfully segment and track the corresponding parts.
Revised: Part-whole segmentation Our dataset is annotated only at the level of whole objects, yet we ﬁnd anecdotal evidence that by conditioning at the level of parts during inference the model can in fact successfully segment and track the corresponding parts.

Original: Consider, for example the two green ﬁsts in Figure 7 which in the dataset are treated as a single composite object.
Revised: Consider, for example the two green ﬁsts in Figure 7 which in the dataset are treated as a single composite object.

Revised: When conditioned with a

Original: When conditioned with a single bounding box encompassing both ﬁsts ( Whole ), or on two bounding boxes ( Parts ), SAVi either segments and tracks either the composite object or the individual parts (see appendix for an additional example).
Revised: [Figure] Conditioningsignal [Figure] single bounding box encompassing both ﬁsts ( Whole ), or on two bounding boxes ( Parts ), SAVi either segments and tracks either the composite object or the individual parts (see appendix for an additional example).

Original: This ﬁnding is remarkable for two reasons:
Revised: This ﬁnding is remarkable for two reasons:

Original: Firstly, it shows that the model can (at least to some degree) generalize to different levels of granularity.
Revised: Firstly, it shows that the model can (at least to some degree) generalize to different levels of granularity.

Original: This kind of generalization is very useful, since the notion of an object is generally task-dependent and the correct level of granularity might not [Figure] Conditioningsignal for:
Revised: This kind of generalization is very useful, since the notion of an object is generally task-dependent and the correct level of granularity might not be known at training time.

Original: Whole

Original: Parts

Original: V i

Original: d e o t= [Figure] be known at training time. Secondly, this experiments demonstrates a powerful means for interfacing with the model: rather than producing a single ﬁxed decomposition, SAVi has learned to adapt its segmentation and processing of the video based on the conditioning signal.
Revised: Secondly, this experiments demonstrates a powerful means for interfacing with the model: rather than producing a single ﬁxed decomposition, SAVi has learned to adapt its segmentation and processing of the video based on the conditioning signal.

Original: In fact, we believe that this way of learning a part-whole hierarchy only implicitly and segmenting different levels on demand is ultimately more ﬂexible and tractable than trying to explicitly represent the entire hierarchy at once.
Revised: We believe that this way of learning a part-whole hierarchy only implicitly and segmenting different levels on demand is ultimately more ﬂexible and tractable than trying to explicitly represent the entire hierarchy at once.

Original: In this example we used conditioning to communicate the desired level of object granularity at test time, but we hypothesize that it could also serve as an interface for other tasks such as visual search or directing attention.

Original: See Appendix A.4 for preliminary experiments about using conditioning to select and track only a subset of objects.

Original: 4. L IMITATIONS
Revised: 4. L IMITATIONS

Original: There are still several obstacles that have to be overcome for successful application of our framework to the full visual and dynamic complexity of the real world.
Revised: There are still several obstacles that have to be overcome for successful application of our framework to the full visual and dynamic complexity of the real world.

Original: First, our training set up assumes the availability of optical ﬂow information at training time, which may not be available in realworld videos.
Revised: First, our training set up assumes the availability of optical ﬂow information at training time, which may not be available in realworld videos.

Original: Our experiments, however, have demonstrated that estimated optical ﬂow from an unsupervised model such as SMURF (Stone et al., 2021) can be sufﬁcient to train SAVi.
Revised: Our experiments, however, have demonstrated that estimated optical ﬂow from an unsupervised model such as SMURF (Stone et al., 2021) can be sufﬁcient to train SAVi.

Original: Secondly, the environments considered in our work, despite taking a large step towards realism compared to prior works, are limited by containing solely rigid objects with simple physics and, in terms of the MOVi datasets, solely moving objects.
Revised: Secondly, the environments considered in our work, despite taking a large step towards realism compared to prior works, are limited by containing solely rigid objects with simple physics and, in terms of the MOVi datasets, solely moving objects.

Original: In this case, optical ﬂow, even in the presence of camera movement, can be interpreted as an indirect supervision signal for foreground-background (but not per-object) segmentation.
Revised: In this case, optical ﬂow, even in the presence of camera movement, can be interpreted as an indirect supervision signal for foreground-background (but not per-object) segmentation.

Original: Furthermore, training solely with optical ﬂow presents a challenge for static objects.
Revised: Furthermore, training solely with optical ﬂow presents a challenge for static objects.

Original: Nonetheless, we ﬁnd evidence that SAVi can reliably decompose videos in a constrained real-world environment: we provide qualitative results demonstrating scene decomposition and long-term tracking in a robotic grasping environment (Cabi et al., 2020) in Appendix A.3.
Revised: Nonetheless, we ﬁnd evidence that SAVi can reliably decompose videos in a constrained real-world environment: we provide qualitative results demonstrating scene decomposition and long-term tracking in a robotic grasping environment (Cabi et al., 2020) in Appendix A.3.

Original: Most real-world video data, however, is still of signiﬁcantly higher complexity than the environments considered here and reliably bridging this gap is an open problem.
Revised: Most real-world video data, however, is still of signiﬁcantly higher complexity than the environments considered here and reliably bridging this gap is an open problem.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have looked at the problem of learning object representations and physical dynamics from video.
Revised: We have looked at the problem of learning object representations and physical dynamics from video.

Original: We introduced SAVi, an object-centric architecture that uses attention over a latent set of slots to discover, represent and temporally track individual entities in the input, and we have shown that video in general and ﬂow prediction in particular are helpful for identifying and tracking objects.
Revised: We introduced SAVi, an object-centric architecture that uses attention over a latent set of slots to discover, represent and temporally track individual entities in the input, and we have shown that video in general and ﬂow prediction in particular are helpful for identifying and tracking objects.

Original: We demonstrated that providing the model with simple and possibly unreliable position information about the objects in the ﬁrst frame is sufﬁcient to steer learning towards the right level of granularity to successfully decompose complex videos and simultaneously track and segment multiple objects.
Revised: We demonstrated that providing the model with simple and possibly unreliable position information about the objects in the ﬁrst frame is sufﬁcient to steer learning towards the right level of granularity to successfully decompose complex videos and simultaneously track and segment multiple objects.

Original: More importantly, we show that the level of granularity can potentially be controlled at inference time, allowing a trained model to track either parts of objects or entire objects depending on the given initialization.
Revised: More importantly, we show that the level of granularity can potentially be controlled at inference time, allowing a trained model to track either parts of objects or entire objects depending on the given initialization.

Original: Since our model achieves very reasonable segmentation and tracking performance, this is evidence that object-centric representation learning is not primarily limited by model capacity.
Revised: Since our model achieves very reasonable segmentation and tracking performance, this is evidence that object-centric representation learning is not primarily limited by model capacity.

Original: This way of conditioning the initialization of slot representations with location information also potentially opens the door for a wide range of semi-supervised approaches.
Revised: This way of conditioning the initialization of slot representations with location information also potentially opens the door for a wide range of semi-supervised approaches.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our analysis, which is focused on multi-object video data with physically simulated common household objects, has no immediate impact on general society.
Revised: Our analysis, which is focused on multi-object video data with physically simulated common household objects, has no immediate impact on general society.

Original: As with any model that performs dynamic scene understanding, applications with potential negative societal impact such as in the area of surveillance, derived from future research in this area, cannot be fully excluded.
Revised: As with any model that performs dynamic scene understanding, applications with potential negative societal impact such as in the area of surveillance, derived from future research in this area, cannot be fully excluded.

Original: We anticipate that future work will take steps to close the gap between the types of environments considered in this work and videos of diverse scenes taken in the real world, which will likely enable applications in various societally relevant domains such as robotics and general video understanding.
Revised: We anticipate that future work will take steps to close the gap between the types of environments considered in this work and videos of diverse scenes taken in the real world, which will likely enable applications in various societally relevant domains such as robotics and general video understanding.

Original: R EPRODUCIBILITY S TATEMENT
Revised: A CKNOWLEDGEMENTS

Original: To ensure reproducibility, we commit to releasing a JAX/Flax (Bradbury et al., 2018;
Revised: We would like to thank Mike Mozer and Daniel Keysers for general advice and feedback on the paper, and David Fleet, Dirk Weissenborn, Jakob Uszkoreit, Marvin Ritter, Andreas Steiner and Etienne

Original: Heek et al., 2020) implementation for SAVi and the MOVi and MOVi++ datasets used in our experiments in the near future (latest upon publication). We  further report all hyperparameters, hardware details, training time, and model details in Section 4 and Appendix A.
Revised: Pot for helpful discussions. We are further grateful to Rishabh Kabra for sharing the CATER (with masks) dataset and to Yi Yang, Misha Denil, Yusuf Aytar and Claudio Fantacci for helping us get started with the Sketchy dataset.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In Section A. we discuss additional related work.
Revised: In Section A.1, we discuss additional related work.

Original: In Sections A.2–A. we show additional qualitative and quantitative results.
Revised: In Sections A.2–A.4, we show additional qualitative and quantitative results.

Original: Section A.4 further contains results for an ablation study on SAVi.
Revised: Section A.4 further contains results for an ablation study on SAVi.

Original: Lastly in Sections A 5–A we report additional details on datasets, model architecture and training setup (hyperparameters, baselines, metrics).
Revised: Lastly, in Sections A.5–A.8, we report additional details on datasets, model architecture, and training setup (hyperparameters, baselines, metrics).

Original: A. A DDITIONAL RELATED WORK
Revised: A. A DDITIONAL RELATED WORK

Original: Attention-basedmodular networks  SAVi shares close connections with other attention-based modular neural networks.
Revised: Attention-based networks for sets of latent variables SAVi shares close connections with other attention-based modular neural networks.

Original: Relational Recurrent Networks (Santoro et al., 2018) and the RIM model (Goyal et al., 2021c) use related attention mechanisms to map from inputs to slots and from slots to slots, the latter of which (RIM) uses specialized neural network parameters for each slot.
Revised: Relational Recurrent Networks (Santoro et al., 2018) and the RIM model (Goyal et al., 2021c) use related attention mechanisms to map from inputs to slots and from slots to slots, the latter of which (RIM) uses specialized neural network parameters for each slot.

Original: Follow-up works (Goyal et al.
Revised: Follow-up works (Goyal et al.

Original: , 2021b;a) introduce additional sparsity and factorization constraints on the dynamics model, which is orthogonal to our contributions of conditional learning and dynamics modeling using optical ﬂow.
Revised: , 2021b;a) introduce additional sparsity and factorization constraints on the dynamics model, which is orthogonal to our contributions of conditional learning and dynamics modeling using optical ﬂow.

Revised: The Perceiver (Jaegle et al., 2021b) uses an architecture similar to Slot

Revised: Attention (Locatello et al., 2020), i.e. using a set of latent vectors which are (recurrently) updated by attending on a set of input feature vectors, for classiﬁcation tasks.

Revised: The Perceiver IO (Jaegle et al., 2021a) model extends this architecture by adding an attention-based decoder component to interface with a variety of structured prediction tasks.

Revised: SAVi similarly maps inputs onto a latent set of slot vectors using an attention mechanism, but differs in the decoder architecture and in its sequential nature: different from Perceiver IO, SAVi processes a sequence of inputs (e.g. video frames) of arbitrary length, encoding and decoding a single input (or output) element per time step while maintaining a latent set of slot variables that is carried forward in time.

Original: Supervised slot-based models for visual tracking Slot-based architectures have also been explored in the context of fully supervised multi-object tracking and segmentation with models such as
Revised: Supervised slot-based models for visual tracking Slot-based architectures have also been explored in the context of fully supervised multi-object tracking and segmentation with models such as

Original: MOHART (Fuchs et al., 2019), TrackFormer (Meinhardt et al., 2021), and TubeR (Zhao et al., 2021).
Revised: MOHART (Fuchs et al., 2019), TrackFormer (Meinhardt et al., 2021), and TubeR (Zhao et al., 2021).

Original: In the image domain, GPV-I (Tanmay Gupta, 2021) and MDETR (Kamath et al., 2021; Carion et al., 2020) explore conditioning supervised slot-based models on auxiliary information, in a form which requires combinatorial matching during training.
Revised: In the image domain, GPV-I (Tanmay Gupta, 2021) and MDETR (Kamath et al., 2021; Carion et al., 2020) explore conditioning supervised slot-based models on auxiliary information, in a form which requires combinatorial matching during training.

Original: SAVi learns to track and segment multiple objects without being directly supervised to do so, using only weak forms of supervision for the ﬁrst frame of the video.
Revised: SAVi learns to track and segment multiple objects without being directly supervised to do so, using only weak forms of supervision for the ﬁrst frame of the video.

Original: Slot conditioning allows us to avoid matching and to consistently identify and track individual objects by their speciﬁed query.
Revised: Slot conditioning allows us to avoid matching and to consistently identify and track individual objects by their speciﬁed query.

Original: Dynamic visual reasoning and action graphs Object-centric models for dynamic scenes have found success in other relational tasks in vision, such as supervised visual reasoning (Yi et al., 2020;
Revised: Dynamic visual reasoning and action graphs Object-centric models for dynamic scenes have found success in other relational tasks in vision, such as supervised visual reasoning (Yi et al., 2020;

Original: Chen et al., 2021; Ding et al., 2021a;b), activity/action recognition (Jain et al., 2016; Girdhar et al., 2019; Herzig et al., 2019), and compositional video synthesis with action graphs (Bar et al., 2021).
Revised: Chen et al., 2021; Ding et al., 2021a;b), activity/action recognition (Jain et al., 2016; Girdhar et al., 2019; Herzig et al., 2019), and compositional video synthesis with action graphs (Bar et al., 2021).

Original: Many of these approaches use pre-trained (Chen et al., 2021) and sometimes frozen (Yi et al., 2020;
Revised: Many of these approaches use pre-trained (Chen et al., 2021) and sometimes frozen (Yi et al., 2020;

Original: Ding et al., 2021a) object detection backbones.
Revised: Ding et al., 2021a) object detection backbones.

Original: Combining these methods with an end-to-end video object discovery architecture such as SAVi using self-supervised learning objectives is an interesting direction for future work.
Revised: Combining these methods with an end-to-end video object discovery architecture such as SAVi using self-supervised learning objectives is an interesting direction for future work.

Original: A. R EAL - WORLD ROBOTICS TASK
Revised: A. R EAL - WORLD ROBOTICS TASK

Original: To evaluate whether SAVi can decompose real-world video data in a limited domain, we train and qualitatively evaluate an unsupervised SAVi model with RGB reconstruction on the Sketchy dataset [1,2].
Revised: To evaluate whether SAVi can decompose real-world video data in a limited domain, we train and qualitatively evaluate an unsupervised SAVi model with RGB reconstruction on the Sketchy dataset (Cabi et al., 2020).

Original: The Sketchy dataset contains videos of a real-world robotic grasper interacting with various objects.
Revised: The Sketchy dataset contains videos of a real-world robotic grasper interacting with various objects.

Original: We use the human demonstration sequences as part of the “ rgb30 all ” subset of the Sketchy dataset (Cabi et al., 2020), resulting in a total of 2930 training videos of 201 frames each.
Revised: We use the human demonstration sequences as part of the “ rgb30 all ” subset of the Sketchy dataset (Cabi et al., 2020), resulting in a total of 2930 training videos of 201 frames each.

Original: In
Revised: In

Original: Figure A.1 and in the supplementary videos (see supplementary material), we show qualitative results on unseen validation data.
Revised: Figure A.1 and in the supplementary videos (see supplementary material), we show qualitative results on unseen validation data.

Original: Our qualitative results in Figure A.1 demonstrate that SAVi can decompose these real-world scenes into meaningful object components and consistently represent and track individual scene components over long time horizons, far beyond what is observed during training.
Revised: Our qualitative results in Figure A.1 demonstrate that SAVi can decompose these real-world scenes into meaningful object components and consistently represent and track individual scene components over long time horizons, far beyond what is observed during training.

Original: SAVi is trained with a per-slot
Revised: SAVi is trained with a per-slot

Original: MLP predictor model (with a single hidden layer of 256 hidden units, a skip connection, and Layer
Revised: MLP predictor model (with a single hidden layer of 256 hidden units, a skip connection, and Layer

Original: Normalization) for 1M steps with otherwise the same architecture as the unsupervised SAVi model used for CATER described in the main paper.V i d e o t= R e c o n .
Revised: Normalization) for 1M steps with otherwise the same architecture as the unsupervised SAVi model used for CATER described in the main paper.

Original: P r e d .

Original: [-S l o t 
-]
Revised: [-S l o t 
-]

Original: S l o t S l o t S l o t S l o t
Revised: A.

Original: S l o t

Original: S l o t

Original: S l o t

Original: S l o t S l o t S l o t A. A DDITIONAL QUALITATIVE RESULTS
Revised: A DDITIONAL QUALITATIVE RESULTS

Original: Part-whole segmentation In Figure A.2 we show another example of steerable part-whole segmentation based on the granularity of the hint provided for the ﬁrst frame.
Revised: Part-whole segmentation In Figure A.2 we show another example of steerable part-whole segmentation based on the granularity of the hint provided for the ﬁrst frame.

Original: In this example, the laptop in the foreground of the scene is either annotated with a single bounding box that covers the full object (which is the granularity of annotation originally provided in the dataset), or we re-annotate this object as two separate instances, where one bounding box covers the screen of the laptop and the other bounding box covers the base incl.
Revised: In this example, the laptop in the foreground of the scene is either annotated with a single bounding box that covers the full object (which is the granularity of annotation originally provided in the dataset), or we re-annotate this object as two separate instances, where one bounding box covers the screen of the laptop and the other bounding box covers the base incl.

Original: the keyboard.
Revised: the keyboard.

Original: Despite never having seen a separate ”screen” object in the dataset, the model can reliably identify, segment, and track each individual part, as can be seen both by the predicted decoder segmentation masks and the corrector attention masks.
Revised: Despite never having seen a separate ”screen” object in the dataset, the model can reliably identify, segment, and track each individual part, as can be seen both by the predicted decoder segmentation masks and the corrector attention masks.

Original: Alternatively, if the conditional input covers the full object, the model tracks and segments the entire laptop using a single slot, here shown in green.
Revised: Alternatively, if the conditional input covers the full object, the model tracks and segments the entire laptop using a single slot, here shown in green.

Original: We further observe that the corrector attention masks much more sharply outline the object compared to the decoder segmentation masks, but have artifacts for large objects which is likely due to the limited receptive ﬁeld of the encoder CNN used in our work (for reasons of simplicity).
Revised: We further observe that the corrector attention masks much more sharply outline the object compared to the decoder segmentation masks, but have artifacts for large objects which is likely due to the limited receptive ﬁeld of the encoder CNN used in our work (for reasons of simplicity).

Original: Both this gap in sharpness and the artifacts seen in the corrector masks for large objects can likely be addressed in future work by using more powerful encoder and decoder architectures.[Figure] [Figure]
Revised: Both this gap in sharpness and the artifacts seen in the corrector masks for large objects can likely be addressed in future work by using more powerful encoder and decoder architectures.

Original: [Figure]
Revised: [Figure]

Original: the two conditioning cases in terms of the decoder segmentation masks and corrector attention masks, respectively.

Original: Unsupervised video decomposition In Figure A.3 we show qualitative results for fullyunsupervised video decomposition with SAVi on the CATER dataset (on unseen validation data).
Revised: Unsupervised video decomposition In Figure A.3 we show qualitative results for fullyunsupervised video decomposition with SAVi on the CATER dataset (on unseen validation data).

Original: Since SAVi is trained with a RGB reconstruction loss in this setting, we can visualize reconstructions of individual objects over multiple time steps in the video.
Revised: Since SAVi is trained with a RGB reconstruction loss in this setting, we can visualize reconstructions of individual objects over multiple time steps in the video.

Original: Extrapolation to long sequences In Figures
Revised: Extrapolation to long sequences In Figures

Original: A.4–A.5 we show several representative examples of applying SAVi with bounding box conditioning on full-length MOVi and MOVi++ videos offrames, after being trained on sequences of only 6 frames.
Revised: A.4–A.5 we show several representative examples of applying SAVi with bounding box conditioning on full-length MOVi and MOVi++ videos offrames, after being trained on sequences of only 6 frames.

Original: Moving camera
Revised: Moving camera

Original: In Figure A.6 we show qualitative results for SAVi (with bounding box conditioning) on a more challenging variant of the MOVi++ dataset with  camera movement, where optical ﬂow encodes both object movement and movement of the background relative to the camera.
Revised: In Figure A.6 we show qualitative results for SAVi (with bounding box conditioning) on a more challenging variant of the MOVi++ dataset with (linear) camera movement, where optical ﬂow encodes both object movement and movement of the background relative to the camera.

Revised: This effectively leads to colored background in the optical ﬂow images, and weakens the usefulness of optical ﬂow for foreground-background segmentation.

Original: We ﬁnd that SAVi is still able to identify, segment, and track objects in this case, but at a lower ﬁdelityand tracking accuracy than a comparable model trained on MOVi++ with a static camera.
Revised: Nonetheless, SAVi with bounding box conditioning achieves approx. 5 ± 0 . 5% FG-ARI on this modiﬁed dataset, and we ﬁnd that it is still able to segment and track objects, although at a lower ﬁdelity than in MOVi++.

Original: Baselines: Segmentation Propagation
Revised: Baselines: Segmentation Propagation

Original: In Figure A.7 we show qualitative results for the Segmentation Propagation baseline on a couple of sequences each in the MOVi and MOVi++ datasets.
Revised: In Figure A.7 we show qualitative results for the Segmentation Propagation baseline on a couple of sequences each in the MOVi and MOVi++ datasets.

Original: In these image grids, ‘Pred.’ corresponds to the soft segmentation masks before argmax similar to the illustrations for SAVi above.
Revised: In these image grids, ‘Pred.’ corresponds to the soft segmentation masks before argmax similar to the illustrations for SAVi above.

Original: We also show ‘Pred.
Revised: We also show ‘Pred.

Original: Argmax’ which corresponds to the hardsegmentation masks post argmax . The last row in each image grid corresponds to optical ﬂow predicted from a single image. This is expected to be poor since ﬂow prediction from a single image [Figure]
Revised: Argmax’ which corresponds to the hard

Original: [Figure]

Original: V i
Revised: V i

Original: d e o t= G 
Revised: d e o t= G . T .

Original: [Figure]

Original: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: G .

Original: T .

Original: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: [Figure]

Original: [Figure]

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: [Figure]

Original: V i

Original: d e o t=

Original: G

Original: [Figure]

Original: T

Original: [Figure]

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .

Original: T .

Original: P r e d .

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d .
Revised: P r e d .

Original: G .
Revised: G .

Original: [Figure] P r [Figure]
Revised: T .

Original: i
Revised: P

Original: [Figure]
Revised: [Figure]

Original: o t
Revised: d .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: V i
Revised: V i

Original: d e o t=
Revised: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d . P r e d .
Revised: P r e d . P r e d .

Original: A r g m a x
Revised: A r g m a x

Original: G .

Original: T .

Original: P r e d .

Original: V i

Original: d e o t=

Original: G .
Revised: G .

Original: T .
Revised: T .

Original: P r e d . P r e d . A r g m a x G .
Revised: P r e d . segmentation masks post argmax .

Original: T .

Original: P r e d . V i d e o t= G .
Revised: The last row in each image grid corresponds to optical ﬂow predicted from a single image.

Original: T .

Original: P r e d .

Original: P r e d .

Original: A r g m a x

Original: G . T . [Figure] is a highly ambiguous task.
Revised: This is expected to be poor since ﬂow prediction from a single image is a highly ambiguous task.

Original: We ﬁnd that the Segmentation Propagation baseline struggles to retain small objects.
Revised: We ﬁnd that the Segmentation Propagation baseline struggles to retain small objects.

Original: These often disappear or become part of a bigger object.
Revised: These often disappear or become part of a bigger object.

Original: The soft masks also suggest that the model is struggling to keep the background separate from objects.
Revised: The soft masks also suggest that the model is struggling to keep the background separate from objects.

Original: 9 continuous reference frames were used for label propagation in Segmentation Propagation.
Revised: 9 continuous reference frames were used for label propagation in Segmentation Propagation.

Original: See Section A.7 for more details.
Revised: See Section A.7 for more details.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Baselines: SCALOR Figures A.8 and A.9 show qualitative results for the SCALOR baseline on examples from the MOVi and MOVi++ datasets.
Revised: Baselines: SCALOR Figures A.8 and A.9 show qualitative results for the SCALOR baseline on examples from the MOVi and MOVi++ datasets.

Original: Often, when small objects in MOVi get close to each other, SCALOR merges them as one object.
Revised: Often, when small objects in MOVi get close to each other, SCALOR merges them as one object.

Original: On MOVi++, SCALOR struggles with object boundaries and often picks up background texture as extra objects.
Revised: On MOVi++, SCALOR struggles with object boundaries and often picks up background texture as extra objects.

Original: A. A DDITIONAL QUANTITATIVE RESULTS
Revised: A. A DDITIONAL QUANTITATIVE RESULTS

Original: Transfer to static scenes A key advantage of SAVi over methods that segment objects using optical ﬂow as input, is that SAVi can be applied to static images where optical ﬂow is not available.
Revised: Transfer to static scenes A key advantage of SAVi over methods that segment objects using optical ﬂow as input, is that SAVi can be applied to static images where optical ﬂow is not available.

Original: We demonstrate generalization to static images by evaluating ‘SAVi + Bounding box’ models on the ﬁrst frame of all videos in the validation set.
Revised: We demonstrate generalization to static images by evaluating ‘SAVi + Bounding box’ models on the ﬁrst frame of all videos in the validation set.

Original: To this end, we repeat the ﬁrst frame twice to form a “boring” video to test whether the model can accurately segment and “track” objects even in the absence of motion.
Revised: To this end, we repeat the ﬁrst frame twice to form a “boring” video to test whether the model can accurately segment and “track” objects even in the absence of motion.

Original: In this setting, our model achieves a FG-ARI score of 88 .
Revised: In this setting, our model achieves a FG-ARI score of 88 .

Original: 7 ± 0 .
Revised: 7 ± 0 .

Original: 2 on the MOVi++ dataset, i.e. it accurately segments the static scene.
Revised: 2 on the MOVi++ dataset, i.e. it accurately segments the static scene.

Original: Tracking subsets of objects We investigate interfacing with SAVi to track a subset of the objects present in the ﬁrst frame.
Revised: Tracking subsets of objects We investigate interfacing with SAVi to track a subset of the objects present in the ﬁrst frame.

Original: We achieve this by conditioning slots using bounding boxes/segmentation masks for some of the objects and ignoring the bounding boxes/segmentation masks for the remaining objects.
Revised: We achieve this by conditioning slots using bounding boxes/segmentation masks for some of the objects and ignoring the bounding boxes/segmentation masks for the remaining objects.

Original: We found it beneﬁcial, for these experiments, to initialize unconditioned slots with random vectors, so that they are free to model any remaining objects.
Revised: We found it beneﬁcial, for these experiments, to initialize unconditioned slots with random vectors, so that they are free to model any remaining objects.

Original: We also found it beneﬁcial to expose the model to this scenario at training time.
Revised: We also found it beneﬁcial to expose the model to this scenario at training time.

Original: We encoded whether a slot received conditional input or not using a binary indicator which we appended to the initial slot value.
Revised: We encoded whether a slot received conditional input or not using a binary indicator which we appended to the initial slot value.

Original: [Figure]
Revised: [Figure]

Original:  experimental conditions conditioning.
Revised: Figure A.10 reports mIoU across various experimental conditions. The top two plots study SAVi under bounding box conditioning and the bottom two plots study SAVi under segmentation mask conditioning.

Original: Varying subset sizes for evaluation: For the right two plots in Figure A.10, we train SAVi with up to 6 objects and evaluate with varying subset sizes.
Revised: Varying subset sizes for evaluation: For the right two plots in Figure A.10, we train SAVi with up to 6 objects and evaluate with varying subset sizes.

Original: We observe that, selecting and tracking subsets of objects is indeed possible, but the introduction of randomly initialized slots comes at a price: with more objects without a conditioning signal, therefore needing to be explained by randomly initialized slots, the model’s ability to connect the provided hints to the correct objects decays, as can be observed by a decrease in the mIoU score.
Revised: We observe that, selecting and tracking subsets of objects is indeed possible, but the introduction of randomly initialized slots comes at a price: with more objects without a conditioning signal, therefore needing to be explained by randomly initialized slots, the model’s ability to connect the provided hints to the correct objects decays, as can be observed by a decrease in the mIoU score.

Original: Varying subset sizes for training:
Revised: Varying subset sizes for training:

Original: In the left two plots in Figure A.10, we train SAVi with various subsets of size s ∈ { 1 , 3 , 6 , 10 } and evaluate on 3 objects at test time.
Revised: In the left two plots in Figure A.10, we train SAVi with various subsets of size s ∈ { 1 , 3 , 6 , 10 } and evaluate on 3 objects at test time.

Original: By training on subsets of objects, we are able to ablate between an unsupervised and a fully conditioned setting.
Revised: By training on subsets of objects, we are able to ablate between an unsupervised and a fully conditioned setting.

Original: As seen in the plots, when ‘maximum training objects’ is 1, that is 10 randomly initialized slots and 1 conditioned slot are used, the model is not able to establish a correspondence between the conditioning hint and the corresponding object in the scene.
Revised: As seen in the plots, when ‘maximum training objects’ is 1, that is 10 randomly initialized slots and 1 conditioned slot are used, the model is not able to establish a correspondence between the conditioning hint and the corresponding object in the scene.

Original: In other words, it treats the conditioning hint similar to the randomly initialized slots.
Revised: In other words, it treats the conditioning hint similar to the randomly initialized slots.

Original: On the other hand, providing conditioning for all objects at training time, that is ‘maximum training objects’ is 10, leads to poor generalization to subsets at test time since the model was not exposed to unconditioned objects during training.
Revised: On the other hand, providing conditioning for all objects at training time, that is ‘maximum training objects’ is 10, leads to poor generalization to subsets at test time since the model was not exposed to unconditioned objects during training.

Original: Conditioning on up to 6 objects presents a reasonable trade off between these two extremes.
Revised: Conditioning on up to 6 objects presents a reasonable trade off between these two extremes.

Original: Providing explicit supervision on the alignment of the hint and the corresponding object in the input (which we do not provide) is likely necessary for stabilizing this setup when only few hints are provided.
Revised: Providing explicit supervision on the alignment of the hint and the corresponding object in the input (which we do not provide) is likely necessary for stabilizing this setup when only few hints are provided.

Original: Alternatively, smarter initialization strategies for unconditioned slots can likely improve this setup, which we leave for future work.
Revised: Alternatively, smarter initialization strategies for unconditioned slots can likely improve this setup, which we leave for future work.

Original: Generalization to unseen objects and backgrounds To test for generlization capabilities, we evaluate a SAVi model with bounding box conditioning, which was trained on MOVi++, on the following evaluation splits (see Table A.1):
Revised: Generalization to unseen objects and backgrounds To test for generlization capabilities, we evaluate a SAVi model with bounding box conditioning, which was trained on MOVi++, on the following evaluation splits (see Table A.1):

Original: • Unseen objects : This split only contains novel objects (approx. 100) not seen during training, which are also not part of the default evaluation split.
Revised: • Unseen objects : This split only contains novel objects (approx. 100) not seen during training, which are also not part of the default evaluation split

Original: Remarkably, SAVi generalizes without a 
Revised: Remarkably, SAVi generalizes without a decrease in FG-ARI or mIoU scores on this split.

Original: • Unseen objects : This split only contains novel backgrounds (approx. 40) not seen during training, but otherwise contains solely known objects.
Revised: • Unseen objects : This split only contains novel backgrounds (approx. 40) not seen during training, but otherwise contains solely known objects.

Original: Generalization is comparable as to the default evaluation set.
Revised: Generalization is comparable as to the default evaluation set.

Original: • Unseen objects & backgrounds : When evaluating with both novel objects and backgrounds, there is only a slight decrease in FG-ARI and mIoU scores, i.e. the model still generalizes well.
Revised: • Unseen objects & backgrounds : When evaluating with both novel objects and backgrounds, there is only a slight decrease in FG-ARI and mIoU scores, i.e. the model still generalizes well.

Original: • MOVi : A model trained on MOVi++ also generalizes well to MOVi, here at a resolution of 128 × 128 to be in line with the training setup on MOVi++.
Revised: • MOVi : A model trained on MOVi++ also generalizes well to MOVi, here at a resolution of 128 × 128 to be in line with the training setup on MOVi++.

Original: [Table]
Revised: [Table]

Original: Unconditional video decomposition on MOVi / MOVi++
Revised: Unconditional video decomposition on MOVi / MOVi++

Original: In Table A.2 we report results on unconditional video decomposition on the MOVi and MOVi++ datasets.
Revised: In Table A.2 we report results on unconditional video decomposition on the MOVi and MOVi++ datasets.

Original: For SIMONe, we use our own reimplementation for which we approximately reproduced the CATER results reported by Kabra et al.
Revised: For SIMONe, we use our own reimplementation for which we approximately reproduced the CATER results reported by Kabra et al.

Original: For SCALOR (Jiang et al., 2020), we use the implementation provided by the authors.
Revised: For SCALOR (Jiang et al., 2020), we use the implementation provided by the authors.

Original: Further baseline details are provided in Section A.7.
Revised: Further baseline details are provided in Section A.7.

Original: [Table]
Revised: [Table]

Original: Architecture ablations In this section, we report results on several architecture ablations on SAVi.
Revised: Architecture ablations In this section, we report results on several architecture ablations on SAVi.

Original: In all cases, we evaluate on MOVi++ and condition SAVi on bounding box information in the ﬁrst frame of the video.
Revised: In all cases, we evaluate on MOVi++ and condition SAVi on bounding box information in the ﬁrst frame of the video.

Original: Results are reported in Table A.3.
Revised: Results are reported in Table A.3.

Original: We consider the following ablations:
Revised: We consider the following ablations:

Original: • No predictor : Instead of a Transformer-based (Vaswani et al., 2017) predictor, we use the identity function.
Revised: • No predictor : Instead of a Transformer-based (Vaswani et al., 2017) predictor, we use the identity function.

Original: Modeling the dynamics of the objects is thus entirely ofﬂoaded to the corrector, which does not model interactions.
Revised: Modeling the dynamics of the objects is thus entirely ofﬂoaded to the corrector, which does not model interactions.

Original: We ﬁnd that this can marginally increase FG-ARI scores, likely because the self-attention mechanism in the default model variant can have a regularizing effect.
Revised: We ﬁnd that this can marginally increase FG-ARI scores, likely because the self-attention mechanism in the default model variant can have a regularizing effect.

Original: It, however, negatively affects the model’s ability to learn the correspondence between provided hints and objects in the video, as indicated by the drop in mIoU scores, likely due to its inability to exchange information between learned slot representations in the absence of the Transformer-based predictor.
Revised: It, however, negatively affects the model’s ability to learn the correspondence between provided hints and objects in the video, as indicated by the drop in mIoU scores, likely due to its inability to exchange information between learned slot representations in the absence of the Transformer-based predictor.

Original: • MLP predictor : In this setup, we replace the predictor with an MLP with residual connection and LayerNorm (Ba et al., 2016), applied independently on each slot with shared parameters.
Revised: • MLP predictor : In this setup, we replace the predictor with an MLP with residual connection and LayerNorm (Ba et al., 2016), applied independently on each slot with shared parameters.

Original: It has the same structure and number of parameters as the MLP in the Transformer block used in our default predictor.
Revised: It has the same structure and number of parameters as the MLP in the Transformer block used in our default predictor.

Original: Similar to the no predictor ablation, this variant does not model interactions between slots.
Revised: Similar to the no predictor ablation, this variant does not model interactions between slots.

Original: The effect on performance is comparable to the no predictor ablation.
Revised: The effect on performance is comparable to the no predictor ablation.

Original: • Inverted corrector attention : By default, we use Slot Attention (Locatello et al., 2020) as our corrector.
Revised: • Inverted corrector attention : By default, we use Slot Attention (Locatello et al., 2020) as our corrector.

Original: The attention matrix in Slot Attention is softmax-normalized over the slots,
Revised: The attention matrix in Slot Attention is softmax-normalized over the slots,

Original: which creates competition between slots so that two slots are discouraged to attend to the same object or part (unless their representations are identical)
Revised: which creates competition between slots so that two slots are discouraged to attend to the same object or part (unless their representations are identical)

Original: In this ablation, we normalize the attention matrix instead over the visual input features, as done in e.g. RIM (Goyal et al., 2021c).
Revised: In this ablation, we normalize the attention matrix instead over the visual input features, as done in e.g. RIM (Goyal et al., 2021c).

Original: We observe a sharp drop in the model’s ability to learn the correspondence between hints and objects, as indicated by the low mIoU score in this setting.
Revised: We observe a sharp drop in the model’s ability to learn the correspondence between hints and objects, as indicated by the low mIoU score in this setting.

Original: [Table]
Revised: [Table]

Original: Comparison to semi-supervised training with matching While our approach of conditioning the model on a set of hints, such as bounding boxes, in the ﬁrst frame can be seen as a form of semi-supervised learning, SAVi is still solely trained with an unsupervised ﬂow prediction (or reconstruction) objective.
Revised: Comparison to semi-supervised training with matching While our approach of conditioning the model on a set of hints, such as bounding boxes, in the ﬁrst frame can be seen as a form of semi-supervised learning, SAVi is still solely trained with an unsupervised ﬂow prediction (or reconstruction) objective.

Original: Instead of providing bounding boxes as conditional input to initialize the slots of SAVi, an interesting variant is to instead initialize slots in an unconditioned way and provide the bounding boxes instead as supervision signal in the form of a matching-based loss during training.
Revised: Instead of providing bounding boxes as conditional input to initialize the slots of SAVi, an interesting variant is to instead initialize slots in an unconditioned way and provide the bounding boxes instead as supervision signal in the form of a matching-based loss during training.

Original: This variant receives the same information as our conditional setup, but now requires matching to associate slots to bounding box labels.
Revised: This variant receives the same information as our conditional setup, but now requires matching to associate slots to bounding box labels.

Original: We train the model in the same way as the supervised set prediction set up of Slot Attention using Hungarian matching and a Huber loss as described by
Revised: We train the model in the same way as the supervised set prediction set up of Slot Attention using Hungarian matching and a Huber loss as described by

Original: Locatello et al.
Revised: Locatello et al.

Original: (2020) with labels provided only during the ﬁrst frame of the video.
Revised: (2020) with labels provided only during the ﬁrst frame of the video.

Original: Additionally we still decode optical ﬂow predictions at every time step using the same loss as in our default model.
Revised: Additionally we still decode optical ﬂow predictions at every time step using the same loss as in our default model.

Original: We found that this setup requires more than a single iteration of Slot Attention to converge to a sensible decomposition in the ﬁrst frame where we apply matching.
Revised: We found that this setup requires more than a single iteration of Slot Attention to converge to a sensible decomposition in the ﬁrst frame where we apply matching.

Original: Results are summarized in Table A.4.
Revised: Results are summarized in Table A.4.

Original: We ﬁnd that using matching as training signal results in signiﬁcantly worse FG-ARI segmentation score at evaluation time compared to using conditioning both at training and at test time.
Revised: We ﬁnd that using matching as training signal results in signiﬁcantly worse FG-ARI segmentation score at evaluation time compared to using conditioning both at training and at test time.

Original: Semantic property readout To evaluate to what degree slots of the SAVi model capture information about objects beyond their location and velocity, we train a simple readout probe in the form of an MLP that is tasked to predict object properties given a learned slot representation in SAVi.
Revised: Semantic property readout To evaluate to what degree slots of the SAVi model capture information about objects beyond their location and velocity, we train a simple readout probe in the form of an MLP that is tasked to predict object properties given a learned slot representation in SAVi.

Original: For these experiments, we train SAVi with bounding box conditioning on MOVi and jointly train an MLP readout head to predict object properties from each slot representation in the ﬁrst frame.
Revised: For these experiments, we train SAVi with bounding box conditioning on MOVi and jointly train an MLP readout head to predict object properties from each slot representation in the ﬁrst frame.

Original: We do not propagate the gradients from the readout head through the rest of the model, i.e. the SAVi model does not receive any supervision information about object properties.
Revised: We do not propagate the gradients from the readout head through the rest of the model, i.e. the SAVi model does not receive any supervision information about object properties.

Original: The MLP readout head has a single hidden layer with 256 units and a ReLU activation.
Revised: The MLP readout head has a single hidden layer with 256 units and a ReLU activation.

Original: It is tasked to predict classiﬁcation logits for (i) the object color out of 8 possible colors, (ii) the object shape out of 3 possible shapes, and (iii) the object material out of 2 possible materials.
Revised: It is tasked to predict classiﬁcation logits for (i) the object color out of 8 possible colors, (ii) the object shape out of 3 possible shapes, and (iii) the object material out of 2 possible materials.

Original: For each of the three classiﬁcation targets we train with a cross-entropy loss using all videos of the MOVi training set.
Revised: For each of the three classiﬁcation targets we train with a cross-entropy loss using all videos of the MOVi training set.

Original: For evaluation, we measure average accuracy where each classiﬁcation target (color, shape, material) is considered separately and the result is averaged over videos and classiﬁcation targets.
Revised: For evaluation, we measure average accuracy where each classiﬁcation target (color, shape, material) is considered separately and the result is averaged over videos and classiﬁcation targets.

Original: In this setting, the readout probe achieves an accuracy of 54 .
Revised: In this setting, the readout probe achieves an accuracy of 54 .

Original: 6 ± 1 . 5 (10 seeds, mean and standard error) on the MOVi evaluation set which is far above chance level, indicating that semantic information[Table] init
Revised: 6 ± 1 . 5 (10 seeds, mean and standard error) on the MOVi evaluation set which is far above chance level, indicating that semantic information

Original: [Table]
Revised: [Table]

Original: about objects is still to some degree retained in the slots even though our training objective (optical ﬂow) is insensitive to color and material appearance.
Revised: about objects is still to some degree retained in the slots even though our training objective (optical ﬂow) is insensitive to color and material appearance.

Original: When training with RGB frame reconstruction instead, we can achieve a signiﬁcantly higher score of 85 .
Revised: When training with RGB frame reconstruction instead, we can achieve a signiﬁcantly higher score of 85 .

Original: 7 ± 0 .
Revised: 7 ± 0 .

Original: 7 , suggesting that one could likely combine the beneﬁts of frame reconstruction and ﬂow prediction by training a model to jointly predict both targets, which is a promising direction for future work.
Revised: 7 , suggesting that one could likely combine the beneﬁts of frame reconstruction and ﬂow prediction by training a model to jointly predict both targets, which is a promising direction for future work.

Original: A. D ATASET DETAILS
Revised: A. D ATASET DETAILS

Original: The Kubric (Greff et al., 2021) dataset generation pipeline is publicly available under an Apache 2. license.
Revised: The Kubric (Greff et al., 2021) dataset generation pipeline is publicly available under an Apache 2. license.

Original: MOVi++ contains approx.
Revised: MOVi++ contains approx.

Original: 380 publicly available CC-0 licensed HDR backgrounds from https://hdrihaven.com/ .
Revised: 380 publicly available CC-0 licensed HDR backgrounds from https://hdrihaven.com/ .

Original: The data does not contain personally identiﬁable information or offensive content.
Revised: The data does not contain personally identiﬁable information or offensive content.

Original: The original CATER (Girdhar & Ramanan, 2019) dataset (without segmentation mask annotations) is publicly available under an Apache 2.0 license.
Revised: The original CATER (Girdhar & Ramanan, 2019) dataset (without segmentation mask annotations) is publicly available under an Apache 2.0 license.

Original: The variant with segmentation masks was provided by the authors of SIMONe (Kabra et al., 2021).
Revised: The variant with segmentation masks was provided by the authors of SIMONe (Kabra et al., 2021).

Original: A. A RCHITECTURE DETAILS AND HYPERPARAMETERS
Revised: A. A RCHITECTURE DETAILS AND HYPERPARAMETERS

Original: All modules use shared parameters across time steps, except for the initializer, which is only applied at the ﬁrst time step.
Revised: All modules use shared parameters across time steps, except for the initializer, which is only applied at the ﬁrst time step.

Original: Encoder Our encoder architecture is inspired by the one used in Slot Attention (Locatello et al., 2020).
Revised: Encoder Our encoder architecture is inspired by the one used in Slot Attention (Locatello et al., 2020).

Original: We summarize the architecture in Table A.5.
Revised: We summarize the architecture in Table A.5.

Original: We use the same linear position embedding as in Slot Attention, i.e. a four-dimensional vector per spatial position encoding the coordinate along the four cardinal directions, normalized to [ − 1 , 1] .
Revised: We use the same linear position embedding as in Slot Attention, i.e. a four-dimensional vector per spatial position encoding the coordinate along the four cardinal directions, normalized to [ − 1 , 1] .

Original: We project it to the same size as the feature maps of the CNN encoder using a learnable linear transformation.
Revised: We project it to the same size as the feature maps of the CNN encoder using a learnable linear transformation.

Original: The projected position embedding is then added to the feature maps.
Revised: The projected position embedding is then added to the feature maps.

Original: For the modiﬁed SAVi (ResNet) model on MOVi++, we replace the convolutional backbone (the ﬁrst 4 convolutional layers) with a ResNet-34 (He et al., 2016) backbone.
Revised: For the modiﬁed SAVi (ResNet) model on MOVi++, we replace the convolutional backbone (the ﬁrst 4 convolutional layers) with a ResNet-34 (He et al., 2016) backbone.

Original: We use a modiﬁed ResNet root block without strides (i.e. 1 × 1 stride), resulting in 16 × 16 feature maps after the backbone.
Revised: We use a modiﬁed ResNet root block without strides (i.e. 1 × 1 stride), resulting in 16 × 16 feature maps after the backbone.

Original: We further use group normalization (Wu & He, 2018) throughout the
Revised: We further use group normalization (Wu & He, 2018) throughout the

Original: ResNet backbone.
Revised: ResNet backbone.

Original: [Table]
Revised: [Table]

Original: Corrector For the Slot Attention (Locatello et al., 2020) corrector, we only use a single iteration of the attention mechanism in all experiments on MOVi and MOVi++, and two iterations on CATER experiments.
Revised: Corrector For the Slot Attention (Locatello et al., 2020) corrector, we only use a single iteration of the attention mechanism in all experiments on MOVi and MOVi++, and two iterations on CATER experiments.

Original: The query/key/value projection size is D = 128 .
Revised: The query/key/value projection size is D = 128 .

Original: Before each projection, we apply
Revised: Before each projection, we apply

Original: Layer Norm (Ba et al., 2016).
Revised: Layer Norm (Ba et al., 2016).

Original: The attention update is followed by a GRU (Cho et al., 2014) that operates on the slot feature size of 128 (unless otherwise mentioned).
Revised: The attention update is followed by a GRU (Cho et al., 2014) that operates on the slot feature size of 128 (unless otherwise mentioned).

Original: Locatello et al.
Revised: Locatello et al.

Original: (2020) further describe placing an optional feedforward layer after the GRU, which we only use when using more than a single iteration of Slot Attention: this MLP has a single hidden layer of size 256 .
Revised: (2020) further describe placing an optional feedforward layer after the GRU, which we only use when using more than a single iteration of Slot Attention: this MLP has a single hidden layer of size 256 .

Original: Predictor We use the default multi-head dot-product attention mechanism ( MultiHeadSelfAttn ) from Vaswani et al.
Revised: Predictor We use the default multi-head dot-product attention mechanism ( MultiHeadSelfAttn ) from Vaswani et al.

Original: (2017) as part of our predictor, for which we use a query/key/value projection size of 128 and a total of 4 attention heads.
Revised: (2017) as part of our predictor, for which we use a query/key/value projection size of 128 and a total of 4 attention heads.

Original: For the MLP, we use a single hidden layer withhidden units and ReLU activation.
Revised: For the MLP, we use a single hidden layer withhidden units and ReLU activation.

Original: On CATER, we use the identity function as predictor, as objects do not interact in this dataset.
Revised: On CATER, we use the identity function as predictor, as objects do not interact in this dataset.

Original: Decoder We use the same decoder architecture as in Slot Attention (Locatello et al., 2020), for which we specify the details in Table A.6.
Revised: Decoder We use the same decoder architecture as in Slot Attention (Locatello et al., 2020), for which we specify the details in Table A.6.

Original: Each slot representation is ﬁrst spatially broadcasted to an 8 × 8 grid (i.e., the same vector is repeated at each spatial location) and then augmented with position embeddings which are computed in the same way as in the encoder.
Revised: Each slot representation is ﬁrst spatially broadcasted to an 8 × 8 grid (i.e., the same vector is repeated at each spatial location) and then augmented with position embeddings which are computed in the same way as in the encoder.

Original: We use a small CNN to arrive at representations with the same resolution as the input frame.
Revised: We use a small CNN to arrive at representations with the same resolution as the input frame.

Original: Finally, the output of this
Revised: Finally, the output of this

Original: CNN is processed by two projection heads in the form of 1 × 1 convolutions, (1) to read out the segmentation mask logits and (2) to read out the per-slot predicted optical ﬂow (or reconstructed RGB image).
Revised: CNN is processed by two projection heads in the form of 1 × 1 convolutions, (1) to read out the segmentation mask logits and (2) to read out the per-slot predicted optical ﬂow (or reconstructed RGB image).

Original: The segmentation mask logits are normalized using a softmax (across slots) and then used to recombine the per-slot predictions into a single image.
Revised: The segmentation mask logits are normalized using a softmax (across slots) and then used to recombine the per-slot predictions into a single image.

Original: The decoder uses shared parameters across slots and time steps.
Revised: The decoder uses shared parameters across slots and time steps.

Original: Note that the ground-truth optical ﬂow is computed w.r.t.
Revised: Note that the ground-truth optical ﬂow is computed w.r.t.

Original: the previous frame, using an additional simulated frame before the ﬁrst time step (which is only used for computation of ground-truth optical ﬂow and later discarded).
Revised: the previous frame, using an additional simulated frame before the ﬁrst time step (which is only used for computation of ground-truth optical ﬂow and later discarded).

Original: [Table]
Revised: [Table]

Original: Initializer To encode bounding boxes and center of mass coordinates we pass the coordinate vector for each bounding box or center of mass location through an MLP with a single hidden layer ofhidden units, a ReLU activation and an output size of 128 .
Revised: Initializer To encode bounding boxes and center of mass coordinates we pass the coordinate vector for each bounding box or center of mass location through an MLP with a single hidden layer ofhidden units, a ReLU activation and an output size of 128 .

Original: The MLP is applied with shared parameters on each bounding box (or center of mass location) independently.
Revised: The MLP is applied with shared parameters on each bounding box (or center of mass location) independently.

Original: Bounding boxes are represented as [ y min , x min , y max , x max ] coordinates and center of mass locations are represented as [ y, x ] .
Revised: Bounding boxes are represented as [ y min , x min , y max , x max ] coordinates and center of mass locations are represented as [ y, x ] .

Original: Any unconditioned slot is provided with [ − 1 , − 1 , − , 1 − 1] bounding box coordinates ( [ − 1 , − 1] for center of mass conditioning).
Revised: Any unconditioned slot is provided with [ − 1 , − 1 , − , 1 − 1] bounding box coordinates ( [ − 1 , − 1] for center of mass conditioning).

Original: When conditioning on segmentation masks, we use a CNN encoder applied independently on the binary segmentation mask for each conditioned foreground object.
Revised: When conditioning on segmentation masks, we use a CNN encoder applied independently on the binary segmentation mask for each conditioned foreground object.

Original: For each unconditioned slot, we provide an empty mask ﬁlled with zeros as input to the CNN.
Revised: For each unconditioned slot, we provide an empty mask ﬁlled with zeros as input to the CNN.

Original: The architecture is summarized in
Revised: The architecture is summarized in

Original: Table A.7.
Revised: Table A.7.

Original: For the semantic conditioning experiment, we embed the categorical values describing color, shape and material of the object using 42-dimensional learnable embedding vectors.

Original: The embedding vectors for color, shape and material are concatenated and form the initial slot representation.

Original: For unconditioned slots, we assign a separate 126-dimensional learnable embedding vector as initial slot representation.

Original: In total, there are 8 possible colors, 3 possible shapes (cylinder, sphere, cube), and two materials (metallic vs. rubber).

Original: Other hyperparameters As described in the main paper, we train for 100k steps with a batch size of 64 using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2 · 10 − 4 and gradient clipping with a maximum norm of 0.05.
Revised: Other hyperparameters As described in the main paper, we train for 100k steps with a batch size of 64 using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2 · 10 − 4 and gradient clipping with a maximum norm of 0.05.

Original: Like in previous work (Locatello et al., 2020), we use learning rate warmup and learning rate decay.
Revised: Like in previous work (Locatello et al., 2020), we use learning rate warmup and learning rate decay.

Original: We linearly warm up the learning rate for 2.5k steps and we use cosine annealing (Loshchilov & Hutter, 2017) to decay the learning rate to 0 throughout the course of training.
Revised: We linearly warm up the learning rate for 2.5k steps and we use cosine annealing (Loshchilov & Hutter, 2017) to decay the learning rate to 0 throughout the course of training.

Original: Like prior work (Locatello et al., 2020), we use a total ofslots in SAVi for our scenes which contain a maximum number of 10 objects.
Revised: Like prior work (Locatello et al., 2020), we use a total ofslots in SAVi for our scenes which contain a maximum number of 10 objects.

Original: In the conditional setup, all unconditioned slots share the same initialization and hence effectively behave as one single slot, irrespective of their number.
Revised: In the conditional setup, all unconditioned slots share the same initialization and hence effectively behave as one single slot, irrespective of their number.

Original: Our chosen hyperparamters are close to the ones used in prior[Table] work and we did not perform an extensive search.
Revised: Our chosen hyperparamters are close to the ones used in prior work and we did not perform an extensive search.

Original: We chose learning rate and gradient clipping norm based on performance measured on a separately generated instance of the MOVi++ dataset.
Revised: We chose learning rate and gradient clipping norm based on performance measured on a separately generated instance of the MOVi++ dataset.

Original: We found that training with a substantially longer schedule (e [Table] g. 1M steps) can still signiﬁcantly improve results for SAVi
Revised: We found that training with a substantially longer schedule (e.g. 1M steps) can still signiﬁcantly improve results for SAVi, but we opted for shorter schedules to allow for easier and faster reproduction of our experiments.

Revised: We only train the modiﬁed SAVi (ResNet) model on MOVi++ with a longer schedule of 1M steps, which we found to be especially beneﬁcial for this signiﬁcantly larger model.

Original: [Table]
Revised: [Table]

Original: but we opted for shorter schedules to allow for easier and faster reproduction of our experiments.

Original: We only train the modiﬁed SAVi (ResNet) model on MOVi++ with a longer schedule of 1M steps, which we found to be especially beneﬁcial for this signiﬁcantly larger model.

Original: Training software and hardware
Revised: Training software and hardware

Original: We implement both SAVi and the T-VOS baseline in JAX (Bradbury et al., 2018) using the Flax (Heek et al., 2020) neural network library.
Revised: We implement both SAVi and the T-VOS baseline in JAX (Bradbury et al., 2018) using the Flax (Heek et al., 2020) neural network library.

Original: We train our models on
Revised: We train our models on

Original: TPU v3 hardware.
Revised: TPU v3 hardware.

Original: We report training time on GPU hardware in Section 4.
Revised: We report training time on GPU hardware in Section 4.

Original: A. B ASELINE DETAILS
Revised: A. B ASELINE DETAILS

Original: SCALOR This baseline (Jiang et al., 2020) patches the image into a d × d grid and uses a discovery network to propose objects in each grid cell.
Revised: SCALOR This baseline (Jiang et al., 2020) patches the image into a d × d grid and uses a discovery network to propose objects in each grid cell.

Original: Then based on a threshold τ on the overlap of the proposed object with the propagated objects from the previous frames, the model rejects or accepts the proposals.
Revised: Then based on a threshold τ on the overlap of the proposed object with the propagated objects from the previous frames, the model rejects or accepts the proposals.

Original: The image is reconstructed with a combination of discovered objects and a background decoder with an encoding bottleneck dimension of D bg-en .
Revised: The image is reconstructed with a combination of discovered objects and a background decoder with an encoding bottleneck dimension of D bg-en .

Original: It is also important to have a correct mean and variance of the scale and ratio for the proposed objects.
Revised: It is also important to have a correct mean and variance of the scale and ratio for the proposed objects.

Original: We performed a hyper-parameter search over the mentioned parameters along with the priors and annealing schedules for discovery and for the proposal rejection.
Revised: We performed a hyper-parameter search over the mentioned parameters along with the priors and annealing schedules for discovery and for the proposal rejection.

Original: We found the foreground-background splitting behavior to be especially sensitive to hyper-parameters, frequently resulting in the reconstruction of the entire image including all (or some) foreground objects with the background decoder, as opposed to the desired case where the background decoder only reconstructs the background of the scene.
Revised: We found the foreground-background splitting behavior to be especially sensitive to hyper-parameters, frequently resulting in the reconstruction of the entire image including all (or some) foreground objects with the background decoder, as opposed to the desired case where the background decoder only reconstructs the background of the scene.

Original: We also found that changing
Revised: We also found that changing

Original: SCALOR’s prior annealing schedule made the model highly unstable.
Revised: SCALOR’s prior annealing schedule made the model highly unstable.

Original: We performed validation on held out 1 / 10 th of the training dataset, choosing the variants with best log likelihood and visual quality.
Revised: We performed validation on held out 1 / 10 th of the training dataset, choosing the variants with best log likelihood and visual quality.

Original: All the hyper-parameters for the reported results are similar to the original SCALOR, except we use a 4 × 4 grid, a rejection threshold of 0 . 2 , glimpse size of 32 , scale mean and variance 0 .
Revised: All the hyper-parameters for the reported results are similar to the original SCALOR, except we use a 4 × 4 grid, a rejection threshold of 0 . 2 , glimpse size of 32 , scale mean and variance 0 .

Original: 2 and 0 . 1 , object height-width ratio mean and variance 1 . 0 and 0 .
Revised: 2 and 0 . 1 , object height-width ratio mean and variance 1 . 0 and 0 .

Original: 5 , and for MOVi we decrease the background encoding dimension from the default 10 to 2 .
Revised: 5 , and for MOVi we decrease the background encoding dimension from the default 10 to 2 .

Original: We use the ofﬁcial implementation provided by the authors, Jiang et al.
Revised: We use the ofﬁcial implementation provided by the authors, Jiang et al.

Original: SIMONe SIMONe (Kabra et al., 2021) is a probabilistic video model which encodes frames using
Revised: SIMONe SIMONe (Kabra et al., 2021) is a probabilistic video model which encodes frames using

Original: CNNs followed by a transformer (Vaswani et al., 2017) encoder.
Revised: CNNs followed by a transformer (Vaswani et al., 2017) encoder.

Original: It subsequently produces per-frame embeddings and per-object embeddings by pooling latent representations over (per-frame) transformer tokens and over time, respectively.
Revised: It subsequently produces per-frame embeddings and per-object embeddings by pooling latent representations over (per-frame) transformer tokens and over time, respectively.

Original: Each pixel (for each time step and for each object embedding) is decoded independently using an MLP, conditioned on both time and object embedding (sampled from a learned posterior).
Revised: Each pixel (for each time step and for each object embedding) is decoded independently using an MLP, conditioned on both time and object embedding (sampled from a learned posterior).

Original: The model is trained using a reconstruction loss in pixel space (in the form of a mixture model likelihood) and a regularization loss on the latent variables.
Revised: The model is trained using a reconstruction loss in pixel space (in the form of a mixture model likelihood) and a regularization loss on the latent variables.

Original: SIMONe encodes all frames in a video in parallel, i.e. it differs from all other considered video baselines in that it is not auto-regressive and thus cannot generalize beyond the (ﬁxed) clip length provided during training.
Revised: SIMONe encodes all frames in a video in parallel, i.e. it differs from all other considered video baselines in that it is not auto-regressive and thus cannot generalize beyond the (ﬁxed) clip length provided during training.

Original: We re-implement the SIMONe model in JAX (as there is no public implementation by the authors at the time of writing) with several simpliﬁcations: 1) we sample latent variables only once per latent variable and time step instead of re-sampling for every generated pixel, 2) we always decode all frames during training instead of sampling random sub-sets of frames, 3) we only train for 200k steps as we found that performance did not signiﬁcantly improve with a longer schedule.
Revised: We re-implement the SIMONe model in JAX (as there is no public implementation by the authors at the time of writing) with several simpliﬁcations: 1) we sample latent variables only once per latent variable and time step instead of re-sampling for every generated pixel, 2) we always decode all frames during training instead of sampling random sub-sets of frames, 3) we only train for 200k steps as we found that performance did not signiﬁcantly improve with a longer schedule.

Original: We veriﬁed that our re-implementation reaches comparable performance to the numbers reported by Kabra et al.
Revised: We veriﬁed that our re-implementation reaches comparable performance to the numbers reported by Kabra et al.

Original: (2021) on CATER.
Revised: (2021) on CATER.

Original: We otherwise utilize the same hyperparameters as reported by Kabra et al.
Revised: We otherwise utilize the same hyperparameters as reported by Kabra et al.

Original: (2021) for CATER in all our experiments, and we re-size all video frames to 64 × 64 before providing them to the model.
Revised: (2021) for CATER in all our experiments, and we re-size all video frames to 64 × 64 before providing them to the model.

Original: Contrastive Random Walk (CRW) CRW (Jabri et al., 2020) is a self-supervised technique that induces a contrastive loss between embeddings of patches in neighboring frames.
Revised: Contrastive Random Walk (CRW) CRW (Jabri et al., 2020) is a self-supervised technique that induces a contrastive loss between embeddings of patches in neighboring frames.

Original: These embeddings are effective for downstream tracking via label propagation.
Revised: These embeddings are effective for downstream tracking via label propagation.

Original: We used the open source implementation provided by the authors to train their model on our datasets, MOVi and MOVi++; and evaluated tracking performance using the Davis 2017 benchmark utilities (Caelles et al., 2019; Pont-Tuset et al., 2017) on the validation sets of MOVi and MOVi++ respectively.
Revised: We used the open source implementation provided by the authors to train their model on our datasets, MOVi and MOVi++; and evaluated tracking performance using the Davis 2017 benchmark utilities (Caelles et al., 2019; Pont-Tuset et al., 2017) on the validation sets of MOVi and MOVi++ respectively.

Original: We resized all images to 256 ×for training and to 480 × 480 for evaluation.
Revised: We resized all images to 256 ×for training and to 480 × 480 for evaluation.

Original: This matches the setup that CRW was originally trained and tested on (Kinetics at 256 × 256 and Davis at 480p).
Revised: This matches the setup that CRW was originally trained and tested on (Kinetics at 256 × 256 and Davis at 480p).

Original: Label propagation is more accurate at high resolution.
Revised: Label propagation is more accurate at high resolution.

Original: We tuned and set the following parameters for MOVi++:
Revised: We tuned and set the following parameters for MOVi++:

Original: Edge-dropout rate 0.05, training temperature 0.01, evaluation temperature 0.05, frame sampling gap 1, number of training epochs 250 with a learning rate drop at 200 epochs.
Revised: Edge-dropout rate 0.05, training temperature 0.01, evaluation temperature 0.05, frame sampling gap 1, number of training epochs 250 with a learning rate drop at 200 epochs.

Original: And for MOVi: Edge-dropout rate 0.05, training temperature 0.03, evaluation temperature 0.01, frame sampling gap 2, number of training epochswith a learning rate drop at 30 epochs.
Revised: And for MOVi: Edge-dropout rate 0.05, training temperature 0.03, evaluation temperature 0.01, frame sampling gap 2, number of training epochswith a learning rate drop at 30 epochs.

Original: Early stopping was necessary for MOVi as training longer hurt downstream performance.
Revised: Early stopping was necessary for MOVi as training longer hurt downstream performance.

Original: We used video clips consisting of 6 frames to train these models.
Revised: We used video clips consisting of 6 frames to train these models.

Original: All other hyper-parameters and the architecture, a ResNet-18 (He et al., 2016) encoder, were kept at their default values.
Revised: All other hyper-parameters and the architecture, a ResNet-18 (He et al., 2016) encoder, were kept at their default values.

Original: Transductive VOS Transductive VOS (T-VOS) (Zhang et al., 2020) propagates segmentation labels from the ﬁrst frame into future frames, thus tracking objects and segmenting them.
Revised: Transductive VOS Transductive VOS (T-VOS) (Zhang et al., 2020) propagates segmentation labels from the ﬁrst frame into future frames, thus tracking objects and segmenting them.

Original: It correlates per-pixel features of a target frame with those of a history of previous frames, called reference.
Revised: It correlates per-pixel features of a target frame with those of a history of previous frames, called reference.

Original: Softmax-normalized correlation weights are used to compute a convex combination of previous frame labels.
Revised: Softmax-normalized correlation weights are used to compute a convex combination of previous frame labels.

Original: This constitutes the prediction for each pixel in the target frame.
Revised: This constitutes the prediction for each pixel in the target frame.

Original: Such predictions in turn get incorporated into the reference for subsequent label propagation.
Revised: Such predictions in turn get incorporated into the reference for subsequent label propagation.

Original: A simple motion prior is used to focus on neighboring regions when propagating labels.
Revised: A simple motion prior is used to focus on neighboring regions when propagating labels.

Original: T-VOS uses references frames to capture the past video context.
Revised: T-VOS uses references frames to capture the past video context.

Original: For T-VOS, we use the ofﬁcial implementation by the authors, using an ImageNet pre-trained ResNet-

Revised: For T-VOS, we use the ofﬁcial implementation by the authors, using an ImageNet pre-trained ResNet-

Original: (He et al., 2016) backbone excluding the last stage for better results.
Revised: (He et al., 2016) backbone excluding the last stage for better results.

Original: We upsample MOVi/MOVi++ frames to 480p resolution, as we found that T-VOS with a ResNetbackbone produces inferior results when applied on frames at the original resolution.
Revised: We upsample MOVi/MOVi++ frames to 480p resolution, as we found that T-VOS with a ResNetbackbone produces inferior results when applied on frames at the original resolution.

Original: Metrics were computed at the original resolution of the two datasets.
Revised: Metrics were computed at the original resolution of the two datasets.

Original: We select the optimal T-VOS temperature hyperparameter ( τ = 1 for MOVi and τ = 1 . 3 for MOVi++) based on evaluation performance.
Revised: We select the optimal T-VOS temperature hyperparameter ( τ = 1 for MOVi and τ = 1 . 3 for MOVi++) based on evaluation performance.

Original:  For a more direct comparison to SAVi, we apply T-VOS (Zhang et al., 2020) on a backbone that is pre-trained using the same data and optical
Revised: Segmentation Propagation For a more direct comparison to SAVi, we apply T-VOS (Zhang et al., 2020) on a backbone that is pre-trained using the same data and optical

Original: ﬂow supervision as SAVi.
Revised: ﬂow supervision as SAVi.

Original: For this experiment, we use a simpliﬁed re-implementation of T-VOS in JAX, for which we use a maximum of 9 reference frames of the immediate past.
Revised: For this experiment, we use a simpliﬁed re-implementation of T-VOS in JAX, for which we use a maximum of 9 reference frames of the immediate past.

Original: We do not make use of the sparse sampling method from (Zhang et al., 2020).
Revised: We do not make use of the sparse sampling method from (Zhang et al., 2020).

Original: This is similar to the “9 frames” column in Table 2 as reported by
Revised: This is similar to the “9 frames” column in Table 2 as reported by

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020), but
Revised: (2020), but

Original: in addition also uses their motion prior.
Revised: in addition also uses their motion prior.

Original: We trained a self-supervised CNN backbone, in domain, using MOVi/MOVi++ frames.
Revised: We trained a self-supervised CNN backbone, in domain, using MOVi/MOVi++ frames.

Original: The self-supervision proxy task was that of single image optical ﬂow prediction, inspired by prior work on predicting optical ﬂow from static images (Mahendran et al., 2019; Walker et al., 2015).
Revised: The self-supervision proxy task was that of single image optical ﬂow prediction, inspired by prior work on predicting optical ﬂow from static images (Mahendran et al., 2019; Walker et al., 2015).

Original: This proxy task requires guessing object motion based on appearance, which in turn requires modeling the objects present in the scene.
Revised: This proxy task requires guessing object motion based on appearance, which in turn requires modeling the objects present in the scene.

Original: In more detail, we regressed optical ﬂow using a fully convolutional network (Long et al., 2015) using the exact same loss function as SAVi.
Revised: In more detail, we regressed optical ﬂow using a fully convolutional network (Long et al., 2015) using the exact same loss function as SAVi.

Original: The image backbone was also identical to SAVi except for an extra 5 × 5 convolution layer to project features into 128 dimensions.
Revised: The image backbone was also identical to SAVi except for an extra 5 × 5 convolution layer to project features into 128 dimensions.

Original: This matches the dimensionality of slot vectors in SAVi making this a fairer comparison.
Revised: This matches the dimensionality of slot vectors in SAVi making this a fairer comparison.

Original: To pre-train the visual backbone, we use a batch size of 64 and we train for 100k steps using the
Revised: To pre-train the visual backbone, we use a batch size of 64 and we train for 100k steps using the

Original: Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2 × 10 − 5 .
Revised: Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2 × 10 − 5 .

Original: We use linear position embeddings in the same way as SAVi, adding them right before a 1 × 1 convolution layer which regresses 3-dimensional RGB encoded optical ﬂow.
Revised: We use linear position embeddings in the same way as SAVi, adding them right before a 1 × 1 convolution layer which regresses 3-dimensional RGB encoded optical ﬂow.

Original: For evaluation of T-VOS, we use their motion prior with σ 1 = 2 . 3 , σ 2 = 4 . 6 , and a temperature of 50 for MOVi and 200 for MOVi++.
Revised: For evaluation of T-VOS, we use their motion prior with σ 1 = 2 . 3 , σ 2 = 4 . 6 , and a temperature of 50 for MOVi and 200 for MOVi++.

Original: We use
Revised: We use

Original: L 2 -normalized per-pixel embeddings in order to make T-VOS less sensitive to temperature.
Revised: L 2 -normalized per-pixel embeddings in order to make T-VOS less sensitive to temperature.

Original: A. M ETRIC DETAILS
Revised: A. M ETRIC DETAILS

Original: For both FG-ARI and mIoU, we skip the ﬁrst frame during evaluation as we provide conditional information (e.g., bounding boxes) for it in most cases.
Revised: For both FG-ARI and mIoU, we skip the ﬁrst frame during evaluation as we provide conditional information (e.g., bounding boxes) for it in most cases.

Original: FG-ARI For our experiments on CATER (unsupervised video decomposition)
Revised: FG-ARI For our experiments on CATER (unsupervised video decomposition)

Original: we follow prior work (Kabra et al., 2021) and report FG-ARI for all frames.
Revised: we follow prior work (Kabra et al., 2021) and report FG-ARI for all frames.

Original: Segmentation mIoU
Revised: Segmentation mIoU

Original: This metric assumes a strict alignment between ground truth and predictions.
Revised: This metric assumes a strict alignment between ground truth and predictions.

Original: That is, if the ﬁrst segment tracks a ball then the ﬁrst slot should track the same ball.
Revised: That is, if the ﬁrst segment tracks a ball then the ﬁrst slot should track the same ball.

Original: No matching is used.
Revised: No matching is used.

Original: Unlike ARI, this metric is sensitive to slots tracking the object they were conditioned with.
Revised: Unlike ARI, this metric is sensitive to slots tracking the object they were conditioned with.

Original: In frames were an object is missing, either due to occlusion or because it has left the scene, mIoU = 1 .if
Revised: In frames were an object is missing, either due to occlusion or because it has left the scene, mIoU = 1 .if

Original: the object is missing in the prediction as well, otherwise mIoU = 0 . 0 .
Revised: the object is missing in the prediction as well, otherwise mIoU = 0 . 0 .

Original: Thus even a single predicted pixel in such frames is penalized heavily.
Revised: Thus even a single predicted pixel in such frames is penalized heavily.

Original: This setting corresponds to the Jaccard-Mean metric used in the DAVIS-2017 object tracking challenge (Caelles et al., 2019; Pont-Tuset et al., 2017).
Revised: This setting corresponds to the Jaccard-Mean metric used in the DAVIS-2017 object tracking challenge (Caelles et al., 2019; Pont-Tuset et al., 2017).