Original: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
Revised: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Unsupervised object-centric learning aims to represent the modular, compositional,and causal structure of a scene as a set of object representations and therebypromises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.
Revised: Unsupervised object-centric learning aims to represent the modular, compositional, and causal structure of a scene as a set of object representations and thereby promises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.

Original: Although there have been manyremarkable advances in recent years, one of the most critical problems in thisdirection has been that previous methods work only with simple and syntheticscenes but not with complex and naturalistic images or videos.
Revised: Although there have been many remarkable advances in recent years, one of the most critical problems in this direction has been that previous methods work only with simple and synthetic scenes but not with complex and naturalistic images or videos.

Original: In this paper,we propose STEVE, an unsupervised model for object-centric learning in videos.
Revised: In this paper, we propose STEVE, an unsupervised model for object-centric learning in videos.

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line ofresearch.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line of research.

Original: Interestingly, this is achieved by neither adding complexity to the modelarchitecture nor introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding complexity to the model architecture nor introducing a new objective or weak supervision.

Original: Rather, it isachieved by a surprisingly simple architecture that uses a transformer-based imagedecoder conditioned on slots and the learning objective is simply to reconstruct theobservation.
Revised: Rather, it is achieved by a surprisingly simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation.

Original: Our experiment results on various complex and naturalistic videosshow significant improvements compared to the previous state-of-the-art. 
Revised: Our experiment results on various complex and naturalistic videos show significant improvements compared to the previous state-of-the-art. https://sites.google.com/view/slot-transformer-for-videos

Original:  Introduction
Revised:  Introduction

Original: The goal of object-centric representation learning is to represent the inherent structure of the physicalworld such as compositionality and causality as a set of representation vectors (and their relations)each corresponding to a conceptual entity module such as an object (Greff et al., 2020; Schölkopfet al., 2021).
Revised: The goal of object-centric representation learning is to represent the inherent structure of the physical world such as compositionality and causality as a set of representation vectors (and their relations) each corresponding to a conceptual entity module such as an object (Greff et al., 2020; Schölkopf et al., 2021).

Original: Many previous works have demonstrated the potential of this approach as a way ofresolving the key limitations of traditional single-vector representations (Greff et al., 2020; Schölkopfet al., 2021;
Revised: Many previous works have demonstrated the potential of this approach as a way of resolving the key limitations of traditional single-vector representations (Greff et al., 2020; Schölkopf et al., 2021;

Original: Dittadi et al., 2021; Eslami et al., 2016;
Revised: Dittadi et al., 2021; Eslami et al., 2016;

Original: Bapst et al., 2019; Mambelli et al., 2022;
Revised: Bapst et al., 2019; Mambelli et al., 2022;

Original: Ghasemipour et al., 2022).
Revised: Ghasemipour et al., 2022).

Original: For example, considering object-centric representations as independentmechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al. ,2022; Chen et al., 2021).
Revised: For example, considering object-centric representations as independent mechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al., 2022; Chen et al., 2021).

Original: Also, the ability to decompose a visual observation into a set of discreteknowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).
Revised: Also, the ability to decompose a visual observation into a set of discrete knowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).

Original: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.
Revised: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.

Original: Achieving this has primary importancebecause it opens up a way to utilize the enormous amounts of images and videos available on theinternet and thereby unleash the full potential of this method by applying it at scale — the machinelearning community has recently observed the power of scale in large-scale language models (Brownet al., 2020; Bommasani et al., 2021).
Revised: Achieving this has primary importance because it opens up a way to utilize the enormous amounts of images and videos available on the internet and thereby unleash the full potential of this method by applying it at scale — the machine learning community has recently observed the power of scale in large-scale language models (Brown et al., 2020; Bommasani et al., 2021).

Original: However, despite a significant amount of effort and many remarkable achievements in recent years,we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.
Revised: However, despite a significant amount of effort and many remarkable achievements in recent years, we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.

Original: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeedremains quite elusive.
Revised: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeed remains quite elusive.

Original: As such, previous works have been shown to work only on toy or syntheticimages/videos or resorted to utilizing some supervision such as optical flow or annotation on theinitial frame (Kipf et al., 2021).
Revised: As such, previous works have been shown to work only on toy or synthetic images/videos or resorted to utilizing some supervision such as optical flow or annotation on the initial frame (Kipf et al., 2021).

Original: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE(Slot-TransformEr for VidEos).
Revised: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE (Slot-TransformEr for VidEos).

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented inthis line of research.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented in this line of research.

Original: Interestingly, this is achieved by neither adding much complexity to the modelarchitecture nor by introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding much complexity to the model architecture nor by introducing a new objective or weak supervision.

Original: Rather, it is achieved by asurprisingly simple architecture, comparable to or simpler than most of the previous methods thathave not worked well on complex videos.
Revised: Rather, it is achieved by a surprisingly simple architecture, comparable to or simpler than most of the previous methods that have not worked well on complex videos.

Original: Moreover, the learning objective is just to reconstruct.
Revised: Moreover, the learning objective is just to reconstruct.

Original: Webelieve this simplicity is one of the best strengths of our model.
Revised: We believe this simplicity is one of the best strengths of our model.

Original: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE(Singh et al., 2022).
Revised: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE (Singh et al., 2022).

Original: Because it is demonstrated in the paper that this decoder is the key to dealingwith some visually complex synthetic images, it is of deep interest to the community (i) whetherand how it can be extended to a temporal model utilizing dynamical observations, (ii) if it caneventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.
Revised: Because it is demonstrated in the paper that this decoder is the key to dealing with some visually complex synthetic images, it is of deep interest to the community (i) whether and how it can be extended to a temporal model utilizing dynamical observations, (ii) if it can eventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.

Original: Tofocus on investigating these central questions systematically, we intentionally choose not to pursuearchitectural novelty at the expense of adding complexity but to test a minimal architecture thatcombines the SLATE decoder with a standard slot-level recurrence model.
Revised: To focus on investigating these central questions systematically, we intentionally choose not to pursue architectural novelty at the expense of adding complexity but to test a minimal architecture that combines the SLATE decoder with a standard slot-level recurrence model.

Original: Our experiment results onvarious complex and naturalistic videos show that the proposed architecture significantly outperformsprevious state-of-the-art baseline models.
Revised: Our experiment results on various complex and naturalistic videos show that the proposed architecture significantly outperforms previous state-of-the-art baseline models.

Original: From these experiments, we discover important new knowledge to share with the community.
Revised: From these experiments, we discover important new knowledge to share with the community.

Original: We findthat SLATE can deal with complex naturalistic images reasonably well despite not using temporalinformation.
Revised: We find that SLATE can deal with complex naturalistic images reasonably well despite not using temporal information.

Original: Although as a static model, SLATE has a fundamental limitation that it cannot takeadvantage of temporal information, e.g., maintaining consistent object identity across time, if we onlylook at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).
Revised: Although as a static model, SLATE has a fundamental limitation that it cannot take advantage of temporal information, e.g., maintaining consistent object identity across time, if we only look at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).

Original: Crucially, we also findthat SLATE alone is not enough since our model, STEVE, shows much better frame segmentationthan SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.
Revised: Crucially, we also find that SLATE alone is not enough since our model, STEVE, shows much better frame segmentation than SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.

Original: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly,suggesting that learning jointly from both temporal information and the powerful SLATE decoder issignificant and essential to realize the full potential of object-centric representation learning in videos.
Revised: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly, suggesting that learning jointly from both temporal information and the powerful SLATE decoder is significant and essential to realize the full potential of object-centric representation learning in videos.

Original: Furthermore, our analysis shows that our model is robust to challenges such as static objects andcamera motion and that it also generalizes well to out-of-distribution number of objects and unseentextures.
Revised: Furthermore, our analysis shows that our model is robust to challenges such as static objects and camera motion and that it also generalizes well to out-of-distribution number of objects and unseen textures.

Original: Lastly, we propose several new datasets that are significantly more complex than thosetackled by the previous unsupervised models.
Revised: Lastly, we propose several new datasets that are significantly more complex than those tackled by the previous unsupervised models.

Original:  Preliminaries
Revised:  Preliminaries

Original: In unsupervised object-centric representation learning, the goal is to learn to obtain the representationof an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).
Revised: In unsupervised object-centric representation learning, the goal is to learn to obtain the representation of an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).

Original: A commonapproach for this is to auto-encode: a slot-encoder
Revised: A common approach for this is to auto-encode: a slot-encoder

Original: f enc ϕ
Revised: f enc ϕ

Original: ( x ) extracts a set of slot representationss =
Revised: ( x ) extracts a set of slot representations s

Original:  { s 1 , . . .
Revised: = { s 1 , . . .

Original: s N } from an image x
Revised: s N } from an image x

Original: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from theseslots, where x ∈ R H × W × C and s n ∈ R D .
Revised: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from these slots, where x ∈ R H × W × C and s n ∈ R D .

Original: In the encoder, the slots compete with each other to learnabout which area of the input will be explained by which slots.
Revised: In the encoder, the slots compete with each other to learn about which area of the input will be explained by which slots.

Original: This provides a form of informationbottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity likean object.
Revised: This provides a form of information bottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity like an object.

Original: During training, a reconstruction objective provides the learning signal for training thecomplete model.
Revised: During training, a reconstruction objective provides the learning signal for training the complete model.

Original: One nice property of this method is that it only requires reconstructing the imageand does not require any weak supervision or object-aware auxiliary loss terms.
Revised: One nice property of this method is that it only requires reconstructing the image and does not require any weak supervision or object-aware auxiliary loss terms.

Original: Ideally, we want tomaintain this simplicity in video models, but current state-of-the-art requires weak supervision andoptical flow to make it work on complex naturalistic videos (Kipf et al., 2021).
Revised: Ideally, we want to maintain this simplicity in video models, but current state-of-the-art requires weak supervision and optical flow to make it work on complex naturalistic videos (Kipf et al., 2021).

Original: Mixture-based Decoder.
Revised: Mixture-based Decoder.

Original: The emergence of object-centric slots is strongly dependent on whatdecoder is used.
Revised: The emergence of object-centric slots is strongly dependent on what decoder is used.

Original: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.
Revised: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.

Original: In this approach, the decoder decodes each slot s n to obtain anobject image ˆ x n and an alpha mask m n via decoding functions g RGB θ
Revised: In this approach, the decoder decodes each slot s n to obtain an object image ˆ x n and an alpha mask m n via decoding functions g RGB θ

Original: and g mask θ , respectively.
Revised: and [Table] θ , respectively.

Original: The decoded object images are then weighted-summed to obtain the full image ˆ x :ˆ x n = g RGB θ
Revised: The decoded object images are then weighted-summed to obtain the full image ˆ x : [Equation] n = g [Equation]

Revised: θ

Original: [Equation]
Revised: [Equation]

Original: These decoder networks are implemented using a CNN.
Revised: These decoder networks are implemented using a CNN.

Original: The reconstruction objective is taken to bethe squared error between the input and the reconstructed image i.e. L mixture =
Revised: The reconstruction objective is taken to be the squared error between the input and the reconstructed image i.e. L mixture =

Original: ∥ ˆ x − x ∥ 2 .
Revised: ∥ ˆ x − x ∥ 2 .

Original: A critical93 limitation of this approach is that it has never been successful in dealing with scenes with high visualcomplexity like natural images.
Revised: A critical limitation of this approach is that it has never been successful in dealing with scenes with high visual complexity like natural images.

Original: Autoregressive Slot-Transformer Decoder.
Revised: Autoregressive Slot-Transformer Decoder.

Original: Recently, Singh et al.
Revised: Recently, Singh et al.

Original: (2022) have challenged thistraditional perspective that we need a mixture decoder for the emergence of objectness in slots.
Revised: (2022) have challenged this traditional perspective that we need a mixture decoder for the emergence of objectness in slots.

Original: Arguing that a mixture decoder severely limits the interaction among slots and the reconstructionquality—hence a problem in obtaining good learning signal for complex images—they proposed anew architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditionedon the slots (Chen et al., 2020).
Revised: Arguing that a mixture decoder severely limits the interaction among slots and the reconstruction quality—hence a problem in obtaining good learning signal for complex images—they proposed a new architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditioned on the slots (Chen et al., 2020).

Original: However, many critical questions about this new approach still remainbecause the focus of the SLATE paper is on image generation ability.
Revised: However, many critical questions about this new approach still remain because the focus of the SLATE paper is on image generation ability.

Original: To get a deeper understandingof the object representations that a model like SLATE produces, we would need to thoroughlyinvestigate the quantitative evidence about its scene decomposition ability, its ability to deal withvisual complexity close to that of natural images, and finally, the effect of extending this architectureto a temporal model.
Revised: To get a deeper understanding of the object representations that a model like SLATE produces, we would need to thoroughly investigate the quantitative evidence about its scene decomposition ability, its ability to deal with visual complexity close to that of natural images, and finally, the effect of extending this architecture to a temporal model.

Original: We aim to answer these in this paper.
Revised: We aim to answer these in this paper.

Original:  STEVE: Slot Transformer for Videos
Revised:  STEVE: Slot Transformer for Videos

Original: To study the effect of extending the slot-transformer decoder to a temporal model that can deal withvideos, we choose to propose a minimal architecture.
Revised: To study the effect of extending the slot-transformer decoder to a temporal model that can deal with videos, we choose to propose a minimal architecture.

Original: Specifically, our proposed model, STEVE,combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder thatupdates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoderof SLATE.
Revised: Specifically, our proposed model, STEVE, combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE.

Original: Although we may obtain additional performance gains by exploring further architecturalnovelty at the cost of additional complexity, we choose to investigate this minimal architecture tofocus on our goal of investigating slot-transformer decoder models for video.
Revised: Although we may obtain additional performance gains by exploring further architectural novelty at the cost of additional complexity, we choose to investigate this minimal architecture to focus on our goal of investigating slot-transformer decoder models for video.

Original: Given a video consisting of frames x 1 , . . .
Revised: Given a video consisting of frames x 1 , . . .

Original: x T , we want to maintain N slots s t = { s t, 1 , . . .
Revised: x T , we want to maintain N slots s t = { s t, 1 , . . .

Original: , s t,N }for each time-step t ∈ { 1 , . . .
Revised: , s t,N } for each time-step t ∈ { 1 , . . .

Original: T } by starting the temporal update from initial slots s 0 .
Revised: T } by starting the temporal update from initial slots s 0 .

Original: We apply arecurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t usingthe information from the input frame x t .
Revised: We apply a recurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t using the information from the input frame x t .

Original: [Equation]
Revised: [Equation]

Original: It is expected that each slot n would represent one object and track it consistently over time in a givenvideo.
Revised: It is expected that each slot n would represent one object and track it consistently over time in a given video.

Original: For each frame, the slot representation s t is used to reconstruct the input frame x t using theslot-transformer decoder.
Revised: For each frame, the slot representation s t is used to reconstruct the input frame x t using the slot-transformer decoder.

Original: For doing such reconstruction, each frame x t is treated as a sequence ofdiscrete tokens provided by a discrete VAE encoder.
Revised: For doing such reconstruction, each frame x t is treated as a sequence of discrete tokens provided by a discrete VAE encoder.

Original: Given the slots s t , the slot-transformer decoderlearns to predict this sequence of tokens auto-regressively by minimizing a cross-entropy loss.
Revised: Given the slots s t , the slot-transformer decoder learns to predict this sequence of tokens auto-regressively by minimizing a cross-entropy loss.

Original: [Equation]
Revised: [Equation]

Original: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .
Revised: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .

Original: , z t,L } are L discrete tokens that act as124 prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by thetransformer for the token at position l for frame t .
Revised: , z t,L } are L discrete tokens that act as prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by the transformer for the token at position l for frame t .

Original: To train the discrete VAE, an image reconstructionloss L dVAE is used as in SLATE.
Revised: To train the discrete VAE, an image reconstruction loss L dVAE is used as in SLATE.

Original: [Equation]
Revised: [Equation]

Revised: ˆ x t =

Original:  The complete model is trained using L STEVE =
Revised: g dVAE θ ( z t, 1 , . . . z t,L ) . The complete model is trained using L STEVE =

Original: L CE + L dVAE .
Revised: L CE + L dVAE .

Original: Recurrent Slot Encoder.
Revised: Recurrent Slot Encoder.

Original: Several recent works have implemented recurrent object-centric modelsby incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al. ,2019; Jiang et al., 2020; 
Revised: Several recent works have implemented recurrent object-centric models by incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al., 2019; Jiang et al., 2020; Kossen et al., 2020; Lin et al., 2020a;

Original: Kossen et al., 2020; Lin et al., 2020a; Stani´c & Schmidhuber, 2019).
Revised: Stani ´ c & Schmidhuber, 2019).

Original: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot
Revised: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot

Original: Attention Video (Kipf et al., 2021).
Revised: Attention Video (Kipf et al., 2021).

Original: Our slot-based recurrent encoder works as follows.
Revised: Our slot-based recurrent encoder works as follows.

Original: At the startof an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution withlearned mean and variance.
Revised: At the start of an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution with learned mean and variance.

Original: For each input frame x t , we compute an encoding in the form of afeature map using a backbone CNN.
Revised: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.

Original: This feature map is flattened to obtain a set of input featurese t = { e t, 1 , e t, 2 , . . .
Revised: This feature map is flattened to obtain a set of input features e t = { e t, 1 , e t, 2 , . . .

Original: , e t,HW } .
Revised: , e t,HW } .

Original: Next, the slots s t − 1 perform attention on the features e t and use137 the attention result to update the slots to s t .
Revised: Next, the slots s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .

Original: For this, the slots s t − 1 compute attention weights
Revised: For this, the slots s t − 1 compute attention weights

Original: A t = {A t, 1 , . . .
Revised: A t = {A t, 1 , . . .

Original: , A t,N } over the input features, where A t,n are the attention weights for slot n .
Revised: , A t,N } over the input features, where A t,n are the attention weights for slot n .

Original: The139 attention weights are used to perform an attention-weighted sum of the input features.
Revised: The attention weights are used to perform an attention-weighted sum of the input features.

Original: The attentionresult r t,n is then used to update the respective slot using an RNN f RNN ϕ 
Revised: The attention result r t,n is then used to update the respective slot using an RNN f RNN ϕ . [Equation] s t,n

Original: [Equation]
Revised: [Equation]

Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original:  Here, q , k and v are linear projections and D is the output size of the projections.
Revised: n [Equation] Here, q , k and v are linear projections and D is the output size of the projections.

Original: Finally, the slotsare made to interact via an interaction network: s t = f interact ϕ
Revised: Finally, the slots are made to interact via an interaction network: s t = f interact ϕ

Original: (˜ s t ) .
Revised: (˜ s t ) .

Original: Following Kipf et al.
Revised: Following Kipf et al.

Original: (2021), weuse the slots ˜ s t before the interaction step for reconstruction.
Revised: (2021), we use the slots ˜ s t before the interaction step for reconstruction.

Original: For complete details of the architecture,see Appendix B.
Revised: For complete details of the architecture, see Appendix B.

Original:  Related Work
Revised:  Related Work

Original: Unsupervised Object-Centric Representation Learning in Images.
Revised: Unsupervised Object-Centric Representation Learning in Images.

Original: Unsupervised object-centriclearning methods for static scenes have received significant interest in recent years (Greff et al., 2016;
Revised: Unsupervised object-centric learning methods for static scenes have received significant interest in recent years (Greff et al., 2016;

Original: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017; Engelcke et al., 2020,2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Denget al., 2020; Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al.,2020).
Revised: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017; Engelcke et al., 2020, 2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Deng et al., 2020; Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al., 2020).

Original: These models learn through reconstruction, commonly adopting a mixture-based decoder.
Revised: These models learn through reconstruction, commonly adopting a mixture-based decoder.

Original: Another line of work has pursued object discovery by minimizing the mutual information between thepredicted segments (Savarese et al., 2021;
Revised: Another line of work has pursued object discovery by minimizing the mutual information between the predicted segments (Savarese et al., 2021;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Approaches based on self-supervisedrepresentation learning have also been investigated (Caron et al., 2021; Löwe et al., 2020; Wang et al.,2022).
Revised: Approaches based on self-supervised representation learning have also been investigated (Caron et al., 2021; Löwe et al., 2020; Wang et al., 2022).

Original: Recently, scene decomposition ability has been shown to emerge in energy-based models (Duet al., 2021a; Yu et al., 2021).
Revised: Recently, scene decomposition ability has been shown to emerge in energy-based models (Du et al., 2021a; Yu et al., 2021).

Original: Complex-valued neural networks have also shown such emergence
Revised: Complex-valued neural networks have also shown such emergence

Original: Lowe et al.
Revised: Lowe et al.

Original: Closely related to our work, Lamb et al.
Revised: Closely related to our work, Lamb et al.

Original: (2021) and Singh et al.
Revised: (2021) and Singh et al.

Original: (2022) haveshown that a slot-based encoder combined with a transformer decoder can enable object emergencein visually complex images.
Revised: (2022) have shown that a slot-based encoder combined with a transformer decoder can enable object emergence in visually complex images.

Original: However, all of these approaches deal only with static images unlikeours which can be applied to videos.
Revised: However, all of these approaches deal only with static images unlike ours which can be applied to videos.

Original: Unsupervised Object-Centric Representation Learning in Videos.
Revised: Unsupervised Object-Centric Representation Learning in Videos.

Original: A large body of work hasapproached fully-unsupervised video segmentation and tracking by combining recurrent slot-basedencoders with a reconstruction objective.
Revised: A large body of work has approached fully-unsupervised video segmentation and tracking by combining recurrent slot-based encoders with a reconstruction objective.

Original: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber,2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al.
Revised: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber, 2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al., 2021;

Original: He et al., 2019) use bounding boxes for tracking.
Revised: He et al., 2019) use bounding boxes for tracking.

Original: A parallel line of research learns to localizeobjects via segmentation masks (Greff et al., 2017;
Revised: A parallel line of research learns to localize objects via segmentation masks (Greff et al., 2017;

Original: Van Steenkiste et al., 2018; Veerapaneni et al. ,2019; Watters et al., 2019; Weis et al., 2020; Du et al., 2020; Kipf et al., 2021; Kabra et al., 2021;
Revised: Van Steenkiste et al., 2018; Veerapaneni et al., 2019; Watters et al., 2019; Weis et al., 2020; Du et al., 2020; Kipf et al., 2021; Kabra et al., 2021;

Original: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).
Revised: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).

Original: Our work lies alongthis line of research.
Revised: Our work lies along this line of research.

Original: Other approaches have considered using contrastive losses (Kipf et al., 2019;
Revised: Other approaches have considered using contrastive losses (Kipf et al., 2019;

Original: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3Dscenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al.
Revised: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3D scenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al., 2021;

Original: ,2021; Du et al., 2021b; Kabra et al., 2021).
Revised: Du et al., 2021b; Kabra et al., 2021).

Original: While dealing with 3D scenes would be an importantfuture direction for our work, it is orthogonal to our current focus.
Revised: While dealing with 3D scenes would be an important future direction for our work, it is orthogonal to our current focus.

Original: However, unlike ours, all ofthe above works have only been successful on visually simple videos.
Revised: However, unlike ours, all of the above works have only been successful on visually simple videos.

Original: In Appendix D, we discussthe related work that deals with unsupervised video object segmentation using motion cues andunsupervised segmentation propagation.
Revised: In Appendix D, we discuss the related work that deals with unsupervised video object segmentation using motion cues and unsupervised segmentation propagation.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate our model on 8 datasets.
Revised: We evaluate our model on 8 datasets.

Original: These include 6 procedurally generated datasets:
Revised: These include 6 procedurally generated datasets:

Original: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E(Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.
Revised: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.

Original: Out of these, 5 datasets are our contributions in this work which we now describe.
Revised: Out of these, 5 datasets are our contributions in this work which we now describe.

Original: i)
Revised: i)

Original: CATERTex.
Revised: CATERTex.

Original: We generated it by combiningobjects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar& Ramanan, 2020).
Revised: We generated it by combining objects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar & Ramanan, 2020).

Original: ii) MOVi-Solid.
Revised: ii) MOVi-Solid.

Original: We used Kubric (Greff et al., 2022) to generate this dataset andintroduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipfet al.(2021).
Revised: We used Kubric (Greff et al., 2022) to generate this dataset and introduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipf et al.

Original: iii) MOVi-Tex.
Revised: iii) MOVi-Tex.

Original: We added the textures from the Describable Textures dataset (Cimpoiet al., 2014) as materials to the MOVi-Solid dataset to make this dataset.
Revised: We added the textures from the Describable Textures dataset (Cimpoi et al., 2014) as materials to the MOVi-Solid dataset to make this dataset.

Original: By applying the samematerial on all surfaces, the objects and their borders become significantly harder to distinguish,resulting in much higher visual complexity over MOVi-Solid.
Revised: By applying the same material on all surfaces, the objects and their borders become significantly harder to distinguish, resulting in much higher visual complexity over MOVi-Solid.

Original: iv) Traffic and Aquarium.
Revised: iv) Traffic and Aquarium.

Original: We createdthese two natural datasets by collecting a 6-hour long video stream from Youtube.
Revised: We created these two natural datasets by collecting a 6-hour long video stream from Youtube.

Original: For training,all models use only the raw videos as input with no other supervision or input whatsoever.
Revised: For training, all models use only the raw videos as input with no other supervision or input whatsoever.

Original: Forevaluation, we use the ground truth instance-level masks.
Revised: For evaluation, we use the ground truth instance-level masks.

Original: For sample video frames and more details,see Appendix A.
Revised: For sample video frames and more details, see Appendix A.

Original: We will release all the proposed datasets which, we believe, will facilitate futureresearch.
Revised: We will release all the proposed datasets which, we believe, will facilitate future research.

Original: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard trainand test splits as prescribed by their respective authors.
Revised: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard train and test splits as prescribed by their respective authors.

Original: We also note that in the MOVi benchmarki.e. MOVi-A to E, versions D and E are the two most challenging versions.
Revised: We also note that in the MOVi benchmark i.e. MOVi-A to E, versions D and E are the two most challenging versions.

Original: Thus, we consider thesesufficient for evaluating our hypothesis.
Revised: Thus, we consider these sufficient for evaluating our hypothesis.

Original: Baselines.
Revised: Baselines.

Original: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.
Revised: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.

Original: These include two baselines that use mixture-basedreconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and onebaseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).
Revised: These include two baselines that use mixture-based reconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and one baseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).

Original: For Slot Attention
Revised: For Slot Attention

Original: Video, we use the fully unsupervised version that is trained only using the image reconstructionobjective without optical flow or label information in the first frame.
Revised: Video, we use the fully unsupervised version that is trained only using the image reconstruction objective without optical flow or label information in the first frame.

Original: For simplicity, we will refer tothis unsupervised Slot Attention Video as SAVi.
Revised: For simplicity, we will refer to this unsupervised Slot Attention Video as SAVi.

Original: Comparison with SAVi is important because ourmodel is most similar to it with one main difference: our model uses transformer-based decodingwhile SAVi uses mixture-based decoding.
Revised: Comparison with SAVi is important because our model is most similar to it with one main difference: our model uses transformer-based decoding while SAVi uses mixture-based decoding.

Original: Therefore, a comparison of our model with SAVi willprovide a clear insight into the effect of the decoding approach.
Revised: Therefore, a comparison of our model with SAVi will provide a clear insight into the effect of the decoding approach.

Original: For the baseline SLATE, we usethe same CNN backbone as our model for a fair comparison.
Revised: For the baseline SLATE, we use the same CNN backbone as our model for a fair comparison.

Original: In line with the previous works, for
Revised: In line with the previous works, for

Original: SAVi and OP3, we take their decoding masks to be their predicted segmentation.
Revised: SAVi and OP3, we take their decoding masks to be their predicted segmentation.

Original: Since our model
Revised: Since our model

Original: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decodingmasks.
Revised: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decoding masks.

Original: Therefore, we use their input attention masks as our predicted segmentation. 
Revised: Therefore, we use their input attention masks as our predicted segmentation. Note that this 0 1 2 3 4 5 6     # Past Frames I m a g e

Original: Note that thisdifference gives an advantage to the baselines but not to our model as in section 5.3, our analysis willshow that the decoding masks perform better than the input attention masks.
Revised: F G - A R I ( i n % ) CATER STEVESLATESAViOP 0 1 2 3 4 5 6     # Past Frames CATERTex 0 1 2 3 4 5 6     # Past Frames

Revised: MOVi-Solid 0 1 2 3 4 5 6     # Past Frames

Revised: MOVi-Tex 0 1 2 3 4 5 6     # Past Frames

Revised: MOVi-D 0 1 2 3 4 5 6     # Past Frames

Revised: MOVi-E difference gives an advantage to the baselines but not to our model as in section 5.3, our analysis will show that the decoding masks perform better than the input attention masks.

Original: 5.1 Unsupervised Image Segmentation
Revised: 5.1 Unsupervised Image Segmentation

Original: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.
Revised: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.

Original: We report Image FG-ARI which measures how well the predicted segmentation matches the groundtruth segmentation of a given single image.
Revised: We report Image FG-ARI which measures how well the predicted segmentation matches the ground truth segmentation of a given single image.

Original: In line with the previous works (Locatello et al., 2020;
Revised: In line with the previous works (Locatello et al., 2020;

Original: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.
Revised: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.

Original: In Figure 1, we plot the Image FG-ARI of per-frame segmentation.
Revised: In Figure 2, we plot the Image FG-ARI of per-frame segmentation.

Original: Benefit over Mixture-based Methods.
Revised: Benefit over Mixture-based Methods.

Original: Comparing STEVE with SAVi and OP3, we see that STEVEachieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.
Revised: Comparing STEVE with SAVi and OP3, we see that STEVE achieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.

Original: All of these5 datasets are characterized by complex textures.
Revised: All of these 5 datasets are characterized by complex textures.

Original: Therefore, our superior performance shows that ourmodel is more effective in dealing with textured scenes compared to the baselines.
Revised: Therefore, our superior performance shows that our model is more effective in dealing with textured scenes compared to the baselines.

Original: Because the maindifference of STEVE with SAVi is the reconstruction approach, this result provides strong evidencethat the transformer-based reconstruction is the key driver of this improvement.
Revised: Because the main difference of STEVE with SAVi is the reconstruction approach, this result provides strong evidence that the transformer-based reconstruction is the key driver of this improvement.

Original: Analyzing thequalitative results in Figures3 (left), 4  and 6 reveals that the baselines fail completely in CATERTex,
Revised: Analyzing the qualitative results in Figures 4 (left), 5 and 9 reveals that the baselines fail completely in CATERTex,

Original: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regionsinstead of meaningful object regions.
Revised: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regions instead of meaningful object regions.

Original: Unlike simple datasets used in prior works such as CLEVRor CATER that provide strong color contrast between objects, our datasets such as CATERTex and
Revised: Unlike simple datasets used in prior works such as CLEVR or CATER that provide strong color contrast between objects, our datasets such as CATERTex and

Original: MOVi-Tex provide no such color cues.
Revised: MOVi-Tex provide no such color cues.

Original: Despite this, our model can remarkably discover the objects.
Revised: Despite this, our model can remarkably discover the objects.

Original: With the state-of-the-art baselines completely failing, it is for the first time in this line of research thatsuch complex datasets have been effectively handled.
Revised: With the state-of-the-art baselines completely failing, it is for the first time in this line of research that such complex datasets have been effectively handled.

Original: In one dataset i.e. CATER, our performance iscomparable to OP3 and slightly worse than SAVi.
Revised: In one dataset i.e. CATER, our performance is comparable to OP3 and slightly worse than SAVi.

Original: As CATER does not have complex textures, it isnot surprising to see baselines performing well on this dataset.
Revised: As CATER does not have complex textures, it is not surprising to see baselines performing well on this dataset.

Original: In Figure 1, we also find that as themodels see more frames, the segmentation tends to improve.[Figure] [Figure]
Revised: In Figure 2, we also find that as the models see more frames, the segmentation tends to improve.

Original: Benefit of Temporal Learning.
Revised: Benefit of Temporal Learning.

Original: We also analyze whether training STEVE on videos has any benefitin image segmentation compared to SLATE.
Revised: We also analyze whether training STEVE on videos has any benefit in image segmentation compared to SLATE.

Original: As SLATE is applicable only for static images, wetrain and test it on one frame ‘videos’ of our datasets.
Revised: As SLATE is applicable only for static images, we train and test it on one frame ‘videos’ of our datasets.

Original: Comparing STEVE with SLATE, we notethat STEVE performs consistently better in all datasets.
Revised: Comparing STEVE with SLATE, we note that STEVE performs consistently better in all datasets.

Original: Noteworthy is the gap in MOVi-Tex whichis especially large.
Revised: Noteworthy is the gap in MOVi-Tex which is especially large.

Original: This suggests that training on temporal data i.e. videos can be helpful for learningto segment.
Revised: This suggests that training on temporal data i.e. videos can be helpful for learning to segment.

Original: We conjecture that due to the similar texture in the background and foreground, inferringcorrect object regions from a static image can be harder than inferring them when the objects aremoving.
Revised: We conjecture that due to the similar texture in the background and foreground, inferring correct object regions from a static image can be harder than inferring them when the objects are moving.

Original: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to theother datasets.
Revised: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to the other datasets.

Original: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. astatic image), indicating that training on temporal data helps STEVE infer segments on static images.
Revised: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. a static image), indicating that training on temporal data helps STEVE infer segments on static images.

Original: 5.2 Unsupervised Video Segmentation
Revised: 5.2 Unsupervised Video Segmentation

Original: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentationin both space and time without switching identities.
Revised: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentation in both space and time without switching identities.

Original: For this, we compute Video FG-ARI whichconsiders the full trajectory of a segment over the video length as one cluster. 
Revised: For this, we compute Video FG-ARI which considers the full trajectory of a segment over the video length as one cluster. In line with the previous [Figure] [Figure]

Original:  In line with the previousworks Locatello et al.
Revised: OursSAViOP of our model on IID and OOD test sets. In (c), we evaluate our performance in unsupervised video segmentation on natural scenes. works Locatello et al.

Original: ; Kabra et al.
Revised: ; Kabra et al.

Original: (2021); Kipf et al.
Revised: (2021); Kipf et al.

Original: (2021), we consider only the foregroundsegments of the ground truth.
Revised: (2021), we consider only the foreground segments of the ground truth.

Original: Video Segmentation Performance.
Revised: Video Segmentation Performance.

Original: In Figure 2, we see that in 5 out of 6 datasets, all of whichare textured, STEVE significantly outperforms the baselines.
Revised: In Figure 3, we see that in 5 out of 6 datasets, all of which are textured, STEVE significantly outperforms the baselines.

Original: In CATER, the baselines are able toperform comparably or slightly better than ours.
Revised: In CATER, the baselines are able to perform comparably or slightly better than ours.

Original: As with the segmentation result of Section 5.1, thisis not surprising given that CATER is visually simple enough for the baselines to handle well.
Revised: As with the segmentation result of Section 5.1, this is not surprising given that CATER is visually simple enough for the baselines to handle well.

Original: Wealso observe the one major drawback of SLATE in these results i.e. SLATE cannot provide alignedslot representations for a video.
Revised: We also observe the one major drawback of SLATE in these results i.e. SLATE cannot provide aligned slot representations for a video.

Original: This is because SLATE can only be applied to individual frames andthe slots of each frame would be randomly permuted.
Revised: This is because SLATE can only be applied to individual frames and the slots of each frame would be randomly permuted.

Original: We also show how the performance is affectedby the length of the video.
Revised: We also show how the performance is affected by the length of the video.

Original: The models were trained on 3-length videos (6-length for CATER) whilewe evaluate on up to 24-length videos.
Revised: The models were trained on 3-length videos (6-length for CATER) while we evaluate on up to 24-length videos.

Original: As the likelihood of switching identities should increase 
Revised: As the likelihood of switching identities should increase with the video length, all models show some downward slope.

Original: [Figure]
Revised: Going from video length 6 to 24 in

Original: easier to
Revised: CATERTex, MOVi-D, and MOVi-E, our model deteriorates noticeably less than the baselines.

Original: Out-of-Distribution Generalization.
Revised: Out-of-Distribution Generalization.

Original: In Figure 3 (a) and (b) we show how well our trained modelcan generalize at test time to out-of-distribution videos which have more number of objects and unseenobjects and materials.
Revised: In Figure 4 (a) and (b), we show how well our trained model can generalize at test time to out-of-distribution videos which have more number of objects and unseen objects and materials.

Original: We see that our model generalizes well and retains a similar performance asthat on the IID test set.
Revised: We see that our model generalizes well and retains a similar performance as that on the IID test set.

Original: One exception is MOVi-Tex in which our model generalizes well to moreobjects but suffers on novel textures.
Revised: One exception is MOVi-Tex in which our model generalizes well to more objects but suffers on novel textures.

Original: Due to the modular structure of the recurrent encoder, each slotbinds to input objects quite independently.
Revised: Due to the modular structure of the recurrent encoder, each slot binds to input objects quite independently.

Original: We conjecture that this modularity is responsible for goodgeneralization to more objects.
Revised: We conjecture that this modularity is responsible for good generalization to more objects.

Original: However, for new materials, generalization depends on how well thebackbone CNN can generalize.
Revised: However, for new materials, generalization depends on how well the backbone CNN can generalize.

Original: As the training set of MOVi-Tex was generated using a library of 240textures, it would not be surprising if this CNN provides poor features of unseen textures.
Revised: As the training set of MOVi-Tex was generated using a library oftextures, it would not be surprising if this CNN provides poor features of unseen textures.

Original: This, webelieve, is affecting the performance of our model.
Revised: This, we believe, is affecting the performance of our model.

Original: Natural Scenes.
Revised: Natural Scenes.

Original: In Figure 3 (c), we evaluate video segmentation on two natural datasets taken fromreal-world videos: Traffic and Aquarium.
Revised: In Figure 4 (c), we evaluate video segmentation on two natural datasets taken from real-world videos: Traffic and Aquarium.

Original: We see that our model performs significantly better thanthe baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI inthe Aquarium dataset.
Revised: We see that our model performs significantly better than the baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI in the Aquarium dataset.

Original: Noteworthy is the visual complexity of the Aquarium dataset in which thecolor and texture of fish are strongly camouflaged against the background and seeing them requirescareful inspection even for a human (see Figure 4).
Revised: Noteworthy is the visual complexity of the Aquarium dataset in which the color and texture of fish are strongly camouflaged against the background and seeing them requires careful inspection even for a human (see Figure 5).

Original: Despite this, we note that our model is able todeal with this challenge effectively.
Revised: Despite this, we note that our model is able to deal with this challenge effectively.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: In this section, we analyze multiple aspects of our model.

Original: As these have not been investigated in theprior works in the context of the slot-transformer decoder, these results contribute new knowledge tothis line of research.

Original: [Figure]

Original: Diagnosing Slot Representation in STEVE.
Revised: Diagnosing Slot Representation in STEVE.

Original: In previous sections, we focused on segmentationquality and used input attention masks of STEVE.
Revised: In previous sections, we focused on segmentation quality and used input attention masks of STEVE.

Original: However, input attention masks do not provide adirect indication of the information content of slots.
Revised: However, input attention masks do not provide a direct indication of the information content of slots.

Original: We would like to diagnose how well the slotsrepresent the object-centric structure of the scene.
Revised: We would like to diagnose how well the slots represent the object-centric structure of the scene.

Original: For this, we take the slots from the pre-trainedencoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-baseddecoder to reconstruct the observation from the slots.
Revised: For this, we take the slots from the pre-trained encoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-based decoder to reconstruct the observation from the slots.

Original: We call this model STEVE-Diagnostic .
Revised: We call this model STEVE-Diagnostic .

Original: In
Revised: In

Original: Figure 5 (a), we compare the performance of unsupervised video segmentation with the baselinesusing the decoding masks of STEVE-Diagnostic.
Revised: Figure 6 (a), we compare the performance of unsupervised video segmentation with the baselines using the decoding masks of STEVE-Diagnostic.

Original: The decoder of STEVE-diagnostic is made to beexactly the same as that of SAVi for a fair comparison.
Revised: The decoder of STEVE-diagnostic is made to be exactly the same as that of SAVi for a fair comparison.

Original: We find that STEVE-Diagnostic performsbetter than the baselines in all textured datasets, in line with our earlier results in Figures 1 and 2.
Revised: We find that STEVE-Diagnostic performs better than the baselines in all textured datasets, in line with our earlier results in Figures 2 and 3. This shows that the slots in STEVE represent the object-centric structure better than the baselines.

Revised: In

Original: This shows that the slots in STEVE represent the object-centric structure better than the baselines.
Revised: Table 9 in Appendix C, we further analyze the slot contents by training a linear regressor to predict the object positions and find that STEVE performs favorably relative to the baselines.

Original:  Effect of Granularity of dVAE on STEVE.
Revised: [Figure] [Figure] Effect of Granularity of dVAE on STEVE.

Original: In this section, we analyze the effect of granularity ofdiscrete VAE on the segmentation performance of STEVE.
Revised: In this section, we analyze the effect of granularity of discrete VAE on the segmentation performance of STEVE.

Original: In the default configuration of our model,the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patchwith one discrete token.
Revised: In the default configuration of our model, the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patch with one discrete token.

Original: That is, an image of size 128 × 128 would be represented by a sequence oflength 1024.
Revised: That is, an image of size 128 × 128 would be represented by a sequence of length 1024.

Original: In this analysis, we test the effect on segmentation performance if dVAE represents theimage using larger patches i.e. 16 × 16 and 32 × 32 .
Revised: In this analysis, we test the effect on segmentation performance if dVAE represents the image using larger patches i.e. 16 × 16 and 32 × 32 .

Original: In our results shown in Figure 5, we note thatusing larger patches results in a worsening of the performance.
Revised: In our results shown in Figure 6, we note that using larger patches results in a worsening of the performance.

Original: This suggests a possible scaling law that smaller patches lead to better performance.
Revised: This suggests a possible scaling law that smaller patches lead to better performance.

Original: However, due to the prohibitive memory needs of thetransformer on longer sequences, we were not able to experiment with smaller patches. 
Revised: However, due to the prohibitive memory needs of the transformer on longer sequences, we were not able to experiment with smaller patches. Effect of Transformer Capacity on STEVE In Fig.

Revised: 7 in the appendix, we analyze whether our segmentation performance is sensitive to the capacity of the decoder.

Revised: For this, we test the effect of increasing the number of layers in the transformer decoder.

Revised: We find that our segmentation performance is unaffected by this change, showing that our model is robust.

Original: Comparison between Decoding Masks and Input Attention Masks in SAVi.
Revised: Comparison between Decoding Masks and Input Attention Masks in SAVi.

Original: In previous sectionson unsupervised image and video decomposition, we used input attention masks of STEVE while forthe baselines, we used the decoding masks.
Revised: In previous sections on unsupervised image and video decomposition, we used input attention masks of STEVE while for the baselines, we used the decoding masks.

Original: We would like to test how different are the performancesof input attention masks and decoding masks.
Revised: We would like to test how different are the performances of input attention masks and decoding masks.

Original: While OP3 only provides decoding masks, fortunately,
Revised: While OP3 only provides decoding masks, fortunately,

Original: SAVi provides both input masks and decoding masks which we compare in Figure 5 (c).
Revised: SAVi provides both input masks and decoding masks which we compare in Figure 6 (c).

Original: We find thatinput attention masks tend to be worse than decoding masks of SAVi.
Revised: We find that input attention masks tend to be worse than decoding masks of SAVi.

Original: Therefore, the attention masksof STEVE outperform both the input and decoding masks of the baseline SAVi.
Revised: Therefore, the attention masks of STEVE outperform both the input and decoding masks of the baseline SAVi.

Original: Mixture-based decoding without Spatial Broadcast.
Revised: Mixture-based decoding without Spatial Broadcast.

Original: Mixture-based decoding requires a CNNto decode the object image and the mask from each slot.
Revised: Mixture-based decoding requires a CNN to decode the object image and the mask from each slot.

Original: To implement this CNN, the baselinesprescribe using a Spatial Broadcast decoder, a type of CNN decoder that limits the expressivenessof the decoder to facilitate the discovery of objects.
Revised: To implement this CNN, the baselines prescribe using a Spatial Broadcast decoder, a type of CNN decoder that limits the expressiveness of the decoder to facilitate the discovery of objects.

Original: We would like to test if a more flexible CNNimplemented completely using transposed convolutions would be enough to improve the performanceor not.
Revised: We would like to test if a more flexible CNN implemented completely using transposed convolutions would be enough to improve the performance or not.

Original: In Figure 5 (d), our results on the 5 textured datasets show that this change leads to a similaror slightly higher performance than the original decoder.
Revised: In Figure 6 (d), our results on the 5 textured datasets show that this change leads to a similar or slightly higher performance than the original decoder.

Original: However, the model still fails by splittingthe image into fixed patches similarly to before and still performs significantly worse than our model.
Revised: However, the model still fails by splitting the image into fixed patches similarly to before and still performs significantly worse than our model.

Original: This shows that simply using a more flexible CNN with mixture-based decoding is not enough andthe expressiveness of an auto-regressive transformer is important for achieving significant gains.
Revised: This shows that simply using a more flexible CNN with mixture-based decoding is not enough and the expressiveness of an auto-regressive transformer is important for achieving significant gains.

Original: Computational Requirements of STEVE.
Revised: Computational Requirements of STEVE.

Original: In Table 6 in the Appendix, we empirically comparethe computational requirements of STEVE and SAVi.
Revised: In Table 6 in the Appendix, we empirically compare the computational requirements of STEVE and SAVi.

Original: We assess by how much the computationalrequirement change in going from the mixture-based to the transformer-based approach.
Revised: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.

Original: We find thatin videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice forimage size 128 × 128 .
Revised: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .

Original: Given the quadratic memory complexity of transformers, the larger memorydemand of STEVE is not surprising.
Revised: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.

Original: However, one should also consider that the mixture-basedbaselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E,unlike ours. 
Revised: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E, unlike ours. We also provide a big-

Original: Another feature ofour method is that the memory demand of transformer decoder isapproximately constant in the number of slots but linear for mixture-based decoders.
Revised: O analysis of the memory costs in Appendix B.1.

Original: Effect of Camera Motion.
Revised: Effect of Camera Motion.

Original: Our segmentation results on MOVi-D and MOVi-E in Figures 1 and 2provide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camerato MOVi-D. We observe that camera motion in MOVi-E leads to a slight improvement in performanceover MOVi-D.
Revised: Our segmentation results on MOVi-D and MOVi-E in Figures 2 andprovide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camera to MOVi-D. We observe that camera motion in MOVi-E leads to a slight improvement in performance over MOVi-D.

Original: We conjecture that this is because a moving camera causes relative motion betweenobjects and can help better in bringing out patterns that tend to remain stable under motion.
Revised: We conjecture that this is because a moving camera causes relative motion between objects and can help better in bringing out patterns that tend to remain stable under motion.

Original: Static Objects.
Revised: Static Objects.

Original: We also note that our model is effective even when static objects are present inthe scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.
Revised: We also note that our model is effective even when static objects are present in the scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.

Original: In methods that rely purely onoptical flow for discovering objects can suffer significantly with static objects.
Revised: In methods that rely purely on optical flow for discovering objects can suffer significantly with static objects.

Original: For MOVi-D and
Revised: For MOVi-D and

Original: MOVi-E, Greff et al.
Revised: MOVi-E, Greff et al.

Original: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flowsupervision.
Revised: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flow supervision.

Original: Compared to this, our model achieves a significantly better 47.67 and 52.15 on thesedatasets, respectively, despite the static objects and without using any supervision whatsoever.
Revised: Compared to this, our model achieves a significantly better 47.67 and 52.15 on these datasets, respectively, despite the static objects and without using any supervision whatsoever.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervisedobject-centric video learning.
Revised: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervised object-centric video learning.

Original: For this, we propose a simple model, STEVE, and demonstrate thatit can deal with various complex and naturalistic videos without any supervision by significantlyoutperforming previous state-of-the-art baseline models.
Revised: For this, we propose a simple model, STEVE, and demonstrate that it can deal with various complex and naturalistic videos without any supervision by significantly outperforming previous state-of-the-art baseline models.

Original: In experiments, we also find that althoughthe slot-transformer decoder alone is an important component, for videos it is essential to learnalso from temporal information for dealing with complex videos.
Revised: In experiments, we also find that although the slot-transformer decoder alone is an important component, for videos it is essential to learn also from temporal information for dealing with complex videos.

Original: The results of this paper and the
Revised: The results of this paper and the

Original: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centriclearning which is contrary to the traditional view advocating the use of weak mixture decoders.
Revised: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centric learning which is contrary to the traditional view advocating the use of weak mixture decoders.

Original: Thisraises a question to the community: is accurate reconstruction from slots all we need to capture theobjectness?
Revised: This raises a question to the community: is accurate reconstruction from slots all we need to capture the objectness?

Original: In the future, it seems worth investigating further in this direction.
Revised: In the future, it seems worth investigating further in this direction.

Original: Also, because wechoose to investigate a minimal architecture in this paper, it would be interesting to see what gains wecould obtain additionally by exploring more advanced architectures.
Revised: Also, because we choose to investigate a minimal architecture in this paper, it would be interesting to see what gains we could obtain additionally by exploring more advanced architectures.

Original: Lastly, it would be interesting toinvestigate applying this method to large-scale datasets.
Revised: Lastly, it would be interesting to investigate applying this method to large-scale datasets.

Original: We further elaborate on these in Appendix F. 
Revised: We further elaborate on these in Appendix F. Acknowledgments and Disclosure of Funding

Revised: This work is supported by Brain Pool Plus (BP+)

Revised: Program (No. 2021H1D3A2A03103645) and

Revised: Young Researcher Program (No. 2022R1C1C1009443) through the National Research Foundation of

Revised: Korea (NRF) funded by the Ministry of Science and ICT.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?[Yes]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] In supplementarymaterial, we provide the architectural details and the hyperparameters used for ourexperiments.
Revised: [Yes] In supplementary material, we provide the architectural details and the hyperparameters used for our experiments.

Original: We will share the code and our proposed datasets upon acceptance.
Revised: We will share the code and our proposed datasets at https://sites.google.com/view/slot-transformer-for-videos .

Original: (b) Did you specify all the training details ( e.g.,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: data splits,

Original: hyperparameters

Original: , how they

Original: were chosen)? [Yes (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide the amount of resources used in Section B.1 in Appendix. 
Revised: [Yes] We provide the amount of resources used in Section B.1 in Appendix. 4.

Original: To preserve anonymity, we will provide

Original: further details upon acceptance. If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We will share them upon acceptance.
Revised: [Yes] We will share them upon acceptance.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? 
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: [Table] Dataset Descriptions. We generated MOVi-Solid dataset using Kubric (Greff et al., 2022) which introduces textured background and more complex shapes to the MOVi dataset Kipf et al. For this, we used 100 HDR backgrounds and Kubric’s KuBasic shapes.

Original: spent on participant compensation?
Revised: We also generate two test sets: one which has the same number of objects as the training set (3-10 objects) and other which has higher out-of-distribution number of objects (11-12 objects). Similarly to MOVi-Solid, for MOVi-Tex, we use Kubric to generate the dataset. In MOVi-Tex, we modify our MOVi-Solid dataset to have complex textures applied to the objects taken from the Describable Textures dataset Cimpoi et al. Describable Textures dataset provides several categories of textures.

Revised: We used 240 textures each for generating the training set.

Revised: For this, we used 120 textures that are from the banded category andtextures that are from the chequered category.

Revised: We also generated two out-of-distribution test sets: one which has the same number of objects as the training set (3-10 objects) but never-before-seen textures and the other which as the same textures but higher number of objects (11-12 objects).

Revised: For the out-of-distribution test set with never-before-seen textures, 120 textures were taken from the grid category and 120 were taken from the grooved category of the Describable Textures dataset.

Revised: For

Revised: CATER, MOVi-D and MOVi-E, we used the standard training and test splits provided in the respective releases of CATER (with masks) and MOVi datasets.

Revised: We generated CATERTex by integrating the codes for CATER and CLEVRTex (Girdhar & Ramanan, 2020; Karazija et al., 2021).

Revised: We generated two sets: the training and the IID test sets which contain 3-8 objects and an out-of-distribution test set which contains 9-10 objects.

Revised: We collected the Youtube datasets by downloading a 6-hour long video streams, center-cropping and resizing them.

Revised: The Youtube-Traffic dataset was generated from a video feed collected by a traffic camera that watches a fixed intersection.

Revised: The camera does not move during the video.

Revised: The lighting conditions change slightly over time and this has a slight effect on the background and the overall scene from one clip to another.

Revised: The vehicles appear to show diversity in size, color, and type.

Revised: The Youtube-Aquarium dataset was generated from a video collected by a static camera mounted inside a fish exhibit.

Revised: The background, lighting conditions, and other characteristics remain stationary throughout the video.

Revised: The size, color, and types of fish show some diversity.

Revised: As these were collected from Youtube and did not have ground truth masks, we manually annotatedvideos of the test set.

Revised: We will release the training and the test splits of all the datasets.

Revised: B STEVE: Architecture Details

Revised: Recurrent Slot Encoder.

Revised: The design of our slot-based recurrent encoder f slot-rnn ϕ is inspired by Kipf et al.

Revised: At the start of an episode, we provide initial slots s 0 by randomly sampling from a

Revised: Gaussian distribution with learned mean and variance.

Revised: [Equation]

Revised: where µ and σ are learned parameters.

Revised: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.

Revised: The architecture of this CNN is described in Table 2.

Revised: The resulting feature map has size H enc × W enc × D enc .

Revised: It is then flattened on spatial dimensions to obtain

Original: [N/A]
Revised: [Table]

Revised: a set of H enc · W enc features.

Revised: A positional embedding p is added to these followed by a 2-layer MLP.

Revised: With these, we obtain a set of input features e t = { e t, 1 , e t, 2 , . . .

Revised: , e t,HW } .

Revised: The positional embedding is computed by applying a linear map to the 4-dimensional spatial position as introduced by Locatello et al.

Revised: (2020) and Kipf et al.

Revised: Next, the slots of the previous time-step s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .

Revised: For this, the slots in s t − 1 first compute the attention weights A t

Revised: = {A t, 1 , . . .

Revised: , A t,N } over the input features.

Revised: Using the attention weights, the attention result r t,n is computed for each slot via an attention-weighted sum of input features.

Revised: [Equation]

Revised: where q , k and v are linear projections and D is the output size of the projections.

Revised: The attention result r t,n is then used to update the respective slot: ˜ s t,n ← f RNN ( r t,n , s t − 1 ,n ) where the update function f

Revised: RNN is implemented as a GRU (Cho et al., 2014).

Revised: For additional expressiveness, a 2-layer MLP (with residual connection) and LayerNorm is applied on the output of the GRU.

Revised: In practice, these attention and update steps can be executed multiple times per frame.

Revised: We use 2 iterations per frame in our experiments.

Revised: Lastly, the slots are made to interact using an interaction function s t ← f interact ϕ

Revised: [Equation]

Revised: implemented using a 1-layer transformer with 4 heads.

Revised: Following Kipf et al.

Revised: (2021), we use the slots ˜ s t before the interaction step for reconstruction.

Revised: For fairness of comparison between our model

Revised: STEVE and SAVi, we use the same implementation of this backbone recurrent slot encoder in both models.

Revised: Discrete VAE.

Revised: The implementation of our discrete VAE is based on that proposed by (Singh et al., 2022).

Revised: It consists of an encoder network f dVAE ϕ and a decoder network g dVAE θ .

Revised: The encoder network essentially divides the whole image x t into patches of size 4 × 4 and encodes each patch as one discrete token belonging to a vocabulary V .

Revised: We take the size of the vocabulary to be 4096 in our experiments.

Revised: This is achieved in parallel for all patches applying a convolutional neural network on the image of size H × W × 3 as follows.

Revised: Layer

Revised: Kernel Size

Revised: Stride

Revised: Padding

Revised: Channels

Revised: Activation

Revised: Conv 4 ×

Revised: ReLU

Revised: Conv 1 ×

Revised: [Table]

Revised: This results in a feature map of size ( H/ 4) ×

Revised: ( W/ 4) × |V| .

Revised: Each cell i ∈ 1 , . . .

Revised: , L (where L = HW/ 16 ) of this feature map can be seen as a vector o dVAE t,i containing log-probabilities over |V| vocabulary tokens.

Revised: For each cell i , a discrete token is sampled by constructing a categorical distribution using these log-probabilities.

Revised: [Equation]

Revised: In the discrete VAE decoder, the input is a discrete feature map of size ( H/ 4) ×

Revised: ( W/ 4) × |V| whose each cell is a one-hot vector

Revised: z i and the decoder tries to reconstruct the original image from this discrete feature map input.

Revised: This mapping is implemented using a convolutional neural network as described below, finally outputting an image of size H × W × 3 .

Revised: Finally, the whole network is trained by minimizing a image reconstruction loss implemented using squared-error.

Revised: [Equation]

Revised: As there is a discrete sampling step required, we use the Gumbel-Softmax relaxation during training while we use hard sampling for generating discrete targets for the transformer, following Singh et al.

Revised: The temperature parameter for the Gumbel-Softmax is decayed from 1.0 to 0.1 in the first 30K iterations of training as prescribed by Singh et al.

Revised: Transformer Decoder.

Revised: We use the standard design of transformer as introduced by Vaswani et al.

Revised: (2017) and adopted by Singh et al.

Revised: Like SLATE, we use learned positional embeddings for each position i ∈ { 1 , . . .

Revised: , L } in the sequence.

Revised: [Table]

Revised: Channels

Revised: [Table]

Revised: [Table]

Revised: During training, the transformer decoder takes the slots and the dVAE tokens as input and learns to predict the next token.

Revised: We learn to do this next-token prediction for all tokens in parallel by applying causal masking in the style of GPT.

Revised: In further detail, the dVAE tokens given as input to the transformer are obtained via argmax on the logits output by the dVAE encoder.

Revised: That is, we take the dVAE logits

Revised: o dVAE t,i and first perform argmax.

Revised: For this argmax index, we retrieve a learned embedding from a codebook.

Revised: To these retrieved embeddings, we add learned positional encodings and the resulting embeddings are then given as input to the transformer decoder.

Revised: Similarly to SLATE, we do not let gradient be propagated across the argmax operation.

Revised: The prediction targets are the argmax indices of the next dVAE token which we learn to predict using a cross-entropy loss.

Revised: Difference between SLATE and STEVE.

Revised: The key difference between STEVE and SLATE is that

Revised: STEVE temporally extends SLATE by adopting a recurrent slot encoder.

Revised: In doing this, STEVE retains the same decoder architecture as that of SLATE.

Revised: Training.

Revised: We trained all models upto a maximum of 200K training iterations (except for CATERTex for which we trained all models to 400K iterations).

Revised: These amounted to training time of approximately 2 days.

Revised: Similarly to SLATE, we applied a learning rate warmup from 0 to the peak learning rate in the first 30K training iterations.

Revised: After that we decay the learning rate exponentially with a half-life of 250K training iterations.

Revised: The loss function is L STEVE =

Revised: L CE + L dVAE as described in Section 3.

Revised: The gradients from the L dVAE loss are only used to train the dVAE encoder and the dVAE decoder.

Revised: The gradients from the

Revised: L CE cross-entropy loss are only used for training the slot encoder and the transformer decoder.

Revised: This is due to the fact that we do not let the gradients be propagated accross the argmax operation used to obtain the discrete representation of the input frame.

Revised: All 3 components i.e. the dVAE, the slot encoder, and the transformer are trained simultaneously in a single training run.

Revised: B.1 Computational Requirements

Revised: In Table 6, we empirically compare the computational requirements of STEVE and SAVi.

Revised: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.

Revised: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .

Revised: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.

Revised: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOViTex, MOVi-D, and MOVi-E, unlike ours.

Revised: Another feature of our method is that the memory demand of transformer decoder is approximately constant in the number of slots but linear for mixture-based decoders.

Revised: GPU Resources.

Revised: We performed our experiments on Quadro RTX 8000 GPUs.

Revised: We used 2 GPUs per each training seed, each of which took approximately 2-3 days to complete training.

Revised: [Table]

Revised: [Table]

Revised: Big- O Analysis.

Revised: The memory complexity of the transformer decoder is dependent on the cost of self-attention between all the dVAE tokens and the cross-attention between the dVAE tokens and the slots.

Revised: Let the number of dVAE tokens be L and the number of slots be N .

Revised: Then the number of dVAE tokens is L = HW/K 2 where H and W are image height and width, respectively; and K is the dVAE patch size.

Revised: To perform self-attention among all L tokens requires a memory complexity of O

Revised: ( L 2 ) .

Revised: Furthermore, the L tokens also perform cross-attention on the N slots which leads to an attention matrix with memory cost O ( LN ) .

Revised: Adding these attention costs leads to a total cost of ≈ O

Revised: ( L 2 )

Revised: + O ( LN ) .

Revised: But since it is common to have just a few slots N (e.g. N = 11 slots for

Revised: CATER) and a much larger number of tokens (e.g. L = 256 tokens for CATER), the first term O

Revised: ( L 2 ) is typically the dominant one in comparison to the other term

Revised: O

Revised: ( LN ) .

Revised: Replacing L = HW/Kgives an approximate cost of O (( HW ) 2 /K 4 ) .

Revised: This can be contrasted with the memory cost of mixture decoder.

Revised: Each object component can be decoded using a CNN with a cost of O ( HW ) and there are N such components.

Revised: Thus, the total cost is approximately O ( HWN ) .

Revised: These suggest that as we continue of scale up the image size H and W (keeping everything else same), the memory needs of the transformer decoder would significantly surpass that of the mixture decoder.

Revised: B.2 Mixture-based Decoding without Spatial Broadcast

Revised: In Section 5.3, we performed an ablation of SAVi where we replaced its original Spatial Broadcast decoder with a more flexible CNN implemented entirely using transposed convolutions.

Revised: In Tables 7 and 8, we describe these two CNN architectures in detail

Revised: [Table]

Revised: We note that the decoder used in

Revised: [Table]

Revised: [Table]

Revised: the ablation of SAVi has more layers and is therefore more expressive.

Revised: It applies 7 transposed convolution layers while the original decoder had 4.

Revised: Furthermore, unlike the original decoder that uses a Spatial Broadcast in the first layer which is intended to lower its flexibility, our ablation of

Revised: SAVi uses transposed convolutions in all the layers of the CNN decoder.

Revised: C Additional Results.

Revised: In this section we provide additional evaluation results for our model.

Revised: [Table]

Revised: [Figure]

Revised: be seen as a benefit of our model

Revised: [Figure]

Revised: D Additional Related Work

Revised: Unsupervised Video Object Segmentation using Motion Cues.

Revised: When objects in the video are moving independently, optical flow provides a strong signal of objectness as pixels with similar flow can be treated as an object.

Revised: Therefore, several works pursue estimating optical flow in an fully unsupervised way (Ren et al., 2017; Janai et al., 2018;

Revised: Wang et al., 2018; Jonschkowski et al., 2020;

Revised: Stone et al., 2021).

Revised: Leveraging flow, several methods have shown success in visually complex videos (Brox & Malik, 2010; Bideau & Learned-Miller, 2016;

Revised: Yang et al., 2019; Tangemann et al., 2021;

Revised: Yang et al., 2021).

Revised: However, these can require access to optical flow even during deployment which can be challenging to obtain.

Revised: In response, recent methods aim to infer objects via RGB-only input, but by using optical flow to supervise the training (Kipf et al., 2021; Bao et al., 2022).

Revised: Even though these can handle naturalistic videos, their learning depends primarily on the flow information.

Revised: If flow is absent e.g. in scenes with static objects, these can suffer (Greff et al., 2022).

Revised: In comparison, our model is trained given only RGB video frames

Revised: and we consider the use of motion information as orthogonal to our approach.

Revised: Unsupervised Segmentation Propagation.

Revised: Several methods learn to propagate mask from one frame to another via unsupervised training and are shown to handle natural videos (Lai et al., 2020; Vondrick et al., 2018; Lai & Xie, 2019; Zhu et al., 2020;

Revised: Oh et al., 2019; Wang et al., 2019a,b).

Revised: However these methods only learn how to propagate the masks, and still require ground truth segmentation or bounding boxes in the first frame to actually perform tracking.

Revised: Therefore, these methods are not fully unsupervised like ours.

Revised: E Additional Experiment Details

Revised: Evaluating the Effect of Number of Past Frames on Image Segmentation.

Revised: We randomly take a video clip from the test set having T frames: x 1 , . . .

Revised: , x T .

Revised: We would like to measure how the segmentation of the T -th frame x T would change as a function of the number of past frames k shown to the model.

Revised: For this, we predict the object masks M kT for the frame x T given that the model has seen k frames x T − k , . . .

Revised: , x T − 1 in the past.

Revised: We compute the Image FG-ARI for the masks M kT by varying the value of k ∈ { 0 , . . .

Revised: , T − 1 } and make the plot in Figure 2.

Revised: We take T = 7 .

Revised: Evaluation of Out-of-Distribution Generalization to More Objects.

Revised: In this experiment, a model is tested on more objects than it was trained on.

Revised: For this experiment, during testing, we increase the number of slots by an amount equal to the difference in maximum number of objects between the test and the training set.

Revised: F Limitations

Revised: We now describe the main limitations of STEVE and the future directions.

Revised: First, there is some opportunity to improve the performance further within the proposed framework by adopting more advanced architectures than the minimal architecture we choose in this work.

Revised: We would say that this is more of a future work than a limitation of this framework.

Revised: Second, more computational efficiency is required in the future to handle images of higher resolution because the transformer decoder has a quadratic memory complexity and also because token-by-token generation can be slow.

Revised: Third, the potential to scale the model to internet-scale datasets is discussed in this work but not investigated.

Revised: Our current datasets typically contain about 200K video frames with an image resolution of 128.

Revised: For scaling this up, we may also need to develop a new model in this framework.

Revised: Fourth, although our focus in this paper is not on synthetic image generation, we found that the quality of image generation in naturalistic datasets has some room for further improvement.

Revised: We believe that this could not only provide the ability to synthesize images and videos but also improve object decomposition as well.

Revised: Fifth, even if our model outperforms previous works in general, we also observe some failure modes in MOVi-E such as splitting of large objects or merging of very small objects.

Revised: To resolve this, we believe that it’s worth investigating a much larger data on a bigger model or with a more sophisticated architecture.

Revised: Lastly, the CNN backbone of our slot encoder is rather lightweight, having just 4-layers and a feature-size 64.

Revised: For scaling this model to more complex natural images, it is likely that this lightweight CNN will not be enough.

Revised: G Impact Statement

Revised: There are no imminent negative societal consequences of our current work, however, future applications should be mindful to avoid malicious uses such as in surveillance and to minimize environmental impact when training larger transformers.

Revised: At the same time, benign applications in scene understanding, robotics and autonomous navigation would benefit positively from our work.