Original: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
Revised: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Unsupervised object-centric learning aims to represent the modular, compositional,and causal structure of a scene as a set of object representations and therebypromises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.
Revised: Unsupervised object-centric learning aims to represent the modular, compositional,and causal structure of a scene as a set of object representations and therebypromises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.

Original: Although there have been manyremarkable advances in recent years, one of the most critical problems in thisdirection has been that previous methods work only with simple and syntheticscenes but not with complex and naturalistic images or videos.
Revised: Although there have been manyremarkable advances in recent years, one of the most critical problems in thisdirection has been that previous methods work only with simple and syntheticscenes but not with complex and naturalistic images or videos.

Original: In this paper,we propose STEVE, an unsupervised model for object-centric learning in videos.
Revised: In this paper,we propose STEVE, an unsupervised model for object-centric learning in videos.

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line ofresearch.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line ofresearch.

Original: Interestingly, this is achieved by neither adding complexity to the modelarchitecture nor introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding complexity to the modelarchitecture nor introducing a new objective or weak supervision.

Original: Rather, it isachieved by a surprisingly simple architecture that uses a transformer-based imagedecoder conditioned on slots and the learning objective is simply to reconstruct theobservation.
Revised: Rather, it isachieved by a surprisingly simple architecture that uses a transformer-based imagedecoder conditioned on slots and the learning objective is simply to reconstruct theobservation.

Original: Our experiment results on various complex and naturalistic videosshow significant improvements compared to the previous state-of-the-art.
Revised: Our experiment results on various complex and naturalistic videosshow significant improvements compared to the previous state-of-the-art.

Original:  Introduction
Revised:  Introduction

Original: The goal of object-centric representation learning is to represent the inherent structure of the physicalworld such as compositionality and causality as a set of representation vectors (and their relations)each corresponding to a conceptual entity module such as an object (Greff et al., 2020a; Schölkopfet al., 2021).
Revised: The goal of object-centric representation learning is to represent the inherent structure of the physicalworld such as compositionality and causality as a set of representation vectors (and their relations)each corresponding to a conceptual entity module such as an object (Greff et al., 2020; Schölkopfet al., 2021).

Original: Many previous works have demonstrated the potential of this approach as a way ofresolving the key limitations of traditional single-vector representations (Greff et al., 2020a; Schölkopfet al., 2021;
Revised: Many previous works have demonstrated the potential of this approach as a way ofresolving the key limitations of traditional single-vector representations (Greff et al., 2020; Schölkopfet al., 2021;

Original: Dittadi et al., 2021; Eslami et al., 2016;
Revised: Dittadi et al., 2021; Eslami et al., 2016;

Original: Bapst et al., 2019; Mambelli et al., 2022;
Revised: Bapst et al., 2019; Mambelli et al., 2022;

Original: Ghasemipour et al., 2022).
Revised: Ghasemipour et al., 2022).

Original: For example, considering object-centric representations as independentmechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al.
Revised: For example, considering object-centric representations as independentmechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al.

Original: ,2022; Chen et al., 2021).
Revised: ,2022; Chen et al., 2021).

Original: Also, the ability to decompose a visual observation into a set of discreteknowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).
Revised: Also, the ability to decompose a visual observation into a set of discreteknowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).

Original: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.
Revised: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.

Original: Achieving this has primary importancebecause it opens up a way to utilize the enormous amounts of images and videos available on theinternet and thereby unleash the full potential of this method by applying it at scale — the machinelearning community has recently observed the power of scale in large-scale language models (Brownet al., 2020; Bommasani et al., 2021).
Revised: Achieving this has primary importancebecause it opens up a way to utilize the enormous amounts of images and videos available on theinternet and thereby unleash the full potential of this method by applying it at scale — the machinelearning community has recently observed the power of scale in large-scale language models (Brownet al., 2020; Bommasani et al., 2021).

Original: However, despite a significant amount of effort and many remarkable achievements in recent years,we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.
Revised: However, despite a significant amount of effort and many remarkable achievements in recent years,we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.

Original: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeedremains quite elusive.
Revised: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeedremains quite elusive.

Original: As such, previous works have been shown to work only on toy or syntheticimages/videos or resorted to utilizing some supervision such as optical flow or annotation on theinitial frame (Kipf et al., 2021).
Revised: As such, previous works have been shown to work only on toy or syntheticimages/videos or resorted to utilizing some supervision such as optical flow or annotation on theinitial frame (Kipf et al., 2021).

Original: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE(Slot-TransformEr for VidEos).
Revised: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE(Slot-TransformEr for VidEos).

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented inthis line of research.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented inthis line of research.

Original: Interestingly, this is achieved by neither adding much complexity to the modelarchitecture nor by introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding much complexity to the modelarchitecture nor by introducing a new objective or weak supervision.

Original: Rather, it is achieved by asurprisingly simple architecture, comparable to or simpler than most of the previous methods thathave not worked well on complex videos.
Revised: Rather, it is achieved by asurprisingly simple architecture, comparable to or simpler than most of the previous methods thathave not worked well on complex videos.

Original: Moreover, the learning objective is just to reconstruct.
Revised: Moreover, the learning objective is just to reconstruct.

Original: Webelieve this simplicity is one of the best strengths of our model.
Revised: Webelieve this simplicity is one of the best strengths of our model.

Original: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE(Singh et al., 2022).
Revised: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE(Singh et al., 2022).

Original: Because it is demonstrated in the paper that this decoder is the key to dealingwith some visually complex synthetic images, it is of deep interest to the community (i) whetherand how it can be extended to a temporal model utilizing dynamical observations, (ii) if it caneventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.
Revised: Because it is demonstrated in the paper that this decoder is the key to dealingwith some visually complex synthetic images, it is of deep interest to the community (i) whetherand how it can be extended to a temporal model utilizing dynamical observations, (ii) if it caneventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.

Original: Tofocus on investigating these central questions systematically, we intentionally choose not to pursuearchitectural novelty at the expense of adding complexity but to test a minimal architecture thatcombines the SLATE decoder with a standard slot-level recurrence model.
Revised: Tofocus on investigating these central questions systematically, we intentionally choose not to pursuearchitectural novelty at the expense of adding complexity but to test a minimal architecture thatcombines the SLATE decoder with a standard slot-level recurrence model.

Original: Our experiment results onvarious complex and naturalistic videos show that the proposed architecture significantly outperformsprevious state-of-the-art baseline models.
Revised: Our experiment results onvarious complex and naturalistic videos show that the proposed architecture significantly outperformsprevious state-of-the-art baseline models.

Original: From these experiments, we discover important new knowledge to share with the community.
Revised: From these experiments, we discover important new knowledge to share with the community.

Original: We findthat SLATE can deal with complex naturalistic images reasonably well despite not using temporalinformation.
Revised: We findthat SLATE can deal with complex naturalistic images reasonably well despite not using temporalinformation.

Original: Although as a static model, SLATE has a fundamental limitation that it cannot takeadvantage of temporal information, e.g., maintaining consistent object identity across time, if we onlylook at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).
Revised: Although as a static model, SLATE has a fundamental limitation that it cannot takeadvantage of temporal information, e.g., maintaining consistent object identity across time, if we onlylook at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).

Original: Crucially, we also findthat SLATE alone is not enough since our model, STEVE, shows much better frame segmentationthan SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.
Revised: Crucially, we also findthat SLATE alone is not enough since our model, STEVE, shows much better frame segmentationthan SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.

Original: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly,suggesting that learning jointly from both temporal information and the powerful SLATE decoder issignificant and essential to realize the full potential of object-centric representation learning in videos.
Revised: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly,suggesting that learning jointly from both temporal information and the powerful SLATE decoder issignificant and essential to realize the full potential of object-centric representation learning in videos.

Original: Furthermore, our analysis shows that our model is robust to challenges such as static objects andcamera motion and that it also generalizes well to out-of-distribution number of objects and unseentextures.
Revised: Furthermore, our analysis shows that our model is robust to challenges such as static objects andcamera motion and that it also generalizes well to out-of-distribution number of objects and unseentextures.

Original: Lastly, we propose several new datasets that are significantly more complex than thosetackled by the previous unsupervised models.
Revised: Lastly, we propose several new datasets that are significantly more complex than thosetackled by the previous unsupervised models.

Original:  Preliminaries
Revised:  Preliminaries

Original: In unsupervised object-centric representation learning, the goal is to learn to obtain the representationof an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).
Revised: In unsupervised object-centric representation learning, the goal is to learn to obtain the representationof an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).

Original: A commonapproach for this is to auto-encode: a slot-encoder
Revised: A commonapproach for this is to auto-encode: a slot-encoder

Original: f enc ϕ
Revised: f enc ϕ

Original: ( x ) extracts a set of slot representationss =
Revised: ( x ) extracts a set of slot representationss =

Original: { s 1 , . . .
Revised: { s 1 , . . .

Original: s N } from an image x
Revised: s N } from an image x

Original: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from theseslots, where x ∈ R H × W × C and s n ∈ R D .
Revised: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from theseslots, where x ∈ R H × W × C and s n ∈ R D .

Original: In the encoder, the slots compete with each other to learnabout which area of the input will be explained by which slots.
Revised: In the encoder, the slots compete with each other to learnabout which area of the input will be explained by which slots.

Original: This provides a form of informationbottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity likean object.
Revised: This provides a form of informationbottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity likean object.

Original: During training, a reconstruction objective provides the learning signal for training thecomplete model.
Revised: During training, a reconstruction objective provides the learning signal for training thecomplete model.

Original: One nice property of this method is that it only requires reconstructing the imageand does not require any weak supervision or object-aware auxiliary loss terms.
Revised: One nice property of this method is that it only requires reconstructing the imageand does not require any weak supervision or object-aware auxiliary loss terms.

Original: Ideally, we want tomaintain this simplicity in video models, but current state-of-the-art requires weak supervision andoptical flow to make it work on complex naturalistic videos (Kipf et al., 2021).
Revised: Ideally, we want tomaintain this simplicity in video models, but current state-of-the-art requires weak supervision andoptical flow to make it work on complex naturalistic videos (Kipf et al., 2021).

Original: Mixture-based Decoder.
Revised: Mixture-based Decoder.

Original: The emergence of object-centric slots is strongly dependent on whatdecoder is used.
Revised: The emergence of object-centric slots is strongly dependent on whatdecoder is used.

Original: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.
Revised: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.

Original: In this approach, the decoder decodes each slot s n to obtain anobject image ˆ x n and an alpha mask m n via decoding functions g RGB θ
Revised: In this approach, the decoder decodes each slot s n to obtain anobject image ˆ x n and an alpha mask m n via decoding functions g RGB θ

Original: and g mask θ , respectively.
Revised: and g mask θ , respectively.

Original: The decoded object images are then weighted-summed to obtain the full image ˆ x :ˆ x n = g RGB θ( s n
Revised: The decoded object images are then weighted-summed to obtain the full image ˆ x :ˆ x n = g RGB θ

Original: [Equation]
Revised: [Equation]

Original: These decoder networks are implemented using a CNN.
Revised: These decoder networks are implemented using a CNN.

Original: The reconstruction objective is taken to bethe squared error between the input and the reconstructed image i.e. L mixture =
Revised: The reconstruction objective is taken to bethe squared error between the input and the reconstructed image i.e. L mixture =

Original: ∥ ˆ x − x ∥ 2 .
Revised: ∥ ˆ x − x ∥ 2 .

Original: A critical 93 limitation of this approach is that it has never been successful in dealing with scenes with high visualcomplexity like natural images.
Revised: A critical 93 limitation of this approach is that it has never been successful in dealing with scenes with high visualcomplexity like natural images.

Original: Autoregressive Slot-Transformer Decoder.
Revised: Autoregressive Slot-Transformer Decoder.

Original: Recently, Singh et al.
Revised: Recently, Singh et al.

Original: (2022) have challenged thistraditional perspective that we need a mixture decoder for the emergence of objectness in slots.
Revised: (2022) have challenged thistraditional perspective that we need a mixture decoder for the emergence of objectness in slots.

Original: Arguing that a mixture decoder severely limits the interaction among slots and the reconstructionquality—hence a problem in obtaining good learning signal for complex images—they proposed anew architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditionedon the slots.
Revised: Arguing that a mixture decoder severely limits the interaction among slots and the reconstructionquality—hence a problem in obtaining good learning signal for complex images—they proposed anew architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditionedon the slots (Chen et al., 2020).

Original: However, many critical questions about this new approach still remain because thefocus of the SLATE paper is on image generation ability.
Revised: However, many critical questions about this new approach still remainbecause the focus of the SLATE paper is on image generation ability.

Original: To get a deeper understanding of the objectrepresentations that a model like SLATE produces, we would need to thoroughly investigate thequantitative evidence about its scene decomposition ability, its ability to deal with visual complexityclose to that of natural images, and finally, the effect of extending this architecture to a temporalmodel.
Revised: To get a deeper understandingof the object representations that a model like SLATE produces, we would need to thoroughlyinvestigate the quantitative evidence about its scene decomposition ability, its ability to deal withvisual complexity close to that of natural images, and finally, the effect of extending this architectureto a temporal model.

Original: We aim to answer these in this paper.
Revised: We aim to answer these in this paper.

Original:  STEVE: Slot Transformer for Videos
Revised:  STEVE: Slot Transformer for Videos

Original: To study the effect of extending the slot-transformer decoder to a temporal model that can deal withvideos, we choose to propose a minimal architecture.
Revised: To study the effect of extending the slot-transformer decoder to a temporal model that can deal withvideos, we choose to propose a minimal architecture.

Original: Specifically, our proposed model, STEVE,combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder thatupdates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoderof SLATE.
Revised: Specifically, our proposed model, STEVE,combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder thatupdates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoderof SLATE.

Original: Although we may obtain additional performance gains by exploring further architecturalnovelty at the cost of additional complexity, we choose to investigate this minimal architecture tofocus on our goal of investigating slot-transformer decoder models for video.
Revised: Although we may obtain additional performance gains by exploring further architecturalnovelty at the cost of additional complexity, we choose to investigate this minimal architecture tofocus on our goal of investigating slot-transformer decoder models for video.

Original: Given a video consisting of frames x 1 , . . .
Revised: Given a video consisting of frames x 1 , . . .

Original: x T , we want to maintain N slots s t = { s t, 1 , . . .
Revised: x T , we want to maintain N slots s t = { s t, 1 , . . .

Original: , s t,N }for each time-step t ∈ { 1 , . . .
Revised: , s t,N }for each time-step t ∈ { 1 , . . .

Original: T } by starting the temporal update from initial slots s 0 .
Revised: T } by starting the temporal update from initial slots s 0 .

Original: We apply arecurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t usingthe information from the input frame x t .
Revised: We apply arecurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t usingthe information from the input frame x t .

Original: [Equation]
Revised: [Equation]

Original: It is expected that each slot n would represent one object and track it consistently over time in a givenvideo.
Revised: It is expected that each slot n would represent one object and track it consistently over time in a givenvideo.

Original: For each frame, the slot representation s t is used to reconstruct the input frame x t using theslot-transformer decoder.
Revised: For each frame, the slot representation s t is used to reconstruct the input frame x t using theslot-transformer decoder.

Original: For doing such reconstruction, each frame x t is treated as a sequence ofdiscrete tokens provided by a discrete VAE encoder.
Revised: For doing such reconstruction, each frame x t is treated as a sequence ofdiscrete tokens provided by a discrete VAE encoder.

Original: Given the slots s t , the slot-transformer decoderlearns to predict this sequence of tokens
Revised: Given the slots s t , the slot-transformer decoderlearns to predict this sequence of tokens

Original: auto-regressively by minimizing a cross-entropy loss.
Revised: auto-regressively by minimizing a cross-entropy loss.

Original: [Equation]
Revised: [Equation]

Original: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .
Revised: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .

Original: , z t,L } are L discrete tokens that act as 124 prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by thetransformer for the token at position l for frame t .
Revised: , z t,L } are L discrete tokens that act as 124 prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by thetransformer for the token at position l for frame t .

Original: To train the discrete VAE, an image reconstructionloss L dVAE is used as in SLATE.
Revised: To train the discrete VAE, an image reconstructionloss L dVAE is used as in SLATE.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The complete model is trained using L STEVE =
Revised: The complete model is trained using L STEVE =

Original: L CE + L dVAE .
Revised: L CE + L dVAE .

Original: Recurrent Slot Encoder.
Revised: Recurrent Slot Encoder.

Original: Several recent works have implemented recurrent object-centric modelsby incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al.
Revised: Several recent works have implemented recurrent object-centric modelsby incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al.

Original: ,2019; Jiang et al., 2020;
Revised: ,2019; Jiang et al., 2020;

Original: Kossen et al., 2020; Lin et al., 2020a; Stani´c & Schmidhuber, 2019).
Revised: Kossen et al., 2020; Lin et al., 2020a; Stani´c & Schmidhuber, 2019).

Original: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot
Revised: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot

Original: Attention Video (Kipf et al., 2021).
Revised: Attention Video (Kipf et al., 2021).

Original: Our slot-based recurrent encoder works as follows.
Revised: Our slot-based recurrent encoder works as follows.

Original: At the startof an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution withlearned mean and variance.
Revised: At the startof an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution withlearned mean and variance.

Original: For each input frame x t , we compute an encoding in the form of afeature map using a backbone CNN.
Revised: For each input frame x t , we compute an encoding in the form of afeature map using a backbone CNN.

Original: This feature map is flattened to obtain a set of input featurese t = { e t, 1 , e t, 2 , . . .
Revised: This feature map is flattened to obtain a set of input featurese t = { e t, 1 , e t, 2 , . . .

Original: , e t,HW } .
Revised: , e t,HW } .

Original: Next, the slots s t − 1 perform attention on the features e t and use 137 the attention result to update the slots to s t .
Revised: Next, the slots s t − 1 perform attention on the features e t and use 137 the attention result to update the slots to s t .

Original: For this, the slots s t − 1 compute attention weights
Revised: For this, the slots s t − 1 compute attention weights

Original: A t = {A t, 1 , . . .
Revised: A t = {A t, 1 , . . .

Original: , A t,N } over the input features, where A t,n are the attention weights for slot n .
Revised: , A t,N } over the input features, where A t,n are the attention weights for slot n .

Original: The 139 attention weights are used to perform an attention-weighted sum of the input features.
Revised: The 139 attention weights are used to perform an attention-weighted sum of the input features.

Original: The attentionresult r t,n is then used to update the respective slot using an RNN f RNN ϕ
Revised: The attentionresult r t,n is then used to update the respective slot using an RNN f RNN ϕ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, q , k and v are linear projections and D is the output size of the projections.
Revised: Here, q , k and v are linear projections and D is the output size of the projections.

Original: Finally, the slotsare made to interact via an interaction network: s t = f interact ϕ
Revised: Finally, the slotsare made to interact via an interaction network: s t = f interact ϕ

Original: (˜ s t ) .
Revised: (˜ s t ) .

Original: Following Kipf et al.
Revised: Following Kipf et al.

Original: (2021), weuse the slots ˜ s t before the interaction step for reconstruction.
Revised: (2021), weuse the slots ˜ s t before the interaction step for reconstruction.

Original: For complete details of the architecture,see Appendix B.
Revised: For complete details of the architecture,see Appendix B.

Original:  Related Work
Revised:  Related Work

Original: Unsupervised Object-Centric Representation Learning in Images.
Revised: Unsupervised Object-Centric Representation Learning in Images.

Original: Unsupervised object-centriclearning methods for static scenes have received significant interest in recent years (Greff et al., 2016;
Revised: Unsupervised object-centriclearning methods for static scenes have received significant interest in recent years (Greff et al., 2016;

Original: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017a; Engelcke et al. ,2020, 2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; 
Revised: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017; Engelcke et al., 2020,2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Denget al., 2020;

Original: Deng et al., 2020; Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al. ,2020b).
Revised: Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al.,2020).

Original: These models learn through reconstruction, commonly adopting a mixture-based decoder.
Revised: These models learn through reconstruction, commonly adopting a mixture-based decoder.

Original: Another line of work has pursued object discovery by minimizing the mutual information between thepredicted segments (Savarese et al., 2021;
Revised: Another line of work has pursued object discovery by minimizing the mutual information between thepredicted segments (Savarese et al., 2021;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Approaches based on self-supervisedrepresentation learning have also been investigated (Caron et al., 2021;
Revised: Approaches based on self-supervisedrepresentation learning have also been investigated (Caron et al., 2021;

Original: Löwe et al., 2020; Wang et al.,2022).
Revised: Löwe et al., 2020; Wang et al.,2022).

Original: Recently, scene decomposition ability has been shown to emerge in energy-based models (Duet al., 2021a; Yu et al., 2021).
Revised: Recently, scene decomposition ability has been shown to emerge in energy-based models (Duet al., 2021a; Yu et al., 2021).

Original: Complex-valued neural networks have also shown such emergence
Revised: Complex-valued neural networks have also shown such emergence

Original: Lowe et al.
Revised: Lowe et al.

Original: Closely related to our work, Lamb et al.
Revised: Closely related to our work, Lamb et al.

Original: (2021) and Singh et al.
Revised: (2021) and Singh et al.

Original: (2022) haveshown that a slot-based encoder combined with a transformer decoder can enable object emergencein visually complex images.
Revised: (2022) haveshown that a slot-based encoder combined with a transformer decoder can enable object emergencein visually complex images.

Original: However, all of these approaches deal only with static images unlikeours which can be applied to videos.
Revised: However, all of these approaches deal only with static images unlikeours which can be applied to videos.

Original: Unsupervised Object-Centric Representation Learning in Videos.
Revised: Unsupervised Object-Centric Representation Learning in Videos.

Original: A large body of work hasapproached fully-unsupervised video segmentation and tracking by combining recurrent slot-basedencoders with a reconstruction objective.
Revised: A large body of work hasapproached fully-unsupervised video segmentation and tracking by combining recurrent slot-basedencoders with a reconstruction objective.

Original: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber,2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al.
Revised: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber,2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al.

Original: He et al., 2019) use bounding boxes for tracking.
Revised: He et al., 2019) use bounding boxes for tracking.

Original: A parallel line of research learns to localizeobjects via segmentation masks (Greff et al., 2017b;
Revised: A parallel line of research learns to localizeobjects via segmentation masks (Greff et al., 2017;

Original: Van Steenkiste et al., 2018; Veerapaneni et al.
Revised: Van Steenkiste et al., 2018; Veerapaneni et al.

Original: ,2019; Watters et al., 2019;
Revised: ,2019; Watters et al., 2019;

Original: Weis et al., 2020; Du et al., 2020; Kipf et al., 2021;
Revised: Weis et al., 2020; Du et al., 2020; Kipf et al., 2021;

Original: Kabra et al., 2021;
Revised: Kabra et al., 2021;

Original: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).
Revised: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).

Original: Our work lies alongthis line of research.
Revised: Our work lies alongthis line of research.

Original: Other approaches have considered using contrastive losses (Kipf et al., 2019;
Revised: Other approaches have considered using contrastive losses (Kipf et al., 2019;

Original: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3Dscenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al.
Revised: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3Dscenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al.

Original: ,2021; Du et al., 2021b; Kabra et al., 2021).
Revised: ,2021; Du et al., 2021b; Kabra et al., 2021).

Original: While dealing with 3D scenes would be an importantfuture direction for our work, it is orthogonal to our current focus.
Revised: While dealing with 3D scenes would be an importantfuture direction for our work, it is orthogonal to our current focus.

Original: However, unlike ours, all ofthe above works have only been successful on visually simple videos.
Revised: However, unlike ours, all ofthe above works have only been successful on visually simple videos.

Original: In Appendix D, we discussthe related work that deals with unsupervised video object segmentation using motion cues andunsupervised segmentation propagation.
Revised: In Appendix D, we discussthe related work that deals with unsupervised video object segmentation using motion cues andunsupervised segmentation propagation.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate our model on 8 datasets.
Revised: We evaluate our model on 8 datasets.

Original: These include 6 procedurally generated datasets:
Revised: These include 6 procedurally generated datasets:

Original: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E(Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.
Revised: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E(Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.

Original: Out of these, 5 datasets are our contributions in this work which we now describe.
Revised: Out of these, 5 datasets are our contributions in this work which we now describe.

Original: i)
Revised: i)

Original: CATERTex.
Revised: CATERTex.

Original: We generated it by combiningobjects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar& Ramanan, 2020).
Revised: We generated it by combiningobjects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar& Ramanan, 2020).

Original: ii) MOVi-Solid.
Revised: ii) MOVi-Solid.

Original: We used Kubric (Greff et al., 2021) to generate this dataset andintroduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipfet al. (2021).
Revised: We used Kubric (Greff et al., 2022) to generate this dataset andintroduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipfet al. (2021).

Original: iii) MOVi-Tex.
Revised: iii) MOVi-Tex.

Original: We added the textures from the Describable Textures dataset (Cimpoiet al., 2014) as materials to the MOVi-Solid dataset to make this dataset.
Revised: We added the textures from the Describable Textures dataset (Cimpoiet al., 2014) as materials to the MOVi-Solid dataset to make this dataset.

Original: By applying the samematerial on all surfaces, the objects and their borders become significantly harder to distinguish,resulting in much higher visual complexity over MOVi-Solid.
Revised: By applying the samematerial on all surfaces, the objects and their borders become significantly harder to distinguish,resulting in much higher visual complexity over MOVi-Solid.

Original: iv) Traffic and Aquarium.
Revised: iv) Traffic and Aquarium.

Original: We createdthese two natural datasets by collecting a 6-hour long video stream from Youtube.
Revised: We createdthese two natural datasets by collecting a 6-hour long video stream from Youtube.

Original: For training,all models use only the raw videos as input with no other supervision or input whatsoever.
Revised: For training,all models use only the raw videos as input with no other supervision or input whatsoever.

Original: Forevaluation, we use the ground truth instance-level masks.
Revised: Forevaluation, we use the ground truth instance-level masks.

Original: For sample video frames and more details,see Appendix A.
Revised: For sample video frames and more details,see Appendix A.

Original: We will release all the proposed datasets which, we believe, will facilitate futureresearch.
Revised: We will release all the proposed datasets which, we believe, will facilitate futureresearch.

Original: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard trainand test splits as prescribed by their respective authors.
Revised: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard trainand test splits as prescribed by their respective authors.

Original: We also note that in the MOVi benchmarki.e.
Revised: We also note that in the MOVi benchmarki.e.

Original: MOVi-A to E, versions D and E are the two most challenging versions.
Revised: MOVi-A to E, versions D and E are the two most challenging versions.

Original: Thus, we consider thesesufficient for evaluating our hypothesis.
Revised: Thus, we consider thesesufficient for evaluating our hypothesis.

Original: Baselines.
Revised: Baselines.

Original: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.
Revised: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.

Original: These include two baselines that use mixture-basedreconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and onebaseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).
Revised: These include two baselines that use mixture-basedreconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and onebaseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).

Original: For Slot Attention
Revised: For Slot Attention

Original: Video, we use the fully unsupervised version that is trained only using the image reconstructionobjective without optical flow or label information in the first frame.
Revised: Video, we use the fully unsupervised version that is trained only using the image reconstructionobjective without optical flow or label information in the first frame.

Original: For simplicity, we will refer tothis unsupervised Slot Attention Video as SAVi.
Revised: For simplicity, we will refer tothis unsupervised Slot Attention Video as SAVi.

Original: Comparison with SAVi is important because ourmodel is most similar to it with one main difference: our model uses transformer-based decodingwhile SAVi uses mixture-based decoding.
Revised: Comparison with SAVi is important because ourmodel is most similar to it with one main difference: our model uses transformer-based decodingwhile SAVi uses mixture-based decoding.

Original: Therefore, a comparison of our model with SAVi willprovide a clear insight into the effect of the decoding approach.
Revised: Therefore, a comparison of our model with SAVi willprovide a clear insight into the effect of the decoding approach.

Original: For the baseline SLATE, we usethe same CNN backbone as our model for a fair comparison.
Revised: For the baseline SLATE, we usethe same CNN backbone as our model for a fair comparison.

Original: In line with the previous works, for
Revised: In line with the previous works, for

Original: SAVi and OP3, we take their decoding masks to be their predicted segmentation.
Revised: SAVi and OP3, we take their decoding masks to be their predicted segmentation.

Original: Since our model
Revised: Since our model

Original: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decodingmasks.
Revised: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decodingmasks.

Original: Therefore, we use their input attention masks as our predicted segmentation.
Revised: Therefore, we use their input attention masks as our predicted segmentation.

Original: Note that thisdifference gives an advantage to the baselines but not to our model as in section 5.3, our analysis willshow that the decoding masks perform better than the input attention masks.
Revised: Note that thisdifference gives an advantage to the baselines but not to our model as in section 5.3, our analysis willshow that the decoding masks perform better than the input attention masks.

Original: 5.1 Unsupervised Image Segmentation
Revised: 5.1 Unsupervised Image Segmentation

Original: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.
Revised: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.

Original: We report Image FG-ARI which measures how well the predicted segmentation matches the groundtruth segmentation of a given single image.
Revised: We report Image FG-ARI which measures how well the predicted segmentation matches the groundtruth segmentation of a given single image.

Original: In line with the previous works (Locatello et al., 2020;
Revised: In line with the previous works (Locatello et al., 2020;

Original: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.
Revised: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.

Original: In Figure 1, we plot the Image FG-ARI of per-frame segmentation.
Revised: In Figure 1, we plot the Image FG-ARI of per-frame segmentation.

Original: Benefit over Mixture-based Methods.
Revised: Benefit over Mixture-based Methods.

Original: Comparing STEVE with SAVi and OP3, we see that STEVEachieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.
Revised: Comparing STEVE with SAVi and OP3, we see that STEVEachieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.

Original: All of these5 datasets are characterized by complex textures.
Revised: All of these5 datasets are characterized by complex textures.

Original: Therefore, our superior performance shows that ourmodel is more effective in dealing with textured scenes compared to the baselines.
Revised: Therefore, our superior performance shows that ourmodel is more effective in dealing with textured scenes compared to the baselines.

Original: Because the maindifference of STEVE with SAVi is the reconstruction approach, this result provides strong evidencethat the transformer-based reconstruction is the key driver of this improvement.
Revised: Because the maindifference of STEVE with SAVi is the reconstruction approach, this result provides strong evidencethat the transformer-based reconstruction is the key driver of this improvement.

Original: Analyzing thequalitative results in Figures 3 (left), 4 and 6 reveals that the baselines fail completely in CATERTex,
Revised: Analyzing thequalitative results in Figures 3 (left), 4 and 6 reveals that the baselines fail completely in CATERTex,

Original: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regionsinstead of meaningful object regions.
Revised: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regionsinstead of meaningful object regions.

Original: Unlike simple datasets used in prior works such as CLEVRor CATER that provide strong color contrast between objects, our datasets such as CATERTex and
Revised: Unlike simple datasets used in prior works such as CLEVRor CATER that provide strong color contrast between objects, our datasets such as CATERTex and

Original: MOVi-Tex provide no such color cues.
Revised: MOVi-Tex provide no such color cues.

Original: Despite this, our model can remarkably discover the objects.
Revised: Despite this, our model can remarkably discover the objects.

Original: With the state-of-the-art baselines completely failing, it is for the first time in this line of research thatsuch complex datasets have been effectively handled.
Revised: With the state-of-the-art baselines completely failing, it is for the first time in this line of research thatsuch complex datasets have been effectively handled.

Original: In one dataset i.e. CATER, our performance iscomparable to OP3 and slightly worse than SAVi.
Revised: In one dataset i.e. CATER, our performance iscomparable to OP3 and slightly worse than SAVi.

Original: As CATER does not have complex textures, it isnot surprising to see baselines performing well on this dataset.
Revised: As CATER does not have complex textures, it isnot surprising to see baselines performing well on this dataset.

Original: In Figure 1, we also find that as themodels see more frames, the segmentation tends to improve.
Revised: In Figure 1, we also find that as themodels see more frames, the segmentation tends to improve.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Benefit of Temporal Learning.
Revised: Benefit of Temporal Learning.

Original: We also analyze whether training STEVE on videos has any benefitin image segmentation compared to SLATE.
Revised: We also analyze whether training STEVE on videos has any benefitin image segmentation compared to SLATE.

Original: As SLATE is applicable only for static images, wetrain and test it on one frame ‘videos’ of our datasets.
Revised: As SLATE is applicable only for static images, wetrain and test it on one frame ‘videos’ of our datasets.

Original: Comparing STEVE with SLATE, we notethat STEVE performs consistently better in all datasets.
Revised: Comparing STEVE with SLATE, we notethat STEVE performs consistently better in all datasets.

Original: Noteworthy is the gap in MOVi-Tex whichis especially large.
Revised: Noteworthy is the gap in MOVi-Tex whichis especially large.

Original: This suggests that training on temporal data i.e. videos can be helpful for learningto segment.
Revised: This suggests that training on temporal data i.e. videos can be helpful for learningto segment.

Original: We conjecture that due to the similar texture in the background and foreground, inferringcorrect object regions from a static image can be harder than inferring them when the objects aremoving.
Revised: We conjecture that due to the similar texture in the background and foreground, inferringcorrect object regions from a static image can be harder than inferring them when the objects aremoving.

Original: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to theother datasets.
Revised: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to theother datasets.

Original: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. astatic image), indicating that training on temporal data helps STEVE infer segments on static images.
Revised: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. astatic image), indicating that training on temporal data helps STEVE infer segments on static images.

Original: 5.2 Unsupervised Video Segmentation
Revised: 5.2 Unsupervised Video Segmentation

Original: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentationin both space and time without switching identities.
Revised: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentationin both space and time without switching identities.

Original: For this, we compute Video FG-ARI whichconsiders the full trajectory of a segment over the video length as one cluster.
Revised: For this, we compute Video FG-ARI whichconsiders the full trajectory of a segment over the video length as one cluster.

Original: In line with the previousworks Locatello et al.
Revised: In line with the previousworks Locatello et al.

Original: ; Kabra et al.
Revised: ; Kabra et al.

Original: (2021); Kipf et al.
Revised: (2021); Kipf et al.

Original: (2021), we consider only the foregroundsegments of the ground truth.
Revised: (2021), we consider only the foregroundsegments of the ground truth.

Original: Video Segmentation Performance.
Revised: Video Segmentation Performance.

Original: In Figure 2, we see that in 5 out of 6 datasets, all of whichare textured, STEVE significantly outperforms the baselines.
Revised: In Figure 2, we see that in 5 out of 6 datasets, all of whichare textured, STEVE significantly outperforms the baselines.

Original: In CATER, the baselines are able toperform comparably or slightly better than ours.
Revised: In CATER, the baselines are able toperform comparably or slightly better than ours.

Original: As with the segmentation result of Section 5.1, thisis not surprising given that CATER is visually simple enough for the baselines to handle well.
Revised: As with the segmentation result of Section 5.1, thisis not surprising given that CATER is visually simple enough for the baselines to handle well.

Original: Wealso observe the one major drawback of SLATE in these results i.e. SLATE cannot provide alignedslot representations for a video.
Revised: Wealso observe the one major drawback of SLATE in these results i.e. SLATE cannot provide alignedslot representations for a video.

Original: This is because SLATE can only be applied to individual frames andthe slots of each frame would be randomly permuted.
Revised: This is because SLATE can only be applied to individual frames andthe slots of each frame would be randomly permuted.

Original: We also show how the performance is affectedby the length of the video.
Revised: We also show how the performance is affectedby the length of the video.

Original: The models were trained on 3-length videos (6-length for CATER) whilewe evaluate on up to 24-length videos.
Revised: The models were trained on 3-length videos (6-length for CATER) whilewe evaluate on up to 24-length videos.

Original: As the likelihood of switching identities should increase
Revised: As the likelihood of switching identities should increase

Original: [Figure]
Revised: [Figure]

Original: We show the video frames and their true masks followed by foreground and background segmentation of each model. We separated the visualization of the foreground segments to make the result easier tointerpret by applying a threshold on the segment area. See the supplementary material for more with the video length models show some downward slope video 6 to 24 in noticeably less than
Revised: easier to

Original: Out-of-Distribution Generalization.
Revised: Out-of-Distribution Generalization.

Original: In Figure 3 (a) and (b), we show how well our trained modelcan generalize at test time to out-of-distribution videos which have more number of objects and unseenobjects and materials.
Revised: In Figure 3 (a) and (b) we show how well our trained modelcan generalize at test time to out-of-distribution videos which have more number of objects and unseenobjects and materials.

Original: We see that our model generalizes well and retains a similar performance asthat on the IID test set.
Revised: We see that our model generalizes well and retains a similar performance asthat on the IID test set.

Original: One exception is MOVi-Tex in which our model generalizes well to moreobjects but suffers on novel textures.
Revised: One exception is MOVi-Tex in which our model generalizes well to moreobjects but suffers on novel textures.

Original: Due to the modular structure of the recurrent encoder, each slotbinds to input objects quite independently.
Revised: Due to the modular structure of the recurrent encoder, each slotbinds to input objects quite independently.

Original: We conjecture that this modularity is responsible for goodgeneralization to more objects.
Revised: We conjecture that this modularity is responsible for goodgeneralization to more objects.

Original: However, for new materials, generalization depends on how well thebackbone CNN can generalize.
Revised: However, for new materials, generalization depends on how well thebackbone CNN can generalize.

Original: As the training set of MOVi-Tex was generated using a library of 240textures, it would not be surprising if this CNN provides poor features of unseen textures.
Revised: As the training set of MOVi-Tex was generated using a library of 240textures, it would not be surprising if this CNN provides poor features of unseen textures.

Original: This, webelieve, is affecting the performance of our model.
Revised: This, webelieve, is affecting the performance of our model.

Original: Natural Scenes.
Revised: Natural Scenes.

Original: In Figure 3 (c), we evaluate video segmentation on two natural datasets taken fromreal-world videos: Traffic and Aquarium.
Revised: In Figure 3 (c), we evaluate video segmentation on two natural datasets taken fromreal-world videos: Traffic and Aquarium.

Original: We see that our model performs significantly better thanthe baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI inthe Aquarium dataset.
Revised: We see that our model performs significantly better thanthe baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI inthe Aquarium dataset.

Original: Noteworthy is the visual complexity of the Aquarium dataset in which thecolor and texture of fish are strongly camouflaged against the background and seeing them requirescareful inspection even for a human (see Figure 4).
Revised: Noteworthy is the visual complexity of the Aquarium dataset in which thecolor and texture of fish are strongly camouflaged against the background and seeing them requirescareful inspection even for a human (see Figure 4).

Original: Despite this, we note that our model is able todeal with this challenge effectively.
Revised: Despite this, we note that our model is able todeal with this challenge effectively.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: In this section, we analyze multiple aspects of our model.
Revised: In this section, we analyze multiple aspects of our model.

Original: As these have not been investigated in theprior works in the context of the slot-transformer decoder, these results contribute new knowledge tothis line of research.
Revised: As these have not been investigated in theprior works in the context of the slot-transformer decoder, these results contribute new knowledge tothis line of research.

Original: [Figure]
Revised: [Figure]

Original: Diagnosing Slot Representation in STEVE.
Revised: Diagnosing Slot Representation in STEVE.

Original: In previous sections, we focused on segmentationquality and used input attention masks of STEVE.
Revised: In previous sections, we focused on segmentationquality and used input attention masks of STEVE.

Original: However, input attention masks do not provide adirect indication of the information content of slots.
Revised: However, input attention masks do not provide adirect indication of the information content of slots.

Original: We would like to diagnose how well the slotsrepresent the object-centric structure of the scene.
Revised: We would like to diagnose how well the slotsrepresent the object-centric structure of the scene.

Original: For this, we take the slots from the pre-trainedencoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-baseddecoder to reconstruct the observation from the slots.
Revised: For this, we take the slots from the pre-trainedencoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-baseddecoder to reconstruct the observation from the slots.

Original: We call this model STEVE-Diagnostic .
Revised: We call this model STEVE-Diagnostic .

Original: In
Revised: In

Original: Figure 5 (a), we compare the performance of unsupervised video segmentation with the baselinesusing the decoding masks of STEVE-Diagnostic.
Revised: Figure 5 (a), we compare the performance of unsupervised video segmentation with the baselinesusing the decoding masks of STEVE-Diagnostic.

Original: The decoder of STEVE-diagnostic is made to beexactly the same as that of SAVi for a fair comparison.
Revised: The decoder of STEVE-diagnostic is made to beexactly the same as that of SAVi for a fair comparison.

Original: We find that STEVE-Diagnostic performsbetter than the baselines in all textured datasets, in line with our earlier results in Figures 1 and 2.
Revised: We find that STEVE-Diagnostic performsbetter than the baselines in all textured datasets, in line with our earlier results in Figures 1 and 2.

Original: This shows that the slots in STEVE represent the object-centric structure better than the baselines.
Revised: This shows that the slots in STEVE represent the object-centric structure better than the baselines.

Original: Effect of Granularity of dVAE on STEVE.
Revised: Effect of Granularity of dVAE on STEVE.

Original: In this section, we analyze the effect of granularity ofdiscrete VAE on the segmentation performance of STEVE.
Revised: In this section, we analyze the effect of granularity ofdiscrete VAE on the segmentation performance of STEVE.

Original: In the default configuration of our model,the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patchwith one discrete token.
Revised: In the default configuration of our model,the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patchwith one discrete token.

Original: That is, an image of size 128 × 128 would be represented by a sequence oflength 1024.
Revised: That is, an image of size 128 × 128 would be represented by a sequence oflength 1024.

Original: In this analysis, we test the effect on segmentation performance if dVAE represents theimage using larger patches i.e. 16 × 16 and 32 × 32 .
Revised: In this analysis, we test the effect on segmentation performance if dVAE represents theimage using larger patches i.e. 16 × 16 and 32 × 32 .

Original: In our results shown in Figure 5, we note thatusing larger patches results in a worsening of the performance.
Revised: In our results shown in Figure 5, we note thatusing larger patches results in a worsening of the performance.

Original: This suggests a possible scaling law that smaller patches lead to better performance.
Revised: This suggests a possible scaling law that smaller patches lead to better performance.

Original: However, due to the prohibitive memory needs of thetransformer on longer sequences, we were not able to experiment with smaller patches.
Revised: However, due to the prohibitive memory needs of thetransformer on longer sequences, we were not able to experiment with smaller patches.

Original: Comparison between Decoding Masks and Input Attention Masks in SAVi.
Revised: Comparison between Decoding Masks and Input Attention Masks in SAVi.

Original: In previous sectionson unsupervised image and video decomposition, we used input attention masks of STEVE while forthe baselines, we used the decoding masks.
Revised: In previous sectionson unsupervised image and video decomposition, we used input attention masks of STEVE while forthe baselines, we used the decoding masks.

Original: We would like to test how different are the performancesof input attention masks and decoding masks.
Revised: We would like to test how different are the performancesof input attention masks and decoding masks.

Original: While OP3 only provides decoding masks, fortunately,
Revised: While OP3 only provides decoding masks, fortunately,

Original: SAVi provides both input masks and decoding masks which we compare in Figure 5 (c).
Revised: SAVi provides both input masks and decoding masks which we compare in Figure 5 (c).

Original: We findthat input attention masks tend to be worse than decoding masks of SAVi.
Revised: We find thatinput attention masks tend to be worse than decoding masks of SAVi.

Original: Therefore, in all thecomparisons, the decoding masks gave an advantage to the baselines but not to our model.

Original: Despitethis disadvantage, the attention  masks ofSTEVE significantly outperformed the baselines.
Revised: Therefore, the attention masksof STEVE outperform both the input and decoding masks of the baseline SAVi.

Original: Effect of Deconvolution-based Decoder in SAVi.
Revised: Mixture-based decoding without Spatial Broadcast.

Original: Mixture-based decoding requires a CNN to decodethe object image and the mask from each slot.
Revised: Mixture-based decoding requires a CNNto decode the object image and the mask from each slot.

Original: To implement this CNN, the baselines have prescribedusing a Spatial Broadcast decoder, a type of CNN decoder that limits the expressiveness of the decoderto facilitate the discovery of objects.
Revised: To implement this CNN, the baselinesprescribe using a Spatial Broadcast decoder, a type of CNN decoder that limits the expressivenessof the decoder to facilitate the discovery of objects.

Original: We would like to test ifapplying a more flexible deconvolutionbased CNN would be enough to improve the performance or not.
Revised: We would like to test if a more flexible CNNimplemented completely using transposed convolutions would be enough to improve the performanceor not.

Original: In Figure 5 (d), our results on the5 textured datasets show that this change leads to a similar or slightly higher performance than theoriginal decoder.
Revised: In Figure 5 (d), our results on the 5 textured datasets show that this change leads to a similaror slightly higher performance than the original decoder.

Original: However, the model still fails by splitting the image into fixed patches similarlyto before and still performs significantly worse than our model.
Revised: However, the model still fails by splittingthe image into fixed patches similarly to before and still performs significantly worse than our model.

Original: This shows that simply increasingthe capacity of CNN in mixture-based decoding is not enough and the expressiveness of a powerfulauto-regressive transformer is important for achieving significant gains.
Revised: This shows that simply using a more flexible CNN with mixture-based decoding is not enough andthe expressiveness of an auto-regressive transformer is important for achieving significant gains.

Original: Computational Requirements of STEVE.
Revised: Computational Requirements of STEVE.

Original: In Table 4 in the Appendix, we empirically comparethe computational requirements of STEVE and SAVi.
Revised: In Table 6 in the Appendix, we empirically comparethe computational requirements of STEVE and SAVi.

Original: We assess by how much the computationalrequirement change in going from the mixture-based to the transformer-based approach.
Revised: We assess by how much the computationalrequirement change in going from the mixture-based to the transformer-based approach.

Original: We find thatin videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice forimage size 128 × 128 .
Revised: We find thatin videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice forimage size 128 × 128 .

Original: Given the quadratic memory complexity of transformers, the larger memorydemand of STEVE is not surprising.
Revised: Given the quadratic memory complexity of transformers, the larger memorydemand of STEVE is not surprising.

Original: However, one should also consider that the mixture-basedbaselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E,unlike ours.
Revised: However, one should also consider that the mixture-basedbaselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E,unlike ours.

Original: Another feature of our method is that the memory demand of transformer decoder isapproximately constant in the number of slots but linear for mixture-based decoders.
Revised: Another feature of our method is that the memory demand of transformer decoder isapproximately constant in the number of slots but linear for mixture-based decoders.

Original: Effect of Camera Motion.
Revised: Effect of Camera Motion.

Original: Our segmentation results on MOVi-D and MOVi-E in Figures 1 and 2provide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camerato
Revised: Our segmentation results on MOVi-D and MOVi-E in Figures 1 and 2provide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camerato

Original: MOVi-D.
Revised: MOVi-D.

Original: We observe that camera motion in MOVi-E leads to a slight improvement in performanceover MOVi-D.
Revised: We observe that camera motion in MOVi-E leads to a slight improvement in performanceover MOVi-D.

Original: We conjecture that this is because a moving camera causes relative motion betweenobjects and can help better in bringing out patterns that tend to remain stable under motion.
Revised: We conjecture that this is because a moving camera causes relative motion betweenobjects and can help better in bringing out patterns that tend to remain stable under motion.

Original: Static Objects.
Revised: Static Objects.

Original: We also note that our model is effective even when static objects are present inthe scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.
Revised: We also note that our model is effective even when static objects are present inthe scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.

Original: In methods that rely purely onoptical flow for discovering objects can suffer significantly with static objects.
Revised: In methods that rely purely onoptical flow for discovering objects can suffer significantly with static objects.

Original: For MOVi-D and
Revised: For MOVi-D and

Original: MOVi-E, Greff et al.
Revised: MOVi-E, Greff et al.

Original: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flowsupervision.
Revised: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flowsupervision.

Original: Compared to this, our model achieves a significantly better 47.67 and 52.15 on thesedatasets, respectively, despite the static objects and without using any supervision whatsoever.
Revised: Compared to this, our model achieves a significantly better 47.67 and 52.15 on thesedatasets, respectively, despite the static objects and without using any supervision whatsoever.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervisedobject-centric video learning.
Revised: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervisedobject-centric video learning.

Original: For this, we propose a simple model, STEVE, and demonstrate thatit can deal with various complex and naturalistic videos without any supervision by significantlyoutperforming previous state-of-the-art baseline models.
Revised: For this, we propose a simple model, STEVE, and demonstrate thatit can deal with various complex and naturalistic videos without any supervision by significantlyoutperforming previous state-of-the-art baseline models.

Original: In experiments, we also find that althoughthe slot-transformer decoder alone is an important component, for videos it is essential to learnalso from temporal information for dealing with complex videos.
Revised: In experiments, we also find that althoughthe slot-transformer decoder alone is an important component, for videos it is essential to learnalso from temporal information for dealing with complex videos.

Original: The results of this paper and the
Revised: The results of this paper and the

Original: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centriclearning which is contrary to the traditional view advocating the use of weak mixture decoders.
Revised: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centriclearning which is contrary to the traditional view advocating the use of weak mixture decoders.

Original: Thisraises a question to the community: is accurate reconstruction from slots all we need to capture theobjectness?
Revised: Thisraises a question to the community: is accurate reconstruction from slots all we need to capture theobjectness?

Original: In the future, it seems worth investigating further in this direction.
Revised: In the future, it seems worth investigating further in this direction.

Original: Also, because wechoose to investigate a minimal architecture in this paper, it would be interesting to see what gains wecould obtain additionally by exploring more advanced architectures.
Revised: Also, because wechoose to investigate a minimal architecture in this paper, it would be interesting to see what gains wecould obtain additionally by exploring more advanced architectures.

Original: Lastly, it would be interesting toinvestigate applying this method to large-scale datasets.
Revised: Lastly, it would be interesting toinvestigate applying this method to large-scale datasets.

Revised: We further elaborate on these in Appendix F.

Original: Checklist
Revised: Checklist

Original: For all authors...(
Revised: For all authors...

Original: a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes
Revised: [Yes]

Original: (b ) Did you describe the limitations of your work
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL

Original: Yes ] In supplementarymaterial, we provide the architectural details and the hyperparameters used for ourexperiments.
Revised: [Yes] In supplementarymaterial, we provide the architectural details and the hyperparameters used for ourexperiments.

Original: We will share the code and our proposed datasets upon acceptance.
Revised: We will share the code and our proposed datasets upon acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details ( e.g.,

Revised: data splits,

Revised: hyperparameters

Original:  how they
Revised: , how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,
Revised: (d) Did you include the total amount of compute and the type of resources used (e g.,

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide the amount of
Revised: [Yes] We provide the amount of

Original: resources used in Section B.2 in Appendix.
Revised: resources used in Section B.1 in Appendix.

Original: To preserve anonymity, we will provide
Revised: To preserve anonymity, we will provide

Original: further details upon acceptance.
Revised: further details upon acceptance.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: We will share them upon acceptance.
Revised: We will share them upon acceptance.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]