Original: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
Revised: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos

Original: Abstract
Revised: Abstract

Original: Unsupervised object-centric learning aims to represent the modular, compositional, and causal structure of a scene as a set of object representations and thereby promises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.
Revised: Unsupervised object-centric learning aims to represent the modular, compositional, and causal structure of a scene as a set of object representations and thereby promises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization.

Original: Although there have been many remarkable advances in recent years, one of the most critical problems in this direction has been that previous methods work only with simple and synthetic scenes but not with complex and naturalistic images or videos.
Revised: Although there have been many remarkable advances in recent years, one of the most critical problems in this direction has been that previous methods work only with simple and synthetic scenes but not with complex and naturalistic images or videos.

Original: In this paper, we propose STEVE, an unsupervised model for object-centric learning in videos.
Revised: In this paper, we propose STEVE, an unsupervised model for object-centric learning in videos.

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line of research.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line of research.

Original: Interestingly, this is achieved by neither adding complexity to the model architecture nor introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding complexity to the model architecture nor introducing a new objective or weak supervision.

Original: Rather, it is achieved by a surprisingly simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation.
Revised: Rather, it is achieved by a surprisingly simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation.

Original: Our experiment results on various complex and naturalistic videos show significant improvements compared to the previous state-of-the-art.
Revised: Our experiment results on various complex and naturalistic videos show significant improvements compared to the previous state-of-the-art.

Original: https://sites.google.com/view/slot-transformer-for-videos
Revised: https://sites.google.com/view/slot-transformer-for-videos

Original:  Introduction
Revised:  Introduction

Original: The goal of object-centric representation learning is to represent the inherent structure of the physical world such as compositionality and causality as a set of representation vectors (and their relations) each corresponding to a conceptual entity module such as an object (Greff et al., 2020; Schölkopf et al., 2021).
Revised: The goal of object-centric representation learning is to represent the inherent structure of the physical world such as compositionality and causality as a set of representation vectors (and their relations) each corresponding to a conceptual entity module such as an object (Greff et al., 2020; Schölkopf et al., 2021).

Original: Many previous works have demonstrated the potential of this approach as a way of resolving the key limitations of traditional single-vector representations (Greff et al., 2020; Schölkopf et al., 2021;
Revised: Many previous works have demonstrated the potential of this approach as a way of resolving the key limitations of traditional single-vector representations (Greff et al., 2020; Schölkopf et al., 2021;

Original: Dittadi et al., 2021; Eslami et al., 2016;
Revised: Dittadi et al., 2021; Eslami et al., 2016;

Original: Bapst et al., 2019; Mambelli et al., 2022;
Revised: Bapst et al., 2019; Mambelli et al., 2022;

Original: Ghasemipour et al., 2022).
Revised: Ghasemipour et al., 2022).

Original: For example, considering object-centric representations as independent mechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al., 2022; Chen et al., 2021).
Revised: For example, considering object-centric representations as independent mechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al., 2022; Chen et al., 2021).

Original: Also, the ability to decompose a visual observation into a set of discrete knowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).
Revised: Also, the ability to decompose a visual observation into a set of discrete knowledge modules has shown to be useful for visual reasoning (Zhou et al., 2021; Wu et al., 2021).

Original: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.
Revised: One of the most critical challenges remaining in object-centric representation learning is to successfully apply it to complex and natural images or videos.

Original: Achieving this has primary importance because it opens up a way to utilize the enormous amounts of images and videos available on the internet and thereby unleash the full potential of this method by applying it at scale — the machine learning community has recently observed the power of scale in large-scale language models (Brown et al., 2020; Bommasani et al., 2021).
Revised: Achieving this has primary importance because it opens up a way to utilize the enormous amounts of images and videos available on the internet and thereby unleash the full potential of this method by applying it at scale — the machine learning community has recently observed the power of scale in large-scale language models (Brown et al., 2020; Bommasani et al., 2021).

Original: However, despite a significant amount of effort and many remarkable achievements in recent years, we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.
Revised: However, despite a significant amount of effort and many remarkable achievements in recent years, we do not yet have an unsupervised object-centric learning method that works well on complex and naturalistic videos.

Original: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeed remains quite elusive.
Revised: In fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeed remains quite elusive.

Original: As such, previous works have been shown to work only on toy or synthetic images/videos or resorted to utilizing some supervision such as optical flow or annotation on the initial frame (Kipf et al., 2021).
Revised: As such, previous works have been shown to work only on toy or synthetic images/videos or resorted to utilizing some supervision such as optical flow or annotation on the initial frame (Kipf et al., 2021).

Original: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE (Slot-TransformEr for VidEos).
Revised: In this paper, we propose an unsupervised model for object-centric learning in videos, called STEVE (Slot-TransformEr for VidEos).

Original: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented in this line of research.
Revised: Our proposed model makes a significant advancement by demonstrating its effectiveness for the first time on various complex and naturalistic videos unprecedented in this line of research.

Original: Interestingly, this is achieved by neither adding much complexity to the model architecture nor by introducing a new objective or weak supervision.
Revised: Interestingly, this is achieved by neither adding much complexity to the model architecture nor by introducing a new objective or weak supervision.

Original: Rather, it is achieved by a surprisingly simple architecture, comparable to or simpler than most of the previous methods that have not worked well on complex videos.
Revised: Rather, it is achieved by a surprisingly simple architecture, comparable to or simpler than most of the previous methods that have not worked well on complex videos.

Original: Moreover, the learning objective is just to reconstruct.
Revised: Moreover, the learning objective is just to reconstruct.

Original: We believe this simplicity is one of the best strengths of our model.
Revised: We believe this simplicity is one of the best strengths of our model.

Original: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE (Singh et al., 2022).
Revised: Behind this success is the adoption of the transformer-based slot decoder recently proposed in SLATE (Singh et al., 2022).

Original: Because it is demonstrated in the paper that this decoder is the key to dealing with some visually complex synthetic images, it is of deep interest to the community (i) whether and how it can be extended to a temporal model utilizing dynamical observations, (ii) if it can eventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.
Revised: Because it is demonstrated in the paper that this decoder is the key to dealing with some visually complex synthetic images, it is of deep interest to the community (i) whether and how it can be extended to a temporal model utilizing dynamical observations, (ii) if it can eventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this.

Original: To focus on investigating these central questions systematically, we intentionally choose not to pursue architectural novelty at the expense of adding complexity but to test a minimal architecture that combines the SLATE decoder with a standard slot-level recurrence model.
Revised: To focus on investigating these central questions systematically, we intentionally choose not to pursue architectural novelty at the expense of adding complexity but to test a minimal architecture that combines the SLATE decoder with a standard slot-level recurrence model.

Original: Our experiment results on various complex and naturalistic videos show that the proposed architecture significantly outperforms previous state-of-the-art baseline models.
Revised: Our experiment results on various complex and naturalistic videos show that the proposed architecture significantly outperforms previous state-of-the-art baseline models.

Original: From these experiments, we discover important new knowledge to share with the community.
Revised: From these experiments, we discover important new knowledge to share with the community.

Original: We find that SLATE can deal with complex naturalistic images reasonably well despite not using temporal information.
Revised: We find that SLATE can deal with complex naturalistic images reasonably well despite not using temporal information.

Original: Although as a static model, SLATE has a fundamental limitation that it cannot take advantage of temporal information, e.g., maintaining consistent object identity across time, if we only look at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).
Revised: Although as a static model, SLATE has a fundamental limitation that it cannot take advantage of temporal information, e.g., maintaining consistent object identity across time, if we only look at its frame-level segmentation ability, it already performs better than the previous state-of-theart temporal model that applies slot attention to videos (Kipf et al., 2021).

Original: Crucially, we also find that SLATE alone is not enough since our model, STEVE, shows much better frame segmentation than SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.
Revised: Crucially, we also find that SLATE alone is not enough since our model, STEVE, shows much better frame segmentation than SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do.

Original: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly, suggesting that learning jointly from both temporal information and the powerful SLATE decoder is significant and essential to realize the full potential of object-centric representation learning in videos.
Revised: In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly, suggesting that learning jointly from both temporal information and the powerful SLATE decoder is significant and essential to realize the full potential of object-centric representation learning in videos.

Original: Furthermore, our analysis shows that our model is robust to challenges such as static objects and camera motion and that it also generalizes well to out-of-distribution number of objects and unseen textures.
Revised: Furthermore, our analysis shows that our model is robust to challenges such as static objects and camera motion and that it also generalizes well to out-of-distribution number of objects and unseen textures.

Original: Lastly, we propose several new datasets that are significantly more complex than those tackled by the previous unsupervised models.
Revised: Lastly, we propose several new datasets that are significantly more complex than those tackled by the previous unsupervised models.

Original:  Preliminaries
Revised:  Preliminaries

Original: In unsupervised object-centric representation learning, the goal is to learn to obtain the representation of an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).
Revised: In unsupervised object-centric representation learning, the goal is to learn to obtain the representation of an observation as a set of object vectors, referred to as slots (Locatello et al., 2020).

Original: A common approach for this is to auto-encode: a slot-encoder
Revised: A common approach for this is to auto-encode: a slot-encoder

Original: f enc ϕ
Revised: f enc ϕ

Original: ( x ) extracts a set of slot representations s
Revised: ( x ) extracts a set of slot representations s

Original: = { s 1 , . . .
Revised: = { s 1 , . . .

Original: s N } from an image x
Revised: s N } from an image x

Original: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from these slots, where x ∈ R H × W × C and s n ∈ R D .
Revised: and then a decoder f dec ϕ ( s ) makes a reconstruction ˆ x from these slots, where x ∈ R H × W × C and s n ∈ R D .

Original: In the encoder, the slots compete with each other to learn about which area of the input will be explained by which slots.
Revised: In the encoder, the slots compete with each other to learn about which area of the input will be explained by which slots.

Original: This provides a form of information bottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity like an object.
Revised: This provides a form of information bottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity like an object.

Original: During training, a reconstruction objective provides the learning signal for training the complete model.
Revised: During training, a reconstruction objective provides the learning signal for training the complete model.

Original: One nice property of this method is that it only requires reconstructing the image and does not require any weak supervision or object-aware auxiliary loss terms.
Revised: One nice property of this method is that it only requires reconstructing the image and does not require any weak supervision or object-aware auxiliary loss terms.

Original: Ideally, we want to maintain this simplicity in video models, but current state-of-the-art requires weak supervision and optical flow to make it work on complex naturalistic videos (Kipf et al., 2021).
Revised: Ideally, we want to maintain this simplicity in video models, but current state-of-the-art requires weak supervision and optical flow to make it work on complex naturalistic videos (Kipf et al., 2021).

Original: Mixture-based Decoder.
Revised: Mixture-based Decoder.

Original: The emergence of object-centric slots is strongly dependent on what decoder is used.
Revised: The emergence of object-centric slots is strongly dependent on what decoder is used.

Original: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.
Revised: The traditional way that most of the previous methods have taken is the mixturebased decoder or some variant of it.

Original: In this approach, the decoder decodes each slot s n to obtain an object image ˆ x n and an alpha mask m n via decoding functions g RGB θ
Revised: In this approach, the decoder decodes each slot s n to obtain an object image ˆ x n and an alpha mask m n via decoding functions g RGB θ

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: θ , respectively.
Revised: θ , respectively.

Original: The decoded object images are then weighted-summed to obtain the full image ˆ x :
Revised: The decoded object images are then weighted-summed to obtain the full image ˆ x :

Original: [Equation]
Revised: [Equation]

Original: n = g
Revised: n = g

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: These decoder networks are implemented using a CNN.
Revised: These decoder networks are implemented using a CNN.

Original: The reconstruction objective is taken to be the squared error between the input and the reconstructed image i.e. L mixture =
Revised: The reconstruction objective is taken to be the squared error between the input and the reconstructed image i.e. L mixture =

Original: ∥ ˆ x − x ∥ 2 .
Revised: ∥ ˆ x − x ∥ 2 .

Original: A critical limitation of this approach is that it has never been successful in dealing with scenes with high visual complexity like natural images.
Revised: A critical limitation of this approach is that it has never been successful in dealing with scenes with high visual complexity like natural images.

Original: Autoregressive Slot-Transformer Decoder.
Revised: Autoregressive Slot-Transformer Decoder.

Original: Recently, Singh et al.
Revised: Recently, Singh et al.

Original: (2022) have challenged this traditional perspective that we need a mixture decoder for the emergence of objectness in slots.
Revised: (2022) have challenged this traditional perspective that we need a mixture decoder for the emergence of objectness in slots.

Original: Arguing that a mixture decoder severely limits the interaction among slots and the reconstruction quality—hence a problem in obtaining good learning signal for complex images—they proposed a new architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditioned on the slots (Chen et al., 2020).
Revised: Arguing that a mixture decoder severely limits the interaction among slots and the reconstruction quality—hence a problem in obtaining good learning signal for complex images—they proposed a new architecture, SLATE, using a powerful autoregressive decoder based on a transformer conditioned on the slots (Chen et al., 2020).

Original: However, many critical questions about this new approach still remain because the focus of the SLATE paper is on image generation ability.
Revised: However, many critical questions about this new approach still remain because the focus of the SLATE paper is on image generation ability.

Original: To get a deeper understanding of the object representations that a model like SLATE produces, we would need to thoroughly investigate the quantitative evidence about its scene decomposition ability, its ability to deal with visual complexity close to that of natural images, and finally, the effect of extending this architecture to a temporal model.
Revised: To get a deeper understanding of the object representations that a model like SLATE produces, we would need to thoroughly investigate the quantitative evidence about its scene decomposition ability, its ability to deal with visual complexity close to that of natural images, and finally, the effect of extending this architecture to a temporal model.

Original: We aim to answer these in this paper.
Revised: We aim to answer these in this paper.

Original:  STEVE: Slot Transformer for Videos
Revised:  STEVE: Slot Transformer for Videos

Original: To study the effect of extending the slot-transformer decoder to a temporal model that can deal with videos, we choose to propose a minimal architecture.
Revised: To study the effect of extending the slot-transformer decoder to a temporal model that can deal with videos, we choose to propose a minimal architecture.

Original: Specifically, our proposed model, STEVE, combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE.
Revised: Specifically, our proposed model, STEVE, combines three main components: (1) a CNN-based image encoder, (2) a recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE.

Original: Although we may obtain additional performance gains by exploring further architectural novelty at the cost of additional complexity, we choose to investigate this minimal architecture to focus on our goal of investigating slot-transformer decoder models for video.
Revised: Although we may obtain additional performance gains by exploring further architectural novelty at the cost of additional complexity, we choose to investigate this minimal architecture to focus on our goal of investigating slot-transformer decoder models for video.

Original: Given a video consisting of frames x 1 , . . .
Revised: Given a video consisting of frames x 1 , . . .

Original: x T , we want to maintain N slots s t = { s t, 1 , . . .
Revised: x T , we want to maintain N slots s t = { s t, 1 , . . .

Original: , s t,N } for each time-step t ∈ { 1 , . . .
Revised: , s t,N } for each time-step t ∈ { 1 , . . .

Original: T } by starting the temporal update from initial slots s 0 .
Revised: T } by starting the temporal update from initial slots s 0 .

Original: We apply a recurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t using the information from the input frame x t .
Revised: We apply a recurrent slot encoder f slot-rnn ϕ at each step t to update the slot representations from s t − 1 to s t using the information from the input frame x t .

Original: [Equation]
Revised: [Equation]

Original: It is expected that each slot n would represent one object and track it consistently over time in a given video.
Revised: It is expected that each slot n would represent one object and track it consistently over time in a given video.

Original: For each frame, the slot representation s t is used to reconstruct the input frame x t using the slot-transformer decoder.
Revised: For each frame, the slot representation s t is used to reconstruct the input frame x t using the slot-transformer decoder.

Original: For doing such reconstruction, each frame x t is treated as a sequence of discrete tokens provided by a discrete VAE encoder.
Revised: For doing such reconstruction, each frame x t is treated as a sequence of discrete tokens provided by a discrete VAE encoder.

Original: Given the slots s t , the slot-transformer decoder learns to predict this sequence of tokens auto-regressively by minimizing a cross-entropy loss.
Revised: Given the slots s t , the slot-transformer decoder learns to predict this sequence of tokens auto-regressively by minimizing a cross-entropy loss.

Original: [Equation]
Revised: [Equation]

Original: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .
Revised: where CE( · , · ) denotes cross-entropy loss, z t = { z t, 1 , . . .

Original: , z t,L } are L discrete tokens that act as prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by the transformer for the token at position l for frame t .
Revised: , z t,L } are L discrete tokens that act as prediction targets for the transformer at time t and o t,l contains the log-probabilities predicted by the transformer for the token at position l for frame t .

Original: To train the discrete VAE, an image reconstruction loss L dVAE is used as in SLATE.
Revised: To train the discrete VAE, an image reconstruction loss L dVAE is used as in SLATE.

Original: [Equation]
Revised: [Equation]

Original: ˆ x t =
Revised: ˆ x t =

Original: g dVAE
Revised: g dVAE

Original: θ ( z t, 1 , . . .
Revised: θ ( z t, 1 , . . .

Original: z t,L ) .
Revised: z t,L ) .

Original: The complete model is trained using L STEVE =
Revised: The complete model is trained using L STEVE =

Original: L CE + L dVAE .
Revised: L CE + L dVAE .

Original: Recurrent Slot Encoder.
Revised: Recurrent Slot Encoder.

Original: Several recent works have implemented recurrent object-centric models by incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al., 2019; Jiang et al., 2020; Kossen et al., 2020; Lin et al., 2020a;
Revised: Several recent works have implemented recurrent object-centric models by incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al., 2019; Jiang et al., 2020; Kossen et al., 2020; Lin et al., 2020a;

Original: Stani ´ c & Schmidhuber, 2019).
Revised: Stani ´ c & Schmidhuber, 2019).

Original: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot
Revised: For our model, we leverage a slot-based recurrent encoder that is also used as the backbone of Slot

Original: Attention Video (Kipf et al., 2021).
Revised: Attention Video (Kipf et al., 2021).

Original: Our slot-based recurrent encoder works as follows.
Revised: Our slot-based recurrent encoder works as follows.

Original: At the start of an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution with learned mean and variance.
Revised: At the start of an episode, we provide initial slots s 0 by randomly sampling from a Gaussian distribution with learned mean and variance.

Original: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.
Revised: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.

Original: This feature map is flattened to obtain a set of input features e t = { e t, 1 , e t, 2 , . . .
Revised: This feature map is flattened to obtain a set of input features e t = { e t, 1 , e t, 2 , . . .

Original: , e t,HW } .
Revised: , e t,HW } .

Original: Next, the slots s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .
Revised: Next, the slots s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .

Original: For this, the slots s t − 1 compute attention weights
Revised: For this, the slots s t − 1 compute attention weights

Original: A t = {A t, 1 , . . .
Revised: A t = {A t, 1 , . . .

Original: , A t,N } over the input features, where A t,n are the attention weights for slot n .
Revised: , A t,N } over the input features, where A t,n are the attention weights for slot n .

Original: The attention weights are used to perform an attention-weighted sum of the input features.
Revised: The attention weights are used to perform an attention-weighted sum of the input features.

Original: The attention result r t,n is then used to update the respective slot using an RNN f RNN ϕ .
Revised: The attention result r t,n is then used to update the respective slot using an RNN f RNN ϕ .

Original: [Equation]
Revised: [Equation]

Original: s t,n
Revised: s t,n

Original: [Equation]
Revised: [Equation]

Original: ϕ
Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Here, q , k and v are linear projections and D is the output size of the projections.
Revised: Here, q , k and v are linear projections and D is the output size of the projections.

Original: Finally, the slots are made to interact via an interaction network: s t = f interact ϕ
Revised: Finally, the slots are made to interact via an interaction network: s t = f interact ϕ

Original: (˜ s t ) .
Revised: (˜ s t ) .

Original: Following Kipf et al.
Revised: Following Kipf et al.

Original: (2021), we use the slots ˜ s t before the interaction step for reconstruction.
Revised: (2021), we use the slots ˜ s t before the interaction step for reconstruction.

Original: For complete details of the architecture, see Appendix B.
Revised: For complete details of the architecture, see Appendix B.

Original:  Related Work
Revised:  Related Work

Original: Unsupervised Object-Centric Representation Learning in Images.
Revised: Unsupervised Object-Centric Representation Learning in Images.

Original: Unsupervised object-centric learning methods for static scenes have received significant interest in recent years (Greff et al., 2016;
Revised: Unsupervised object-centric learning methods for static scenes have received significant interest in recent years (Greff et al., 2016;

Original: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017; Engelcke et al., 2020, 2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Deng et al., 2020; Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al., 2020).
Revised: Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2017; Engelcke et al., 2020, 2021; Eslami et al., 2016; Crawford & Pineau, 2019b; Lin et al., 2020b; Jiang & Ahn, 2020; Deng et al., 2020; Chen et al., 2021; Anciukevicius et al., 2020; von Kügelgen et al., 2020; Greff et al., 2020).

Original: These models learn through reconstruction, commonly adopting a mixture-based decoder.
Revised: These models learn through reconstruction, commonly adopting a mixture-based decoder.

Original: Another line of work has pursued object discovery by minimizing the mutual information between the predicted segments (Savarese et al., 2021;
Revised: Another line of work has pursued object discovery by minimizing the mutual information between the predicted segments (Savarese et al., 2021;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Approaches based on self-supervised representation learning have also been investigated (Caron et al., 2021; Löwe et al., 2020; Wang et al., 2022).
Revised: Approaches based on self-supervised representation learning have also been investigated (Caron et al., 2021; Löwe et al., 2020; Wang et al., 2022).

Original: Recently, scene decomposition ability has been shown to emerge in energy-based models (Du et al., 2021a; Yu et al., 2021).
Revised: Recently, scene decomposition ability has been shown to emerge in energy-based models (Du et al., 2021a; Yu et al., 2021).

Original: Complex-valued neural networks have also shown such emergence
Revised: Complex-valued neural networks have also shown such emergence

Original: Lowe et al.
Revised: Lowe et al.

Original: Closely related to our work, Lamb et al.
Revised: Closely related to our work, Lamb et al.

Original: (2021) and Singh et al.
Revised: (2021) and Singh et al.

Original: (2022) have shown that a slot-based encoder combined with a transformer decoder can enable object emergence in visually complex images.
Revised: (2022) have shown that a slot-based encoder combined with a transformer decoder can enable object emergence in visually complex images.

Original: However, all of these approaches deal only with static images unlike ours which can be applied to videos.
Revised: However, all of these approaches deal only with static images unlike ours which can be applied to videos.

Original: Unsupervised Object-Centric Representation Learning in Videos.
Revised: Unsupervised Object-Centric Representation Learning in Videos.

Original: A large body of work has approached fully-unsupervised video segmentation and tracking by combining recurrent slot-based encoders with a reconstruction objective.
Revised: A large body of work has approached fully-unsupervised video segmentation and tracking by combining recurrent slot-based encoders with a reconstruction objective.

Original: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber, 2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al., 2021;
Revised: In this class, (Kosiorek et al., 2018; Stani´c & Schmidhuber, 2019; Jiang et al., 2020; Crawford & Pineau, 2019a; Lin et al., 2020a; Wu et al., 2021; Singh et al., 2021;

Original: He et al., 2019) use bounding boxes for tracking.
Revised: He et al., 2019) use bounding boxes for tracking.

Original: A parallel line of research learns to localize objects via segmentation masks (Greff et al., 2017;
Revised: A parallel line of research learns to localize objects via segmentation masks (Greff et al., 2017;

Original: Van Steenkiste et al., 2018; Veerapaneni et al., 2019; Watters et al., 2019; Weis et al., 2020; Du et al., 2020; Kipf et al., 2021; Kabra et al., 2021;
Revised: Van Steenkiste et al., 2018; Veerapaneni et al., 2019; Watters et al., 2019; Weis et al., 2020; Du et al., 2020; Kipf et al., 2021; Kabra et al., 2021;

Original: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).
Revised: Zoran et al., 2021; Besbinar & Frossard, 2021; Creswell et al., 2020, 2021).

Original: Our work lies along this line of research.
Revised: Our work lies along this line of research.

Original: Other approaches have considered using contrastive losses (Kipf et al., 2019;
Revised: Other approaches have considered using contrastive losses (Kipf et al., 2019;

Original: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3D scenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al., 2021;
Revised: Carvalho et al., 2020) while some works have explored unsupervised object-centric learning in 3D scenes (Chen et al., 2021; Henderson & Lampert, 2020; Crawford & Pineau, 2020; Stelzner et al., 2021;

Original: Du et al., 2021b; Kabra et al., 2021).
Revised: Du et al., 2021b; Kabra et al., 2021).

Original: While dealing with 3D scenes would be an important future direction for our work, it is orthogonal to our current focus.
Revised: While dealing with 3D scenes would be an important future direction for our work, it is orthogonal to our current focus.

Original: However, unlike ours, all of the above works have only been successful on visually simple videos.
Revised: However, unlike ours, all of the above works have only been successful on visually simple videos.

Original: In Appendix D, we discuss the related work that deals with unsupervised video object segmentation using motion cues and unsupervised segmentation propagation.
Revised: In Appendix D, we discuss the related work that deals with unsupervised video object segmentation using motion cues and unsupervised segmentation propagation.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate our model on 8 datasets.
Revised: We evaluate our model on 8 datasets.

Original: These include 6 procedurally generated datasets:
Revised: These include 6 procedurally generated datasets:

Original: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.
Revised: CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Greff et al., 2022); and 2 natural datasets: Traffic and Aquarium.

Original: Out of these, 5 datasets are our contributions in this work which we now describe.
Revised: Out of these, 5 datasets are our contributions in this work which we now describe.

Original: i)
Revised: i)

Original: CATERTex.
Revised: CATERTex.

Original: We generated it by combining objects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar & Ramanan, 2020).
Revised: We generated it by combining objects and textures of CLEVRTex (Karazija et al., 2021) with the object motion of CATER (Girdhar & Ramanan, 2020).

Original: ii) MOVi-Solid.
Revised: ii) MOVi-Solid.

Original: We used Kubric (Greff et al., 2022) to generate this dataset and introduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipf et al.
Revised: We used Kubric (Greff et al., 2022) to generate this dataset and introduced textured backgrounds and more complex shapes to the MOVi dataset proposed by Kipf et al.

Original: iii) MOVi-Tex.
Revised: iii) MOVi-Tex.

Original: We added the textures from the Describable Textures dataset (Cimpoi et al., 2014) as materials to the MOVi-Solid dataset to make this dataset.
Revised: We added the textures from the Describable Textures dataset (Cimpoi et al., 2014) as materials to the MOVi-Solid dataset to make this dataset.

Original: By applying the same material on all surfaces, the objects and their borders become significantly harder to distinguish, resulting in much higher visual complexity over MOVi-Solid.
Revised: By applying the same material on all surfaces, the objects and their borders become significantly harder to distinguish, resulting in much higher visual complexity over MOVi-Solid.

Original: iv) Traffic and Aquarium.
Revised: iv) Traffic and Aquarium.

Original: We created these two natural datasets by collecting a 6-hour long video stream from Youtube.
Revised: We created these two natural datasets by collecting a 6-hour long video stream from Youtube.

Original: For training, all models use only the raw videos as input with no other supervision or input whatsoever.
Revised: For training, all models use only the raw videos as input with no other supervision or input whatsoever.

Original: For evaluation, we use the ground truth instance-level masks.
Revised: For evaluation, we use the ground truth instance-level masks.

Original: For sample video frames and more details, see Appendix A.
Revised: For sample video frames and more details, see Appendix A.

Original: We will release all the proposed datasets which, we believe, will facilitate future research.
Revised: We will release all the proposed datasets which, we believe, will facilitate future research.

Original: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard train and test splits as prescribed by their respective authors.
Revised: For existing benchmarks i.e. CATER, MOVi-D and MOVi-E, we use the standard train and test splits as prescribed by their respective authors.

Original: We also note that in the MOVi benchmark i.e. MOVi-A to E, versions D and E are the two most challenging versions.
Revised: We also note that in the MOVi benchmark i.e. MOVi-A to E, versions D and E are the two most challenging versions.

Original: Thus, we consider these sufficient for evaluating our hypothesis.
Revised: Thus, we consider these sufficient for evaluating our hypothesis.

Original: Baselines.
Revised: Baselines.

Original: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.
Revised: We compare our performance with three baselines which are state-of-the-art in unsupervised object-centric scene decomposition.

Original: These include two baselines that use mixture-based reconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and one baseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).
Revised: These include two baselines that use mixture-based reconstruction: Slot Attention Video (Kipf et al., 2021) and OP3 (Veerapaneni et al., 2019); and one baseline which uses transformer-based reconstruction: SLATE (Singh et al., 2022).

Original: For Slot Attention
Revised: For Slot Attention

Original: Video, we use the fully unsupervised version that is trained only using the image reconstruction objective without optical flow or label information in the first frame.
Revised: Video, we use the fully unsupervised version that is trained only using the image reconstruction objective without optical flow or label information in the first frame.

Original: For simplicity, we will refer to this unsupervised Slot Attention Video as SAVi.
Revised: For simplicity, we will refer to this unsupervised Slot Attention Video as SAVi.

Original: Comparison with SAVi is important because our model is most similar to it with one main difference: our model uses transformer-based decoding while SAVi uses mixture-based decoding.
Revised: Comparison with SAVi is important because our model is most similar to it with one main difference: our model uses transformer-based decoding while SAVi uses mixture-based decoding.

Original: Therefore, a comparison of our model with SAVi will provide a clear insight into the effect of the decoding approach.
Revised: Therefore, a comparison of our model with SAVi will provide a clear insight into the effect of the decoding approach.

Original: For the baseline SLATE, we use the same CNN backbone as our model for a fair comparison.
Revised: For the baseline SLATE, we use the same CNN backbone as our model for a fair comparison.

Original: In line with the previous works, for
Revised: In line with the previous works, for

Original: SAVi and OP3, we take their decoding masks to be their predicted segmentation.
Revised: SAVi and OP3, we take their decoding masks to be their predicted segmentation.

Original: Since our model
Revised: Since our model

Original: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decoding masks.
Revised: STEVE and the baseline SLATE are based on transformer decoder, they do not provide decoding masks.

Original: Therefore, we use their input attention masks as our predicted segmentation.
Revised: Therefore, we use their input attention masks as our predicted segmentation.

Original: Note that this
Revised: Note that this

Original: 0 1 2 3 4 5 6     # Past Frames
Revised: 0 1 2 3 4 5 6     # Past Frames

Original: I m a g e
Revised: I m a g e

Original: F G - A R I ( i n % )
Revised: F G - A R I ( i n % )

Original: CATER
Revised: CATER

Original: STEVESLATESAViOP 0 1 2 3 4 5 6     # Past Frames
Revised: STEVESLATESAViOP 0 1 2 3 4 5 6     # Past Frames

Original: CATERTex 0 1 2 3 4 5 6     # Past Frames
Revised: CATERTex 0 1 2 3 4 5 6     # Past Frames

Original: MOVi-Solid 0 1 2 3 4 5 6     # Past Frames MOVi-Tex 0 1 2 3 4 5 6     # Past Frames
Revised: MOVi-Solid 0 1 2 3 4 5 6     # Past Frames MOVi-Tex 0 1 2 3 4 5 6     # Past Frames

Original: MOVi-D 0 1 2 3 4 5 6     # Past Frames
Revised: MOVi-D 0 1 2 3 4 5 6     # Past Frames

Original: MOVi-E difference gives an advantage to the baselines but not to our model as in section 5.3, our analysis will show that the decoding masks perform better than the input attention masks.
Revised: MOVi-E difference gives an advantage to the baselines but not to our model as in section 5.3, our analysis will show that the decoding masks perform better than the input attention masks.

Original: 5.1 Unsupervised Image Segmentation
Revised: 5.1 Unsupervised Image Segmentation

Original: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.
Revised: In this section, we evaluate how effective our model is for unsupervised image segmentation in videos.

Original: We report Image FG-ARI which measures how well the predicted segmentation matches the ground truth segmentation of a given single image.
Revised: We report Image FG-ARI which measures how well the predicted segmentation matches the ground truth segmentation of a given single image.

Original: In line with the previous works (Locatello et al., 2020;
Revised: In line with the previous works (Locatello et al., 2020;

Original: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.
Revised: Kabra et al., 2021; Kipf et al., 2021), we consider only the foreground segments of the ground truth.

Original: In Figure 2, we plot the Image FG-ARI of per-frame segmentation.
Revised: In Figure 2, we plot the Image FG-ARI of per-frame segmentation.

Original: Benefit over Mixture-based Methods.
Revised: Benefit over Mixture-based Methods.

Original: Comparing STEVE with SAVi and OP3, we see that STEVE achieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.
Revised: Comparing STEVE with SAVi and OP3, we see that STEVE achieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets.

Original: All of these 5 datasets are characterized by complex textures.
Revised: All of these 5 datasets are characterized by complex textures.

Original: Therefore, our superior performance shows that our model is more effective in dealing with textured scenes compared to the baselines.
Revised: Therefore, our superior performance shows that our model is more effective in dealing with textured scenes compared to the baselines.

Original: Because the main difference of STEVE with SAVi is the reconstruction approach, this result provides strong evidence that the transformer-based reconstruction is the key driver of this improvement.
Revised: Because the main difference of STEVE with SAVi is the reconstruction approach, this result provides strong evidence that the transformer-based reconstruction is the key driver of this improvement.

Original: Analyzing the qualitative results in Figures 4 (left), 5 and 9 reveals that the baselines fail completely in CATERTex,
Revised: Analyzing the qualitative results in Figures 4 (left), 5 and 9 reveals that the baselines fail completely in CATERTex,

Original: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regions instead of meaningful object regions.
Revised: MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regions instead of meaningful object regions.

Original: Unlike simple datasets used in prior works such as CLEVR or CATER that provide strong color contrast between objects, our datasets such as CATERTex and
Revised: Unlike simple datasets used in prior works such as CLEVR or CATER that provide strong color contrast between objects, our datasets such as CATERTex and

Original: MOVi-Tex provide no such color cues.
Revised: MOVi-Tex provide no such color cues.

Original: Despite this, our model can remarkably discover the objects.
Revised: Despite this, our model can remarkably discover the objects.

Original: With the state-of-the-art baselines completely failing, it is for the first time in this line of research that such complex datasets have been effectively handled.
Revised: With the state-of-the-art baselines completely failing, it is for the first time in this line of research that such complex datasets have been effectively handled.

Original: In one dataset i.e. CATER, our performance is comparable to OP3 and slightly worse than SAVi.
Revised: In one dataset i.e. CATER, our performance is comparable to OP3 and slightly worse than SAVi.

Original: As CATER does not have complex textures, it is not surprising to see baselines performing well on this dataset.
Revised: As CATER does not have complex textures, it is not surprising to see baselines performing well on this dataset.

Original: In Figure 2, we also find that as the models see more frames, the segmentation tends to improve.
Revised: In Figure 2, we also find that as the models see more frames, the segmentation tends to improve.

Original: Benefit of Temporal Learning.
Revised: Benefit of Temporal Learning.

Original: We also analyze whether training STEVE on videos has any benefit in image segmentation compared to SLATE.
Revised: We also analyze whether training STEVE on videos has any benefit in image segmentation compared to SLATE.

Original: As SLATE is applicable only for static images, we train and test it on one frame ‘videos’ of our datasets.
Revised: As SLATE is applicable only for static images, we train and test it on one frame ‘videos’ of our datasets.

Original: Comparing STEVE with SLATE, we note that STEVE performs consistently better in all datasets.
Revised: Comparing STEVE with SLATE, we note that STEVE performs consistently better in all datasets.

Original: Noteworthy is the gap in MOVi-Tex which is especially large.
Revised: Noteworthy is the gap in MOVi-Tex which is especially large.

Original: This suggests that training on temporal data i.e. videos can be helpful for learning to segment.
Revised: This suggests that training on temporal data i.e. videos can be helpful for learning to segment.

Original: We conjecture that due to the similar texture in the background and foreground, inferring correct object regions from a static image can be harder than inferring them when the objects are moving.
Revised: We conjecture that due to the similar texture in the background and foreground, inferring correct object regions from a static image can be harder than inferring them when the objects are moving.

Original: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to the other datasets.
Revised: This may explain the significantly larger gap with SLATE in MOVi-Tex compared to the other datasets.

Original: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. a static image), indicating that training on temporal data helps STEVE infer segments on static images.
Revised: Note that this gap exists even when evaluating STEVE with zero past frames (i.e. a static image), indicating that training on temporal data helps STEVE infer segments on static images.

Original: 5.2 Unsupervised Video Segmentation
Revised: 5.2 Unsupervised Video Segmentation

Original: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentation in both space and time without switching identities.
Revised: In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentation in both space and time without switching identities.

Original: For this, we compute Video FG-ARI which considers the full trajectory of a segment over the video length as one cluster.
Revised: For this, we compute Video FG-ARI which considers the full trajectory of a segment over the video length as one cluster.

Original: In line with the previous
Revised: In line with the previous

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: OursSAViOP

Revised: We show the frames and their true masks followed by foreground and background segmentation of each

Revised: We separated the foreground to make the result easier to interpret by applying a threshold on the segment area

Original: OursSAViOP of our model on IID and OOD test sets.
Revised: See the supplementary and We plot the Video FG-ARI. In (a) and (b), we compare the performance of our model on IID and OOD test sets.

Original: In (c), we evaluate our performance in unsupervised video segmentation on natural scenes.
Revised: In (c), we evaluate our performance in unsupervised video segmentation on natural scenes.

Original: works Locatello et al.
Revised: works Locatello et al.

Original: ; Kabra et al.
Revised: ; Kabra et al.

Original: (2021); Kipf et al.
Revised: (2021); Kipf et al.

Original: (2021), we consider only the foreground segments of the ground truth.
Revised: (2021), we consider only the foreground segments of the ground truth.

Original: Video Segmentation Performance.
Revised: Video Segmentation Performance.

Original: In Figure 3, we see that in 5 out of 6 datasets, all of which are textured, STEVE significantly outperforms the baselines.
Revised: In Figure 3, we see that in 5 out of 6 datasets, all of which are textured, STEVE significantly outperforms the baselines.

Original: In CATER, the baselines are able to perform comparably or slightly better than ours.
Revised: In CATER, the baselines are able to perform comparably or slightly better than ours.

Original: As with the segmentation result of Section 5.1, this is not surprising given that CATER is visually simple enough for the baselines to handle well.
Revised: As with the segmentation result of Section 5.1, this is not surprising given that CATER is visually simple enough for the baselines to handle well.

Original: We also observe the one major drawback of SLATE in these results i.e. SLATE cannot provide aligned slot representations for a video.
Revised: We also observe the one major drawback of SLATE in these results i.e. SLATE cannot provide aligned slot representations for a video.

Original: This is because SLATE can only be applied to individual frames and the slots of each frame would be randomly permuted.
Revised: This is because SLATE can only be applied to individual frames and the slots of each frame would be randomly permuted.

Original: We also show how the performance is affected by the length of the video.
Revised: We also show how the performance is affected by the length of the video.

Original: The models were trained on 3-length videos (6-length for CATER) while we evaluate on up to 24-length videos.
Revised: The models were trained on 3-length videos (6-length for CATER) while we evaluate on up to 24-length videos.

Original: As the likelihood of switching identities should increase with the video length, all models show some downward slope.
Revised: As the likelihood of switching identities should increase with the video length, all models show some downward slope.

Original: Going from video length 6 to 24 in
Revised: Going from video length 6 to 24 in

Original: CATERTex, MOVi-D, and MOVi-E, our model deteriorates noticeably less than the baselines.
Revised: CATERTex, MOVi-D, and MOVi-E, our model deteriorates noticeably less than the baselines.

Original: Out-of-Distribution Generalization.
Revised: Out-of-Distribution Generalization.

Original: In Figure 4 (a) and (b), we show how well our trained model can generalize at test time to out-of-distribution videos which have more number of objects and unseen objects and materials.
Revised: In Figure 4 (a) and (b), we show how well our trained model can generalize at test time to out-of-distribution videos which have more number of objects and unseen objects and materials.

Original: We see that our model generalizes well and retains a similar performance as that on the IID test set.
Revised: We see that our model generalizes well and retains a similar performance as that on the IID test set.

Original: One exception is MOVi-Tex in which our model generalizes well to more objects but suffers on novel textures.
Revised: One exception is MOVi-Tex in which our model generalizes well to more objects but suffers on novel textures.

Original: Due to the modular structure of the recurrent encoder, each slot binds to input objects quite independently.
Revised: Due to the modular structure of the recurrent encoder, each slot binds to input objects quite independently.

Original: We conjecture that this modularity is responsible for good generalization to more objects.
Revised: We conjecture that this modularity is responsible for good generalization to more objects.

Original: However, for new materials, generalization depends on how well the backbone CNN can generalize.
Revised: However, for new materials, generalization depends on how well the backbone CNN can generalize.

Original: As the training set of MOVi-Tex was generated using a library oftextures, it would not be surprising if this CNN provides poor features of unseen textures.
Revised: As the training set of MOVi-Tex was generated using a library oftextures, it would not be surprising if this CNN provides poor features of unseen textures.

Original: This, we believe, is affecting the performance of our model.
Revised: This, we believe, is affecting the performance of our model.

Original: Natural Scenes.
Revised: Natural Scenes.

Original: In Figure 4 (c), we evaluate video segmentation on two natural datasets taken from real-world videos: Traffic and Aquarium.
Revised: In Figure 4 (c), we evaluate video segmentation on two natural datasets taken from real-world videos: Traffic and Aquarium.

Original: We see that our model performs significantly better than the baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI in the Aquarium dataset.
Revised: We see that our model performs significantly better than the baselines, approximately doubling the FG-ARI in the Traffic dataset and tripling the FG-ARI in the Aquarium dataset.

Original: Noteworthy is the visual complexity of the Aquarium dataset in which the color and texture of fish are strongly camouflaged against the background and seeing them requires careful inspection even for a human (see Figure 5).
Revised: Noteworthy is the visual complexity of the Aquarium dataset in which the color and texture of fish are strongly camouflaged against the background and seeing them requires careful inspection even for a human (see Figure 5).

Original: Despite this, we note that our model is able to deal with this challenge effectively.
Revised: Despite this, we note that our model is able to deal with this challenge effectively.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: Diagnosing Slot Representation in STEVE.
Revised: Diagnosing Slot Representation in STEVE.

Original: In previous sections, we focused on segmentation quality and used input attention masks of STEVE.
Revised: In previous sections, we focused on segmentation quality and used input attention masks of STEVE.

Original: However, input attention masks do not provide a direct indication of the information content of slots.
Revised: However, input attention masks do not provide a direct indication of the information content of slots.

Original: We would like to diagnose how well the slots represent the object-centric structure of the scene.
Revised: We would like to diagnose how well the slots represent the object-centric structure of the scene.

Original: For this, we take the slots from the pre-trained encoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-based decoder to reconstruct the observation from the slots.
Revised: For this, we take the slots from the pre-trained encoder of STEVE and, without propagating the gradient to the encoder, we train a mixture-based decoder to reconstruct the observation from the slots.

Original: We call this model STEVE-Diagnostic .
Revised: We call this model STEVE-Diagnostic .

Original: In
Revised: In

Original: Figure 6 (a), we compare the performance of unsupervised video segmentation with the baselines using the decoding masks of STEVE-Diagnostic.
Revised: Figure 6 (a), we compare the performance of unsupervised video segmentation with the baselines using the decoding masks of STEVE-Diagnostic.

Original: The decoder of STEVE-diagnostic is made to be exactly the same as that of SAVi for a fair comparison.
Revised: The decoder of STEVE-diagnostic is made to be exactly the same as that of SAVi for a fair comparison.

Original: We find that STEVE-Diagnostic performs better than the baselines in all textured datasets, in line with our earlier results in Figures 2 and 3.
Revised: We find that STEVE-Diagnostic performs better than the baselines in all textured datasets, in line with our earlier results in Figures 2 and 3.

Original: This shows that the slots in STEVE represent the object-centric structure better than the baselines.
Revised: This shows that the slots in STEVE represent the object-centric structure better than the baselines.

Original: In
Revised: In

Original:  Table 9 in Appendix C, we further analyze the slot contents by training a linear regressor to predict the object positions and find that STEVE performs favorably relative to the baselines.[Figure]
Revised: [Figure] [Figure] Table 9 in Appendix C, we further analyze the slot contents by training a linear regressor to predict the object positions and find that STEVE performs favorably relative to the baselines.

Original: [Figure]

Original: Effect of Granularity of dVAE on STEVE.
Revised: Effect of Granularity of dVAE on STEVE.

Original: In this section, we analyze the effect of granularity of discrete VAE on the segmentation performance of STEVE.
Revised: In this section, we analyze the effect of granularity of discrete VAE on the segmentation performance of STEVE.

Original: In the default configuration of our model, the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patch with one discrete token.
Revised: In the default configuration of our model, the discrete VAE divides the image uniformly into patches of size 4 × 4 and represents each patch with one discrete token.

Original: That is, an image of size 128 × 128 would be represented by a sequence of length 1024.
Revised: That is, an image of size 128 × 128 would be represented by a sequence of length 1024.

Original: In this analysis, we test the effect on segmentation performance if dVAE represents the image using larger patches i.e. 16 × 16 and 32 × 32 .
Revised: In this analysis, we test the effect on segmentation performance if dVAE represents the image using larger patches i.e. 16 × 16 and 32 × 32 .

Original: In our results shown in Figure 6, we note that using larger patches results in a worsening of the performance.
Revised: In our results shown in Figure 6, we note that using larger patches results in a worsening of the performance.

Original: This suggests a possible scaling law that smaller patches lead to better performance.
Revised: This suggests a possible scaling law that smaller patches lead to better performance.

Original: However, due to the prohibitive memory needs of the transformer on longer sequences, we were not able to experiment with smaller patches.
Revised: However, due to the prohibitive memory needs of the transformer on longer sequences, we were not able to experiment with smaller patches.

Original: Effect of Transformer Capacity on STEVE
Revised: Effect of Transformer Capacity on STEVE.

Original: In Fig.
Revised: In Fig.

Original: 7 in the appendix, we analyze whether our segmentation performance is sensitive to the capacity of the decoder.
Revised: 7 in the appendix, we analyze whether our segmentation performance is sensitive to the capacity of the decoder.

Original: For this, we test the effect of increasing the number of layers in the transformer decoder.
Revised: For this, we test the effect of increasing the number of layers in the transformer decoder.

Original: We find that our segmentation performance is unaffected by this change, showing that our model is robust.
Revised: We find that our segmentation performance is unaffected by this change, showing that our model is robust.

Original: Comparison between Decoding Masks and Input Attention Masks in SAVi.
Revised: Comparison between Decoding Masks and Input Attention Masks in SAVi.

Original: In previous sections on unsupervised image and video decomposition, we used input attention masks of STEVE while for the baselines, we used the decoding masks.
Revised: In previous sections on unsupervised image and video decomposition, we used input attention masks of STEVE while for the baselines, we used the decoding masks.

Original: We would like to test how different are the performances of input attention masks and decoding masks.
Revised: We would like to test how different are the performances of input attention masks and decoding masks.

Original: While OP3 only provides decoding masks, fortunately,
Revised: While OP3 only provides decoding masks, fortunately,

Original: SAVi provides both input masks and decoding masks which we compare in Figure 6 (c).
Revised: SAVi provides both input masks and decoding masks which we compare in Figure 6 (c).

Original: We find that input attention masks tend to be worse than decoding masks of SAVi.
Revised: We find that input attention masks tend to be worse than decoding masks of SAVi.

Original: Therefore, the attention masks of STEVE outperform both the input and decoding masks of the baseline SAVi.
Revised: Therefore, the attention masks of STEVE outperform both the input and decoding masks of the baseline SAVi.

Original: Mixture-based decoding without Spatial Broadcast.
Revised: Mixture-based decoding without Spatial Broadcast.

Original: Mixture-based decoding requires a CNN to decode the object image and the mask from each slot.
Revised: Mixture-based decoding requires a CNN to decode the object image and the mask from each slot.

Original: To implement this CNN, the baselines prescribe using a Spatial Broadcast decoder, a type of CNN decoder that limits the expressiveness of the decoder to facilitate the discovery of objects.
Revised: To implement this CNN, the baselines prescribe using a Spatial Broadcast decoder, a type of CNN decoder that limits the expressiveness of the decoder to facilitate the discovery of objects.

Original: We would like to test if a more flexible CNN implemented completely using transposed convolutions would be enough to improve the performance or not.
Revised: We would like to test if a more flexible CNN implemented completely using transposed convolutions would be enough to improve the performance or not.

Original: In Figure 6 (d), our results on the 5 textured datasets show that this change leads to a similar or slightly higher performance than the original decoder.
Revised: In Figure 6 (d), our results on the 5 textured datasets show that this change leads to a similar or slightly higher performance than the original decoder.

Original: However, the model still fails by splitting the image into fixed patches similarly to before and still performs significantly worse than our model.
Revised: However, the model still fails by splitting the image into fixed patches similarly to before and still performs significantly worse than our model.

Original: This shows that simply using a more flexible CNN with mixture-based decoding is not enough and the expressiveness of an auto-regressive transformer is important for achieving significant gains.
Revised: This shows that simply using a more flexible CNN with mixture-based decoding is not enough and the expressiveness of an auto-regressive transformer is important for achieving significant gains.

Original: Computational Requirements of STEVE.
Revised: Computational Requirements of STEVE.

Original: In Table 6 in the Appendix, we empirically compare the computational requirements of STEVE and SAVi.
Revised: In Table 6 in the Appendix, we empirically compare the computational requirements of STEVE and SAVi.

Original: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.
Revised: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.

Original: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .
Revised: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .

Original: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.
Revised: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.

Original: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E, unlike ours.
Revised: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E, unlike ours.

Original: We also provide a big-
Revised: We also provide a big-

Original: O analysis of the memory costs in Appendix B.1.
Revised: O analysis of the memory costs in Appendix B.1.

Original: Effect of Camera Motion.
Revised: Effect of Camera Motion.

Original: Our segmentation results on MOVi-D and MOVi-E in Figures 2 andprovide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camera to MOVi-D. We observe that camera motion in MOVi-E leads to a slight improvement in performance over MOVi-D.
Revised: Our segmentation results on MOVi-D and MOVi-E in Figures 2 andprovide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camera to MOVi-D. We observe that camera motion in MOVi-E leads to a slight improvement in performance over MOVi-D.

Original: We conjecture that this is because a moving camera causes relative motion between objects and can help better in bringing out patterns that tend to remain stable under motion.
Revised: We conjecture that this is because a moving camera causes relative motion between objects and can help better in bringing out patterns that tend to remain stable under motion.

Original: Static Objects.
Revised: Static Objects.

Original: We also note that our model is effective even when static objects are present in the scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.
Revised: We also note that our model is effective even when static objects are present in the scene such as in CATER, CATERTex, MOVi-D, and MOVi-E.

Original: In methods that rely purely on optical flow for discovering objects can suffer significantly with static objects.
Revised: In methods that rely purely on optical flow for discovering objects can suffer significantly with static objects.

Original: For MOVi-D and
Revised: For MOVi-D and

Original: MOVi-E, Greff et al.
Revised: MOVi-E, Greff et al.

Original: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flow supervision.
Revised: (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flow supervision.

Original: Compared to this, our model achieves a significantly better 47.67 and 52.15 on these datasets, respectively, despite the static objects and without using any supervision whatsoever.
Revised: Compared to this, our model achieves a significantly better 47.67 and 52.15 on these datasets, respectively, despite the static objects and without using any supervision whatsoever.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervised object-centric video learning.
Revised: In this paper, we study the effect of a powerful transformer-based slot decoder for unsupervised object-centric video learning.

Original: For this, we propose a simple model, STEVE, and demonstrate that it can deal with various complex and naturalistic videos without any supervision by significantly outperforming previous state-of-the-art baseline models.
Revised: For this, we propose a simple model, STEVE, and demonstrate that it can deal with various complex and naturalistic videos without any supervision by significantly outperforming previous state-of-the-art baseline models.

Original: In experiments, we also find that although the slot-transformer decoder alone is an important component, for videos it is essential to learn also from temporal information for dealing with complex videos.
Revised: In experiments, we also find that although the slot-transformer decoder alone is an important component, for videos it is essential to learn also from temporal information for dealing with complex videos.

Original: The results of this paper and the
Revised: The results of this paper and the

Original: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centric learning which is contrary to the traditional view advocating the use of weak mixture decoders.
Revised: SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centric learning which is contrary to the traditional view advocating the use of weak mixture decoders.

Original: This raises a question to the community: is accurate reconstruction from slots all we need to capture the objectness?
Revised: This raises a question to the community: is accurate reconstruction from slots all we need to capture the objectness?

Original: In the future, it seems worth investigating further in this direction.
Revised: In the future, it seems worth investigating further in this direction.

Original: Also, because we choose to investigate a minimal architecture in this paper, it would be interesting to see what gains we could obtain additionally by exploring more advanced architectures.
Revised: Also, because we choose to investigate a minimal architecture in this paper, it would be interesting to see what gains we could obtain additionally by exploring more advanced architectures.

Original: Lastly, it would be interesting to investigate applying this method to large-scale datasets.
Revised: Lastly, it would be interesting to investigate applying this method to large-scale datasets.

Original: We further elaborate on these in Appendix F.
Revised: We further elaborate on these in Appendix F.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work is supported by Brain Pool Plus (BP+)
Revised: This work is supported by Brain Pool Plus (BP+)

Original: Program (No. 2021H1D3A2A03103645) and
Revised: Program (No. 2021H1D3A2A03103645) and

Original: Young Researcher Program (No. 2022R1C1C1009443) through the National Research Foundation of
Revised: Young Researcher Program (No. 2022R1C1C1009443) through the National Research Foundation of

Original: Korea (NRF) funded by the Ministry of Science and ICT.
Revised: Korea (NRF) funded by the Ministry of Science and ICT.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] In supplementary material, we provide the architectural details and the hyperparameters used for our experiments.
Revised: [Yes] In supplementary material, we provide the architectural details and the hyperparameters used for our experiments.

Original: We will share the code and our proposed datasets at https://sites.google.com/view/slot-transformer-for-videos .
Revised: We will share the code and our proposed datasets at https://sites.google.com/view/slot-transformer-for-videos .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide the amount of resources used in Section B.1 in Appendix. 4.
Revised: [Yes] We provide the amount of resources used in Section B.1 in Appendix. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We will share them upon acceptance.
Revised: [Yes] We will share them upon acceptance.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: Dataset Descriptions.
Revised: Dataset Descriptions.

Original: We generated MOVi-Solid dataset using Kubric (Greff et al., 2022) which introduces textured background and more complex shapes to the MOVi dataset Kipf et al.
Revised: We generated MOVi-Solid dataset using Kubric (Greff et al., 2022) which introduces textured background and more complex shapes to the MOVi dataset Kipf et al.

Original: For this, we used 100 HDR backgrounds and Kubric’s KuBasic shapes.
Revised: For this, we used 100 HDR backgrounds and Kubric’s KuBasic shapes.

Original: We also generate two test sets: one which has the same number of objects as the training set (3-10 objects) and other which has higher out-of-distribution number of objects (11-12 objects).
Revised: We also generate two test sets: one which has the same number of objects as the training set (3-10 objects) and other which has higher out-of-distribution number of objects (11-12 objects).

Original: Similarly to MOVi-Solid, for MOVi-Tex, we use Kubric to generate the dataset.
Revised: Similarly to MOVi-Solid, for MOVi-Tex, we use Kubric to generate the dataset.

Original: In MOVi-Tex, we modify our MOVi-Solid dataset to have complex textures applied to the objects taken from the Describable Textures dataset Cimpoi et al.
Revised: In MOVi-Tex, we modify our MOVi-Solid dataset to have complex textures applied to the objects taken from the Describable Textures dataset Cimpoi et al.

Original: Describable Textures dataset provides several categories of textures.
Revised: Describable Textures dataset provides several categories of textures.

Original: We used 240 textures each for generating the training set.
Revised: We used 240 textures each for generating the training set.

Original: For this, we used 120 textures that are from the banded category andtextures that are from the chequered category.
Revised: For this, we used 120 textures that are from the banded category andtextures that are from the chequered category.

Original: We also generated two out-of-distribution test sets: one which has the same number of objects as the training set (3-10 objects) but never-before-seen textures and the other which as the same textures but higher number of objects (11-12 objects).
Revised: We also generated two out-of-distribution test sets: one which has the same number of objects as the training set (3-10 objects) but never-before-seen textures and the other which as the same textures but higher number of objects (11-12 objects).

Original: For the out-of-distribution test set with never-before-seen textures, 120 textures were taken from the grid category and 120 were taken from the grooved category of the Describable Textures dataset.
Revised: For the out-of-distribution test set with never-before-seen textures, 120 textures were taken from the grid category and 120 were taken from the grooved category of the Describable Textures dataset.

Original: For
Revised: For

Original: CATER, MOVi-D and MOVi-E, we used the standard training and test splits provided in the respective releases of CATER (with masks) and MOVi datasets.
Revised: CATER, MOVi-D and MOVi-E, we used the standard training and test splits provided in the respective releases of CATER (with masks) and MOVi datasets.

Original: We generated CATERTex by integrating the codes for CATER and CLEVRTex (Girdhar & Ramanan, 2020; Karazija et al., 2021).
Revised: We generated CATERTex by integrating the codes for CATER and CLEVRTex (Girdhar & Ramanan, 2020; Karazija et al., 2021).

Original: We generated two sets: the training and the IID test sets which contain 3-8 objects and an out-of-distribution test set which contains 9-10 objects.
Revised: We generated two sets: the training and the IID test sets which contain 3-8 objects and an out-of-distribution test set which contains 9-10 objects.

Original: We collected the Youtube datasets by downloading a 6-hour long video streams, center-cropping and resizing them.
Revised: We collected the Youtube datasets by downloading a 6-hour long video streams, center-cropping and resizing them.

Original: The Youtube-Traffic dataset was generated from a video feed collected by a traffic camera that watches a fixed intersection.
Revised: The Youtube-Traffic dataset was generated from a video feed collected by a traffic camera that watches a fixed intersection.

Original: The camera does not move during the video.
Revised: The camera does not move during the video.

Original: The lighting conditions change slightly over time and this has a slight effect on the background and the overall scene from one clip to another.
Revised: The lighting conditions change slightly over time and this has a slight effect on the background and the overall scene from one clip to another.

Original: The vehicles appear to show diversity in size, color, and type.
Revised: The vehicles appear to show diversity in size, color, and type.

Original: The Youtube-Aquarium dataset was generated from a video collected by a static camera mounted inside a fish exhibit.
Revised: The Youtube-Aquarium dataset was generated from a video collected by a static camera mounted inside a fish exhibit.

Original: The background, lighting conditions, and other characteristics remain stationary throughout the video.
Revised: The background, lighting conditions, and other characteristics remain stationary throughout the video.

Original: The size, color, and types of fish show some diversity.
Revised: The size, color, and types of fish show some diversity.

Original: As these were collected from Youtube and did not have ground truth masks, we manually annotatedvideos of the test set.
Revised: As these were collected from Youtube and did not have ground truth masks, we manually annotatedvideos of the test set.

Original: We will release the training and the test splits of all the datasets.
Revised: We will release the training and the test splits of all the datasets.

Original: B STEVE: Architecture Details
Revised: B STEVE: Architecture Details

Original: Recurrent Slot Encoder.
Revised: Recurrent Slot Encoder.

Original: The design of our slot-based recurrent encoder f slot-rnn ϕ is inspired by Kipf et al.
Revised: The design of our slot-based recurrent encoder f slot-rnn ϕ is inspired by Kipf et al.

Original: At the start of an episode, we provide initial slots s 0 by randomly sampling from a
Revised: At the start of an episode, we provide initial slots s 0 by randomly sampling from a

Original: Gaussian distribution with learned mean and variance.
Revised: Gaussian distribution with learned mean and variance.

Original: [Equation]
Revised: [Equation]

Original: where µ and σ are learned parameters.
Revised: where µ and σ are learned parameters.

Original: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.
Revised: For each input frame x t , we compute an encoding in the form of a feature map using a backbone CNN.

Original: The architecture of this CNN is described in Table 2.
Revised: The architecture of this CNN is described in Table 2.

Original: The resulting feature map has size H enc × W enc × D enc .
Revised: The resulting feature map has size H enc × W enc × D enc .

Original: It is then flattened on spatial dimensions to obtain
Revised: It is then flattened on spatial dimensions to obtain

Original: [Table]
Revised: [Table]

Original: a set of H enc · W enc features.
Revised: a set of H enc · W enc features.

Original: A positional embedding p is added to these followed by a 2-layer MLP.
Revised: A positional embedding p is added to these followed by a 2-layer MLP.

Original: With these, we obtain a set of input features e t = { e t, 1 , e t, 2 , . . .
Revised: With these, we obtain a set of input features e t = { e t, 1 , e t, 2 , . . .

Original: , e t,HW } .
Revised: , e t,HW } .

Original: The positional embedding is computed by applying a linear map to the 4-dimensional spatial position as introduced by Locatello et al.
Revised: The positional embedding is computed by applying a linear map to the 4-dimensional spatial position as introduced by Locatello et al.

Original: (2020) and Kipf et al.
Revised: (2020) and Kipf et al.

Original: Next, the slots of the previous time-step s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .
Revised: Next, the slots of the previous time-step s t − 1 perform attention on the features e t and use the attention result to update the slots to s t .

Original: For this, the slots in s t − 1 first compute the attention weights A t
Revised: For this, the slots in s t − 1 first compute the attention weights A t

Original: = {A t, 1 , . . .
Revised: = {A t, 1 , . . .

Original: , A t,N } over the input features.
Revised: , A t,N } over the input features.

Original: Using the attention weights, the attention result r t,n is computed for each slot via an attention-weighted sum of input features.
Revised: Using the attention weights, the attention result r t,n is computed for each slot via an attention-weighted sum of input features.

Original: [Equation]
Revised: [Equation]

Original: where q , k and v are linear projections and D is the output size of the projections.
Revised: where q , k and v are linear projections and D is the output size of the projections.

Original: The attention result r t,n is then used to update the respective slot: ˜ s t,n ← f RNN ( r t,n , s t − 1 ,n ) where the update function f
Revised: The attention result r t,n is then used to update the respective slot: ˜ s t,n ← f RNN ( r t,n , s t − 1 ,n ) where the update function f

Original: RNN is implemented as a GRU (Cho et al., 2014).
Revised: RNN is implemented as a GRU (Cho et al., 2014).

Original: For additional expressiveness, a 2-layer MLP (with residual connection) and LayerNorm is applied on the output of the GRU.
Revised: For additional expressiveness, a 2-layer MLP (with residual connection) and LayerNorm is applied on the output of the GRU.

Original: In practice, these attention and update steps can be executed multiple times per frame.
Revised: In practice, these attention and update steps can be executed multiple times per frame.

Original: We use 2 iterations per frame in our experiments.
Revised: We use 2 iterations per frame in our experiments.

Original: Lastly, the slots are made to interact using an interaction function s t ← f interact ϕ
Revised: Lastly, the slots are made to interact using an interaction function s t ← f interact ϕ

Original: [Equation]
Revised: [Equation]

Original: implemented using a 1-layer transformer with 4 heads.
Revised: implemented using a 1-layer transformer with 4 heads.

Original: Following Kipf et al.
Revised: Following Kipf et al.

Original: (2021), we use the slots ˜ s t before the interaction step for reconstruction.
Revised: (2021), we use the slots ˜ s t before the interaction step for reconstruction.

Original: For fairness of comparison between our model
Revised: For fairness of comparison between our model

Original: STEVE and SAVi, we use the same implementation of this backbone recurrent slot encoder in both models.
Revised: STEVE and SAVi, we use the same implementation of this backbone recurrent slot encoder in both models.

Original: Discrete VAE.
Revised: Discrete VAE.

Original: The implementation of our discrete VAE is based on that proposed by (Singh et al., 2022).
Revised: The implementation of our discrete VAE is based on that proposed by (Singh et al., 2022).

Original: It consists of an encoder network f dVAE ϕ and a decoder network g dVAE θ .
Revised: It consists of an encoder network f dVAE ϕ and a decoder network g dVAE θ .

Original: The encoder network essentially divides the whole image x t into patches of size 4 × 4 and encodes each patch as one discrete token belonging to a vocabulary V .
Revised: The encoder network essentially divides the whole image x t into patches of size 4 × 4 and encodes each patch as one discrete token belonging to a vocabulary V .

Original: We take the size of the vocabulary to be 4096 in our experiments.
Revised: We take the size of the vocabulary to be 4096 in our experiments.

Original: This is achieved in parallel for all patches applying a convolutional neural network on the image of size H × W × 3 as follows.
Revised: This is achieved in parallel for all patches applying a convolutional neural network on the image of size H × W × 3 as follows.

Original: Layer
Revised: Layer

Original: Kernel Size
Revised: Kernel Size

Original: Stride
Revised: Stride

Original: Padding
Revised: Padding

Original: Channels
Revised: Channels

Original: Activation
Revised: Activation

Original: Conv 4 ×
Revised: Conv 4 ×

Original: ReLU
Revised: ReLU

Original: Conv 1 ×
Revised: Conv 1 ×

Original: [Table]
Revised: [Table]

Original: This results in a feature map of size ( H/ 4) ×
Revised: This results in a feature map of size ( H/ 4) ×

Original: ( W/ 4) × |V| .
Revised: ( W/ 4) × |V| .

Original: Each cell i ∈ 1 , . . .
Revised: Each cell i ∈ 1 , . . .

Original: , L (where L = HW/ 16 ) of this feature map can be seen as a vector o dVAE t,i containing log-probabilities over |V| vocabulary tokens.
Revised: , L (where L = HW/ 16 ) of this feature map can be seen as a vector o dVAE t,i containing log-probabilities over |V| vocabulary tokens.

Original: For each cell i , a discrete token is sampled by constructing a categorical distribution using these log-probabilities.
Revised: For each cell i , a discrete token is sampled by constructing a categorical distribution using these log-probabilities.

Original: [Equation]
Revised: [Equation]

Original: In the discrete VAE decoder, the input is a discrete feature map of size ( H/ 4) ×
Revised: In the discrete VAE decoder, the input is a discrete feature map of size ( H/ 4) ×

Original: ( W/ 4) × |V| whose each cell is a one-hot vector
Revised: ( W/ 4) × |V| whose each cell is a one-hot vector

Original: z i and the decoder tries to reconstruct the original image from this discrete feature map input.
Revised: z i and the decoder tries to reconstruct the original image from this discrete feature map input.

Original: This mapping is implemented using a convolutional neural network as described below, finally outputting an image of size H × W × 3 .
Revised: This mapping is implemented using a convolutional neural network as described below, finally outputting an image of size H × W × 3 .

Original: Finally, the whole network is trained by minimizing a image reconstruction loss implemented using squared-error.
Revised: Finally, the whole network is trained by minimizing a image reconstruction loss implemented using squared-error.

Original: [Equation]
Revised: [Equation]

Original: As there is a discrete sampling step required, we use the Gumbel-Softmax relaxation during training while we use hard sampling for generating discrete targets for the transformer, following Singh et al.
Revised: As there is a discrete sampling step required, we use the Gumbel-Softmax relaxation during training while we use hard sampling for generating discrete targets for the transformer, following Singh et al.

Original: The temperature parameter for the Gumbel-Softmax is decayed from 1.0 to 0.1 in the first 30K iterations of training as prescribed by Singh et al.
Revised: The temperature parameter for the Gumbel-Softmax is decayed from 1.0 to 0.1 in the first 30K iterations of training as prescribed by Singh et al.

Original: Transformer Decoder.
Revised: Transformer Decoder.

Original: We use the standard design of transformer as introduced by Vaswani et al.
Revised: We use the standard design of transformer as introduced by Vaswani et al.

Original: (2017) and adopted by Singh et al.
Revised: (2017) and adopted by Singh et al.

Original: Like SLATE, we use learned positional embeddings for each position i ∈ { 1 , . . .
Revised: Like SLATE, we use learned positional embeddings for each position i ∈ { 1 , . . .

Original: , L } in the sequence.
Revised: , L } in the sequence.

Original: [Table]
Revised: [Table]

Original: Channels
Revised: Channels

Original: [Table] [Table]
Revised: [Table] [Table]

Original: During training, the transformer decoder takes the slots and the dVAE tokens as input and learns to predict the next token.
Revised: During training, the transformer decoder takes the slots and the dVAE tokens as input and learns to predict the next token.

Original: We learn to do this next-token prediction for all tokens in parallel by applying causal masking in the style of GPT.
Revised: We learn to do this next-token prediction for all tokens in parallel by applying causal masking in the style of GPT.

Original: In further detail, the dVAE tokens given as input to the transformer are obtained via argmax on the logits output by the dVAE encoder.
Revised: In further detail, the dVAE tokens given as input to the transformer are obtained via argmax on the logits output by the dVAE encoder.

Original: That is, we take the dVAE logits
Revised: That is, we take the dVAE logits

Original: o dVAE t,i and first perform argmax.
Revised: o dVAE t,i and first perform argmax.

Original: For this argmax index, we retrieve a learned embedding from a codebook.
Revised: For this argmax index, we retrieve a learned embedding from a codebook.

Original: To these retrieved embeddings, we add learned positional encodings and the resulting embeddings are then given as input to the transformer decoder.
Revised: To these retrieved embeddings, we add learned positional encodings and the resulting embeddings are then given as input to the transformer decoder.

Original: Similarly to SLATE, we do not let gradient be propagated across the argmax operation.
Revised: Similarly to SLATE, we do not let gradient be propagated across the argmax operation.

Original: The prediction targets are the argmax indices of the next dVAE token which we learn to predict using a cross-entropy loss.
Revised: The prediction targets are the argmax indices of the next dVAE token which we learn to predict using a cross-entropy loss.

Original: Difference between SLATE and STEVE.
Revised: Difference between SLATE and STEVE.

Original: The key difference between STEVE and SLATE is that
Revised: The key difference between STEVE and SLATE is that

Original: STEVE temporally extends SLATE by adopting a recurrent slot encoder.
Revised: STEVE temporally extends SLATE by adopting a recurrent slot encoder.

Original: In doing this, STEVE retains the same decoder architecture as that of SLATE.
Revised: In doing this, STEVE retains the same decoder architecture as that of SLATE.

Original: Training.
Revised: Training.

Original: We trained all models upto a maximum of 200K training iterations (except for CATERTex for which we trained all models to 400K iterations).
Revised: We trained all models upto a maximum of 200K training iterations (except for CATERTex for which we trained all models to 400K iterations).

Original: These amounted to training time of approximately 2 days.
Revised: These amounted to training time of approximately 2 days.

Original: Similarly to SLATE, we applied a learning rate warmup from 0 to the peak learning rate in the first 30K training iterations.
Revised: Similarly to SLATE, we applied a learning rate warmup from 0 to the peak learning rate in the first 30K training iterations.

Original: After that we decay the learning rate exponentially with a half-life of 250K training iterations.
Revised: After that we decay the learning rate exponentially with a half-life of 250K training iterations.

Original: The loss function is L STEVE =
Revised: The loss function is L STEVE =

Original: L CE + L dVAE as described in Section 3.
Revised: L CE + L dVAE as described in Section 3.

Original: The gradients from the L dVAE loss are only used to train the dVAE encoder and the dVAE decoder.
Revised: The gradients from the L dVAE loss are only used to train the dVAE encoder and the dVAE decoder.

Original: The gradients from the
Revised: The gradients from the

Original: L CE cross-entropy loss are only used for training the slot encoder and the transformer decoder.
Revised: L CE cross-entropy loss are only used for training the slot encoder and the transformer decoder.

Original: This is due to the fact that we do not let the gradients be propagated accross the argmax operation used to obtain the discrete representation of the input frame.
Revised: This is due to the fact that we do not let the gradients be propagated accross the argmax operation used to obtain the discrete representation of the input frame.

Original: All 3 components i.e. the dVAE, the slot encoder, and the transformer are trained simultaneously in a single training run.
Revised: All 3 components i.e. the dVAE, the slot encoder, and the transformer are trained simultaneously in a single training run.

Original: B.1 Computational Requirements
Revised: B.1 Computational Requirements

Original: In Table 6, we empirically compare the computational requirements of STEVE and SAVi.
Revised: In Table 6, we empirically compare the computational requirements of STEVE and SAVi.

Original: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.
Revised: We assess by how much the computational requirement change in going from the mixture-based to the transformer-based approach.

Original: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .
Revised: We find that in videos with image size 64 × 64 , STEVE requires similar resources as SAVi and about twice for image size 128 × 128 .

Original: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.
Revised: Given the quadratic memory complexity of transformers, the larger memory demand of STEVE is not surprising.

Original: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOViTex, MOVi-D, and MOVi-E, unlike ours.
Revised: However, one should also consider that the mixture-based baselines fail almost completely on datasets like CATERTex, MOViTex, MOVi-D, and MOVi-E, unlike ours.

Original: Another feature of our method is that the memory demand of transformer decoder is approximately constant in the number of slots but linear for mixture-based decoders.
Revised: Another feature of our method is that the memory demand of transformer decoder is approximately constant in the number of slots but linear for mixture-based decoders.

Original: GPU Resources.
Revised: GPU Resources.

Original: We performed our experiments on Quadro RTX 8000 GPUs.
Revised: We performed our experiments on Quadro RTX 8000 GPUs.

Original: We used 2 GPUs per each training seed, each of which took approximately 2-3 days to complete training.
Revised: We used 2 GPUs per each training seed, each of which took approximately 2-3 days to complete training.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Big- O Analysis.
Revised: Big- O Analysis.

Original: The memory complexity of the transformer decoder is dependent on the cost of self-attention between all the dVAE tokens and the cross-attention between the dVAE tokens and the slots.
Revised: The memory complexity of the transformer decoder is dependent on the cost of self-attention between all the dVAE tokens and the cross-attention between the dVAE tokens and the slots.

Original: Let the number of dVAE tokens be L and the number of slots be N .
Revised: Let the number of dVAE tokens be L and the number of slots be N .

Original: Then the number of dVAE tokens is L = HW/K 2 where H and W are image height and width, respectively; and K is the dVAE patch size.
Revised: Then the number of dVAE tokens is L = HW/K 2 where H and W are image height and width, respectively; and K is the dVAE patch size.

Original: To perform self-attention among all L tokens requires a memory complexity of O
Revised: To perform self-attention among all L tokens requires a memory complexity of O

Original: ( L 2 ) .
Revised: ( L 2 ) .

Original: Furthermore, the L tokens also perform cross-attention on the N slots which leads to an attention matrix with memory cost O ( LN ) .
Revised: Furthermore, the L tokens also perform cross-attention on the N slots which leads to an attention matrix with memory cost O ( LN ) .

Original: Adding these attention costs leads to a total cost of ≈ O
Revised: Adding these attention costs leads to a total cost of ≈ O

Original: ( L 2 )
Revised: ( L 2 )

Original: + O ( LN ) .
Revised: + O ( LN ) .

Original: But since it is common to have just a few slots N (e.g. N = 11 slots for
Revised: But since it is common to have just a few slots N (e.g. N = 11 slots for

Original: CATER) and a much larger number of tokens (e.g. L = 256 tokens for CATER), the first term O
Revised: CATER) and a much larger number of tokens (e.g. L = 256 tokens for CATER), the first term O

Original: ( L 2 ) is typically the dominant one in comparison to the other term
Revised: ( L 2 ) is typically the dominant one in comparison to the other term

Original: O
Revised: O

Original: ( LN ) .
Revised: ( LN ) .

Original: Replacing L = HW/Kgives an approximate cost of O (( HW ) 2 /K 4 ) .
Revised: Replacing L = HW/Kgives an approximate cost of O (( HW ) 2 /K 4 ) .

Original: This can be contrasted with the memory cost of mixture decoder.
Revised: This can be contrasted with the memory cost of mixture decoder.

Original: Each object component can be decoded using a CNN with a cost of O ( HW ) and there are N such components.
Revised: Each object component can be decoded using a CNN with a cost of O ( HW ) and there are N such components.

Original: Thus, the total cost is approximately O ( HWN ) .
Revised: Thus, the total cost is approximately O ( HWN ) .

Original: These suggest that as we continue of scale up the image size H and W (keeping everything else same), the memory needs of the transformer decoder would significantly surpass that of the mixture decoder.
Revised: These suggest that as we continue of scale up the image size H and W (keeping everything else same), the memory needs of the transformer decoder would significantly surpass that of the mixture decoder.

Original: B.2 Mixture-based Decoding without Spatial Broadcast
Revised: B.2 Mixture-based Decoding without Spatial Broadcast

Original: In Section 5.3, we performed an ablation of SAVi where we replaced its original Spatial Broadcast decoder with a more flexible CNN implemented entirely using transposed convolutions.
Revised: In Section 5.3, we performed an ablation of SAVi where we replaced its original Spatial Broadcast decoder with a more flexible CNN implemented entirely using transposed convolutions.

Original: In Tables 7 and 8, we describe these two CNN architectures in detail
Revised: In Tables 7 and 8, we describe these two CNN architectures in detail

Original: [Table]
Revised: [Table]

Original: We note that the decoder used in
Revised: We note that the decoder used in

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the ablation of SAVi has more layers and is therefore more expressive.
Revised: the ablation of SAVi has more layers and is therefore more expressive.

Original: It applies 7 transposed convolution layers while the original decoder had 4.
Revised: It applies 7 transposed convolution layers while the original decoder had 4.

Original: Furthermore, unlike the original decoder that uses a Spatial Broadcast in the first layer which is intended to lower its flexibility, our ablation of
Revised: Furthermore, unlike the original decoder that uses a Spatial Broadcast in the first layer which is intended to lower its flexibility, our ablation of

Original: SAVi uses transposed convolutions in all the layers of the CNN decoder.
Revised: SAVi uses transposed convolutions in all the layers of the CNN decoder.

Original: C Additional Results.
Revised: C Additional Results.

Original: In this section we provide additional evaluation results for our model.
Revised: In this section we provide additional evaluation results for our model.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: be seen as a benefit of our model
Revised: be seen as a benefit of our model

Original: [Figure]
Revised: [Figure]

Original: D Additional Related Work
Revised: D Additional Related Work

Original: Unsupervised Video Object Segmentation using Motion Cues.
Revised: Unsupervised Video Object Segmentation using Motion Cues.

Original: When objects in the video are moving independently, optical flow provides a strong signal of objectness as pixels with similar flow can be treated as an object.
Revised: When objects in the video are moving independently, optical flow provides a strong signal of objectness as pixels with similar flow can be treated as an object.

Original: Therefore, several works pursue estimating optical flow in an fully unsupervised way (Ren et al., 2017; Janai et al., 2018;
Revised: Therefore, several works pursue estimating optical flow in an fully unsupervised way (Ren et al., 2017; Janai et al., 2018;

Original: Wang et al., 2018; Jonschkowski et al., 2020;
Revised: Wang et al., 2018; Jonschkowski et al., 2020;

Original: Stone et al., 2021).
Revised: Stone et al., 2021).

Original: Leveraging flow, several methods have shown success in visually complex videos (Brox & Malik, 2010; Bideau & Learned-Miller, 2016;
Revised: Leveraging flow, several methods have shown success in visually complex videos (Brox & Malik, 2010; Bideau & Learned-Miller, 2016;

Original: Yang et al., 2019; Tangemann et al., 2021;
Revised: Yang et al., 2019; Tangemann et al., 2021;

Original: Yang et al., 2021).
Revised: Yang et al., 2021).

Original: However, these can require access to optical flow even during deployment which can be challenging to obtain.
Revised: However, these can require access to optical flow even during deployment which can be challenging to obtain.

Original: In response, recent methods aim to infer objects via RGB-only input, but by using optical flow to supervise the training (Kipf et al., 2021; Bao et al., 2022).
Revised: In response, recent methods aim to infer objects via RGB-only input, but by using optical flow to supervise the training (Kipf et al., 2021; Bao et al., 2022).

Original: Even though these can handle naturalistic videos, their learning depends primarily on the flow information.
Revised: Even though these can handle naturalistic videos, their learning depends primarily on the flow information.

Original: If flow is absent e.g. in scenes with static objects, these can suffer (Greff et al., 2022).
Revised: If flow is absent e.g. in scenes with static objects, these can suffer (Greff et al., 2022).

Original: In comparison, our model is trained given only RGB video frames
Revised: In comparison, our model is trained given only RGB video frames

Original: and we consider the use of motion information as orthogonal to our approach.
Revised: and we consider the use of motion information as orthogonal to our approach.

Original: Unsupervised Segmentation Propagation.
Revised: Unsupervised Segmentation Propagation.

Original: Several methods learn to propagate mask from one frame to another via unsupervised training and are shown to handle natural videos (Lai et al., 2020; Vondrick et al., 2018; Lai & Xie, 2019; Zhu et al., 2020;
Revised: Several methods learn to propagate mask from one frame to another via unsupervised training and are shown to handle natural videos (Lai et al., 2020; Vondrick et al., 2018; Lai & Xie, 2019; Zhu et al., 2020;

Original: Oh et al., 2019; Wang et al., 2019a,b).
Revised: Oh et al., 2019; Wang et al., 2019a,b).

Original: However these methods only learn how to propagate the masks, and still require ground truth segmentation or bounding boxes in the first frame to actually perform tracking.
Revised: However these methods only learn how to propagate the masks, and still require ground truth segmentation or bounding boxes in the first frame to actually perform tracking.

Original: Therefore, these methods are not fully unsupervised like ours.
Revised: Therefore, these methods are not fully unsupervised like ours.

Original: E Additional Experiment Details
Revised: E Additional Experiment Details

Original: Evaluating the Effect of Number of Past Frames on Image Segmentation.
Revised: Evaluating the Effect of Number of Past Frames on Image Segmentation.

Original: We randomly take a video clip from the test set having T frames: x 1 , . . .
Revised: We randomly take a video clip from the test set having T frames: x 1 , . . .

Original: , x T .
Revised: , x T .

Original: We would like to measure how the segmentation of the T -th frame x T would change as a function of the number of past frames k shown to the model.
Revised: We would like to measure how the segmentation of the T -th frame x T would change as a function of the number of past frames k shown to the model.

Original: For this, we predict the object masks M kT for the frame x T given that the model has seen k frames x T − k , . . .
Revised: For this, we predict the object masks M kT for the frame x T given that the model has seen k frames x T − k , . . .

Original: , x T − 1 in the past.
Revised: , x T − 1 in the past.

Original: We compute the Image FG-ARI for the masks M kT by varying the value of k ∈ { 0 , . . .
Revised: We compute the Image FG-ARI for the masks M kT by varying the value of k ∈ { 0 , . . .

Original: , T − 1 } and make the plot in Figure 2.
Revised: , T − 1 } and make the plot in Figure 2.

Original: We take T = 7 .
Revised: We take T = 7 .

Original: Evaluation of Out-of-Distribution Generalization to More Objects.
Revised: Evaluation of Out-of-Distribution Generalization to More Objects.

Original: In this experiment, a model is tested on more objects than it was trained on.
Revised: In this experiment, a model is tested on more objects than it was trained on.

Original: For this experiment, during testing, we increase the number of slots by an amount equal to the difference in maximum number of objects between the test and the training set.
Revised: For this experiment, during testing, we increase the number of slots by an amount equal to the difference in maximum number of objects between the test and the training set.

Original: F Limitations
Revised: F Limitations

Original: We now describe the main limitations of STEVE and the future directions.
Revised: We now describe the main limitations of STEVE and the future directions.

Original: First, there is some opportunity to improve the performance further within the proposed framework by adopting more advanced architectures than the minimal architecture we choose in this work.
Revised: First, there is some opportunity to improve the performance further within the proposed framework by adopting more advanced architectures than the minimal architecture we choose in this work.

Original: We would say that this is more of a future work than a limitation of this framework.
Revised: We would say that this is more of a future work than a limitation of this framework.

Original: Second, more computational efficiency is required in the future to handle images of higher resolution because the transformer decoder has a quadratic memory complexity and also because token-by-token generation can be slow.
Revised: Second, more computational efficiency is required in the future to handle images of higher resolution because the transformer decoder has a quadratic memory complexity and also because token-by-token generation can be slow.

Original: Third, the potential to scale the model to internet-scale datasets is discussed in this work but not investigated.
Revised: Third, the potential to scale the model to internet-scale datasets is discussed in this work but not investigated.

Original: Our current datasets typically contain about 200K video frames with an image resolution of 128.
Revised: Our current datasets typically contain about 200K video frames with an image resolution of 128.

Original: For scaling this up, we may also need to develop a new model in this framework.
Revised: For scaling this up, we may also need to develop a new model in this framework.

Original: Fourth, although our focus in this paper is not on synthetic image generation, we found that the quality of image generation in naturalistic datasets has some room for further improvement.
Revised: Fourth, although our focus in this paper is not on synthetic image generation, we found that the quality of image generation in naturalistic datasets has some room for further improvement.

Original: We believe that this could not only provide the ability to synthesize images and videos but also improve object decomposition as well.
Revised: We believe that this could not only provide the ability to synthesize images and videos but also improve object decomposition as well.

Original: Fifth, even if our model outperforms previous works in general, we also observe some failure modes in MOVi-E such as splitting of large objects or merging of very small objects.
Revised: Fifth, even if our model outperforms previous works in general, we also observe some failure modes in MOVi-E such as splitting of large objects or merging of very small objects.

Original: To resolve this, we believe that it’s worth investigating a much larger data on a bigger model or with a more sophisticated architecture.
Revised: To resolve this, we believe that it’s worth investigating a much larger data on a bigger model or with a more sophisticated architecture.

Original: Lastly, the CNN backbone of our slot encoder is rather lightweight, having just 4-layers and a feature-size 64.
Revised: Lastly, the CNN backbone of our slot encoder is rather lightweight, having just 4-layers and a feature-size 64.

Original: For scaling this model to more complex natural images, it is likely that this lightweight CNN will not be enough.
Revised: For scaling this model to more complex natural images, it is likely that this lightweight CNN will not be enough.

Original: G Impact Statement
Revised: G Impact Statement

Original: There are no imminent negative societal consequences of our current work, however, future applications should be mindful to avoid malicious uses such as in surveillance and to minimize environmental impact when training larger transformers.
Revised: There are no imminent negative societal consequences of our current work, however, future applications should be mindful to avoid malicious uses such as in surveillance and to minimize environmental impact when training larger transformers.

Original: At the same time, benign applications in scene understanding, robotics and autonomous navigation would benefit positively from our work.
Revised: At the same time, benign applications in scene understanding, robotics and autonomous navigation would benefit positively from our work.