Original: L EARNING TO PERCEIVE OBJECTS BY PREDICTION
Revised: L EARNING TO PERCEIVE OBJECTS BY PREDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The representation of objects is the building block of higher-level concepts.
Revised: The representation of objects is the building block of higher-level concepts.

Original: Infants develop the notion of objects without supervision.
Revised: Infants develop the notion of objects without supervision.

Original: The prediction error of future sensory input is likely the major teaching signal for infants.
Revised: The prediction error of future sensory input is likely the major teaching signal for infants.

Original: Inspired by this, we propose a new framework to extract object-centric representation from single 2D images by learning to predict future scenes in the presence of moving objects.
Revised: Inspired by this, we propose a new framework to extract object-centric representation from single 2D images by learning to predict future scenes in the presence of moving objects.

Original: We treat objects as latent causes whose function to an agent is to facilitate efﬁcient prediction of the coherent motion of their parts in visual input.
Revised: We treat objects as latent causes whose function to an agent is to facilitate efﬁcient prediction of the coherent motion of their parts in visual input.

Original: Distinct from previous object-centric models, our model learn to explicitly infer objects’ location in 3D environment in addition to segmenting objects.
Revised: Distinct from previous object-centric models, our model learns to explicitly infer objects’ locations in 3D environment in addition to segmenting objects.

Original: Further, the network learns a latent code space where objects with the same geometric shape and texture/color frequently group together.
Revised: Further, the network learns a latent code space where objects with the same geometric shape and texture/color frequently group together.

Original: The model requires no supervision or pre-training of any part of the network.
Revised: The model requires no supervision or pre-training of any part of the network.

Original: We provide a new synthetic dataset with more complex textures on objects and background and found several previous models not based on predictive learning overly rely on clustering colors and lose speciﬁcity in object segmentation.
Revised: We provide a new synthetic dataset with more complex textures on objects and background and found several previous models not based on predictive learning overly rely on clustering colors and lose speciﬁcity in object segmentation.

Original: Our work demonstrates a new approach for learning symbolic representation grounded in sensation and action.
Revised: Our work demonstrates a new approach for learning symbolic representation grounded in sensation and action.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Visual scenes are composed of various objects in front of backgrounds.
Revised: Visual scenes are composed of various objects in front of backgrounds.

Original: Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin et al., 2018; Wang et al., 2019) and this can potentially provide better abstraction of the environment for reinforcement learning (RL), e.g. Veerapaneni et al.
Revised: Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin et al., 2018; Wang et al., 2019) and this can potentially provide better abstraction of the environment for reinforcement learning (RL), e.g. Veerapaneni et al.

Original: The appearance and spatial arrangement of objects, together with the lighting and the viewing angle, determine the 2D images formed on the retina or a camera.
Revised: The appearance and spatial arrangement of objects, together with the lighting and the viewing angle, determine the 2D images formed on the retina or a camera.

Original: Therefore, objects are latent causes of 2D images and discovering object is a process of inferring latent causes (Kersten et al., 2004).
Revised: Therefore, objects are latent causes of 2D images, and discovering object is a process of inferring latent causes (Kersten et al., 2004).

Original: The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes (Ren et al., 2015;
Revised: The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes (Ren et al., 2015;

Original: Redmon et al., 2016) or pixel-level segmentation of objects (Chen et al., 2017).
Revised: Redmon et al., 2016) or pixel-level segmentation of objects (Chen et al., 2017).

Original: However, the supervised approach requires expensive human labeling.
Revised: However, the supervised approach requires expensive human labeling.

Original: It is also difﬁcult to label every possible category of objects.
Revised: It is also difﬁcult to label every possible category of objects.

Original: Therefore, an increasing interest has developed recently to build unsupervised or self-supervised models to infer objects from images, such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019) slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019; 2021), C-SWM (Kipf et al., 2019) and mulMON (Nanbo et al., 2020).
Revised: Therefore, an increasing interest has developed recently to build unsupervised or self-supervised models to infer objects from images, such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019) slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019; 2021), C-SWM (Kipf et al., 2019) and mulMON (Nanbo et al., 2020).

Original: Our work also focuses on the same unsupervised object-centric representation learning (OCRL) problem, but offers a new learning objective and architecture to overcome the limitation of existing works in segmenting more complex scenes and explicitly represents objects’ 3D locations.
Revised: Our work also focuses on the same unsupervised object-centric representation learning (OCRL) problem, but offers a new learning objective and architecture to overcome the limitation of existing works in segmenting more complex scenes and explicitly represents objects’ 3D locations.

Original: The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures.
Revised: The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures.

Original: As recently pointed out, the success of several recent models based on a variational auto-encoder (VAE) architecture (Kingma & Welling, 2013; Rezende et al., 2014) depends on a reconstruction bottleneck that needs to be intricately balanced (Engelcke et al., 2020).
Revised: As recently pointed out, the success of several recent models based on a variational auto-encoder (VAE) architecture (Kingma & Welling, 2013; Rezende et al., 2014) depends on a reconstruction bottleneck that needs to be intricately balanced (Engelcke et al., 2020).

Original: To evaluate how such models perform on scenes with more complex surface textures, we created a new dataset of indoor scenes with diverse texture patterns on the objects and background.
Revised: To evaluate how such models perform on scenes with more complex surface textures, we created a new dataset of indoor scenes with diverse texture patterns on the objects and background.

Original: We found that several existing unsupervised OCRL models overly rely on clustering pixels based on their colors.
Revised: We found that several existing unsupervised OCRL models overly rely on clustering pixels based on their colors.

Original: A challenge in our dataset and in real-world perception is that sharp boundaries between different colors exist both at contours of objects and within the surface of the same object.
Revised: A challenge in our dataset and in real-world perception is that sharp boundaries between different colors exist both at contours of objects and within the surface of the same object.

Original: A model essentially has to implicitly learn a prior knowledge of which types of boundaries are more likely to be real object contours.
Revised: A model essentially has to implicitly learn a prior knowledge of which types of boundaries are more likely to be real object contours.

Original: The reconstruction loss in existing works appears to be insufﬁcient for learning this prior.
Revised: The reconstruction loss in existing works appears to be insufﬁcient for learning this prior.

Original: To tackle this challenge, we draw our inspiration from development psychology and neuroscience.
Revised: To tackle this challenge, we draw our inspiration from development psychology and neuroscience.

Original: Infants understand the concept of object as early as 8 months, before they can associate objects with names (Piaget & Cook, 1952; Flavell, 1963).
Revised: Infants understand the concept of object as early as 8 months, before they can associate objects with names (Piaget & Cook, 1952; Flavell, 1963).

Original: The fact that infants are surprised when object are hidden indicates that they have already learned to segment discrete objects from the scene and that their brains constantly make prediction and check for deviation from expected outcome.
Revised: The fact that infants are surprised when object are hidden indicates that they have already learned to segment discrete objects from the scene and that their brains constantly make prediction and check for deviation from expected outcome.

Original: As the brain lacks direct external supervision for object segmentation, the most likely learning signal is from the error of this prediction.
Revised: As the brain lacks direct external supervision for object segmentation, the most likely learning signal is from the error of this prediction.

Original: In the brain, a copy of the motor command (efference copy) is sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate  prediction of changes in sensory input due to self-generated motion (Feinberg, 1978).
Revised: In the brain, a copy of the motor command (efference copy) is sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate the prediction of changes in sensory input due to self-generated motion (Feinberg, 1978).

Original: What remains to be predicted are changes in visual input due to the motion of external objects.
Revised: What remains to be predicted are changes in visual input due to the motion of external objects.

Original: Therefore, we believe that the functional purpose of grouping pixels into object is to allow the prediction of the motion of the constituting pixels in a coherent way by tracking very few parameters (e.g., the location, pose, and speed of an object).
Revised: Therefore, we believe that the functional purpose of grouping pixels into object is to allow the prediction of the motion of the constituting pixels in a coherent way by tracking very few parameters (e.g., the location, pose, and speed of an object).

Original: Driven by this hypothesis, our contribution in this paper is: (1) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware object-centric representation; (2) we provide a new dataset 1 with complex surface texture and motion by both the camera and objects  evaluate object-centric representation models; (3) we found several previous models overly rely on clustering colors to segment objects; (4) although our model leverages image prediction as learning objective, the architecture generalize the ability of object segmentation and spatial localization to single-frame images.
Revised: Driven by this hypothesis, our contribution in this paper is: (1) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware objectcentric representation, which we call Object Perception by Predictive LEarning (OPPLE); (2) we provide a new dataset 1 with complex surface texture and motion by both the camera and objects to evaluate object-centric representation models; (3) we found several previous models overly rely on clustering colors to segment objects; (4) although our model leverages image prediction as learning objective, the architecture generalize the ability of object segmentation and spatial localization to single-frame images.

Original:  M ETHOD 2. P ROBLEM FORMULATION
Revised:  M ETHOD 2. P ROBLEM FORMULATION

Original: We denote a scene as a set of distinct objects and a background S =
Revised: We denote a scene as a set of distinct objects and a background S =

Original: { O 1 , O 2 , . . .
Revised: { O 1 , O 2 , . . .

Original: , O K , B } , where K is the number of objects in scene.
Revised: , O K , B } , where K is the number of objects in scene.

Original: At any moment t , we denote two state variables, the location and pose of each object relative to the perspective of an observer (camera), as x ( t ) 1: K and φ ( t ) 1: K , where x ( t ) k is the 3-d coordinate of the k -th object and φ ( t ) k is its yaw angle from a canonical pose, as viewed from the reference frame of the camera (for simplicity, we do not consider pitch and roll here but our framework can easily generalize to 3D pose).
Revised: At any moment t , we denote two state variables, the location and pose of each object relative to the perspective of an observer (camera), as x ( t ) 1: K and φ ( t ) 1: K , where x ( t ) k is the 3-d coordinate of the k -th object and φ ( t ) k is its yaw angle from a canonical pose, as viewed from the reference frame of the camera (for simplicity, we do not consider pitch and roll here and leave it for future work to extend to 3D pose).

Original: At time t , given the location of the camera o ( t ) ∈ Rand its facing direction α ( t ) , S renders a 2D image on the camera as I ( t ) ∈
Revised: At time t , given the location of the camera o ( t ) ∈ Rand its facing direction α ( t ) , S renders a 2D image on the camera as I ( t ) ∈

Original: R w × h × 3 , where w × h is the size of the image.
Revised: R w × h × 3 , where w × h is the size of the image.

Original: Our goal is to train a neural network that infers properties of objects given only a single image
Revised: Our goal is to train a neural network that infers properties of objects given only a single image

Original: I ( t ) as the sole input without external supervision:
Revised: I ( t ) as the sole input without external supervision and with only the information of the intrinsics and egomotion of the camera without explicit knowledge of its extrinsic matrix:

Original: [Equation]
Revised: [Equation]

Original: Here, z ( t ) 1: K is a set of view-invariant vectors representing the identity of each object k .
Revised: Here, z ( t ) 1: K is a set of view-invariant vectors representing the identity of each object k .

Original: “Viewinvariant” is loosely deﬁned as | z ( t )
Revised: “Viewinvariant” is loosely deﬁned as | z ( t )

Original: k − z
Revised: k − z

Original: ( t +∆ t ) k | < | z ( t )
Revised: ( t +∆ t ) k | < | z ( t )

Original: k − z ( t ) l | for k (cid:54) =
Revised: k − z ( t ) l | for k (cid:54) =

Original: l and ∆ t > 0 in most cases, i.e., the vector codes are more similar for the same object across views than they are different across objects.
Revised: l and ∆ t > 0 in most cases, i.e., the vector codes are more similar for the same object across views than they are different across objects.

Original: π ( t ) 1: K +1 ∈ R ( K +1) × w × h are the probabilities that each pixel belongs to any of the objects or the background ( (cid:80)
Revised: π ( t ) 1: K +1 ∈ R ( K +1) × w × h are the probabilities that each pixel belongs to any of the objects or the background ( (cid:80)

Original: k
Revised: k

Original: π kij = 1 for any pixel at i, j ), which achieve object segmentation.
Revised: π kij = 1 for any pixel at i, j ), which achieve object segmentation.

Original: To localize objects, ˆ x ( t ) 1: K are the estimated locations of each object relative to the observer and p ( t ) φ 1: K are estimated probability distribution of the poses of each object.
Revised: To localize objects, ˆ x ( t ) 1: K are the estimated locations of each object relative to the observer and p ( t ) φ 1: K are estimated probability distribution of the poses of each object.

Original: Each p ( t ) φ k ∈ R b is a probability distribution over b equally-spaced bins of yaw angles in (0 , 2 π ) .
Revised: Each p ( t ) φ k ∈ R b is a probability distribution over b equally-spaced bins of yaw angles in (0 , 2 π ) .

Original: 2. N ETWORK ARCHITECTURE
Revised: 2. O VERALL PRINCIPLE : LEARNING OBJECT REPRESENTATION BY PREDICTING THE FUTURE

Revised: Our hypothesis is that the notion of object emerges to meet the need of efﬁciently predicting the future fates of all parts of an object. With the (to be learned) ability to infer an object’s pose and location from each frame, the object’s speed of translation and rotation can be estimated from consecutive frames. If depth is further inferred for each pixel belonging to an object, then the optical ﬂow of each pixel can be predicted based on the object’s speed and the position of each pixel relative to the object’s center.

Revised: The pixel-segmentation of an object essentially prescribes which pixels should move together with the object.

Revised: With the predicted optical ﬂow, one can further predict part of the next image by warping the current image.

Revised: The parts of the next image unpredictable by warping include surfaces of objects or the background that are currently occluded but will become visible, and the region of a scene newly entering the view due to self- or object-motion.

Revised: These portions can only be predicted based on the learned statistics of the appearance of objects and background, which we call ”imagination”.

Revised: In this work, we will show that with the information of self-motion, knowledge of geometry (rule of rigid-body movement) and the assumption of smooth object movement, the object representations captured by function f obj and depth perception can be learned without supervision.

Original: The ﬂow of information in networks is displayed in Figure1.
Revised: To demonstrate the hypothesized principle above, we build our OPPLE networks as illustrated in Figure 1, which process two consecutive images individually and make prediction for the next image with the information extracted from them. [Figure]

Revised: The segmented object images and depth, together with their motion information and the observer’s motion, are used by the imagination network to imagine the next scene and ﬁll the gap not predictable by warping.

Revised: The error between the ﬁnal combined prediction and the ground truth of the next image provides teaching signals for all three networks.

Original: Object extraction network.
Revised: Object extraction network.

Original: We build the object extraction network f θ obj by modiﬁcaiton of a U-Net (Ronneberger et al., 2015).
Revised: We build an object extraction network f θ obj by modiﬁcaiton of a U-Net (Ronneberger et al., 2015) to extract representation for each object, the pixels it occupies and its spatial location and pose.

Original: A basic U-Net is composed of a convolutional encoder and a transposed convolutional decoder, while each encoder layer sends a skip connection to the corresponding decoder layer, so that  both global and local information can be combined in the decoder.
Revised: A basic U-Net is composed of a convolutional encoder and a transposed convolutional decoder, while each encoder layer sends a skip connection to the corresponding decoder layer, so that the decoder can combine both global and local information.

Original: Inside our f θ obj , an image I ( t +1)
Revised: Inside our f θ obj , an image I ( t +1)

Original: ﬁrst passes through the encoder.
Revised: ﬁrst passes through the encoder.

Original: Additional Atrous spatial pyramid pooling layer (Chen et al., 2017) is inserted between the middle two convolutional layers of the encoder to[Figure] networks expand the receptive ﬁeld.
Revised: Additional Atrous spatial pyramid pooling layer (Chen et al., 2017) is inserted between the middle two convolutional layers of the encoder to expand the receptive ﬁeld.

Original: The top layer of the encoder outputs a feature vector e t capturing the global information of the scene.
Revised: The top layer of the encoder outputs a feature vector e t capturing the global information of the scene.

Original: An LSTM further repeatedly reads in e ( t ) and sequentially outputs one code for an object at a time.
Revised: A Long-Short Term Memory (LSTM) network further repeatedly reads in e ( t ) and sequentially outputs one code for an object at a time.

Original: Each object code is then mapped through a one-layer fully connected network to predict object code z ( t ) k , object location ˆ x ( t ) k and object pose probability p φ ( t ) k , k = 1 , 2 , · · · , K .
Revised: Each object code is then mapped through a one-layer fully connected network to predict object code z ( t ) k , object location ˆ x ( t ) k and object pose probability p φ ( t ) k , k = 1 , 2 , · · · , K .

Original: The  locationprediction is restricted to the  range ofpossible value within the virtual environment by logistic function and linear scaling.
Revised: The inferred location is restricted to the visible range of the camera with an upper limit of distance.

Original: The pose prediction is represented as log p φ ( t ) k for numerical stability.
Revised: The pose prediction is represented as log p φ ( t ) k for numerical stability.

Original: Each object code vector z ( t ) k is then independently fed through the decoder with shared skip connection from the encoder.
Revised: Each object code vector z ( t ) k is then independently fed through the decoder with shared skip connection from the encoder.

Original: The decoder outputs one channel for each pixel, representing an un-normalized log likelihood that the pixel belongs to the object k .
Revised: The decoder outputs one channel for each pixel, representing an un-normalized log likelihood that the pixel belongs to the object k .

Original: The unnormalized logit maps for all objects are concatenated with a map of all zero for the background, and compete through a softmax function to output the probabilistic segmentation map π ( t ) k .
Revised: The unnormalized logit maps for all objects are concatenated with a map of all zero for the background, and compete through a softmax function to output the probabilistic segmentation map π ( t ) k .

Original: Depth perception network.
Revised: Depth perception network.

Original: We use a standard U-Net for depth perception function h θ depth that processes images I ( t ) and output a single-channel depth map D ( t ) .
Revised: We use a standard U-Net for depth perception function h θ depth that processes images I ( t ) and output a single-channel depth map D ( t ) .

Original: Object-based imagination network.
Revised: Object-based imagination network.

Original: We build the imagination network g θ imag also with a modiﬁed
Revised: We build the imagination network g θ imag also with a modiﬁed

Original: U-Net.
Revised: U-Net.

Original: The input is concatenated image I ( t ) and log of depth log ( D ( t ) ) inferred by the depth perception network, both multiplied element-wise by one probabilistic mask π ( t ) k .
Revised: The input is concatenated image I ( t ) and log of depth log ( D ( t ) ) inferred by the depth perception network, both multiplied element-wise by one probabilistic mask π ( t ) k .

Original: The output of the encoder network is concatenated with a vector composed of the observer’s moving velocity v ( t ) obs and rotational speed ω ( t ) obs , and the estimated object location ˆ x ( t ) k , velocity ˆ v ( t ) k and rotational speed ˆ ω ( t ) k before entering the decoder.
Revised: The output of the encoder network is concatenated with a vector composed of the observer’s moving velocity v ( t ) obs and rotational speed ω ( t ) obs , and the estimated object location ˆ x ( t ) k , velocity ˆ v ( t ) k and rotational speed ˆ ω ( t ) k before entering the decoder.

Original: The decoder outputs ﬁve channels for each pixel: three for predicting
Revised: The decoder outputs ﬁve channels for each pixel: three for predicting

Original: RGB colors, one for depth and one for the probability of the pixel belonging to any object k or background and is used to weight the predicted color and depth for the ﬁnal ”imagination”.
Revised: RGB colors, one for depth and one for the probability of the pixel belonging to any object k or background and is used to weight the predicted color and depth for the ﬁnal ”imagination”.

Original: 2. L EARNING OBJECT REPRESENTATION BY PREDICTING THE FUTURE
Revised: 2. L EARNING OBJECT REPRESENTATION BY PREDICTION

Original: Our hypothesis is that the notion of object emerges to meet the need of efﬁciently predicting the future fates of all parts of an object.

Original: With the (to be learned) ability to infer an object’s pose and location from each frame, the object’s speed of translation and rotation can be estimated from consecutive frames.

Original: If depth is further inferred for each pixel belonging to an object, then the optical ﬂow of each pixel can be predicted based on the object’s speed and the position of each pixel relative to the object’s center.

Original: The pixel-segmentation of an object essentially prescribes which pixels should move together with the object.

Original: With the predicted optical ﬂow, one can further predict part of the next image by warping the current image.

Original: The parts of the next image unpredictable by warping include surfaces of objects or the background that are currently occluded but will become visible, and the region of a scene newly entering the view due to self- or object-motion.

Original: These portions can only be predicted based on the learned statistics of the appearance of objects and background, which we call ”imagination”.

Original: We will show that with the information of self-motion, knowledge of geometry (rule of rigid-body movement) and the assumption of smooth object movement, the object representations captured by function f obj and depth perception can be learned without supervision.

Original: Below, we explain details of the approach.
Revised: Below, we explain details of the approach.

Original: The pseudocode of the algorithm is provided in the appendix.
Revised: The pseudocode of the algorithm is provided in the appendix.

Original: 2.3. P REDICTING THE FUTURE SCENE AND OBJECT STATES
Revised: 2.4. P REDICTION BY WARPING

Original: We ﬁrst describe the prediction of part of the next image by warping the current image.
Revised: We ﬁrst describe the prediction of part of the next image by warping the current image.

Original: Here we consider only rigid objects and the fates of all visible pixel belonging to an object.
Revised: Here we consider only rigid objects and the fates of all visible pixels belonging to an object.

Original: With depth
Revised: With depth

Original: D ( t ) ∈
Revised: D ( t ) ∈

Original: R w × h
Revised: R w × h

Original: = h θ
Revised: = h θ

Original: ( I ( t ) ) of all pixel in a view inferred by the Depth Perception network based on visual features in the image I ( t ) , the 3D location of a pixel  at any coordinate ( i, j ) in the image,where | i | ≤ w − 1 2 , | j | ≤ h − 12 , can be determined given the focal length d of the camera as ˆ m ( t ) ( i,j ) = D ( t ) ( i,j ) √ i 2 + j 2 + d 2 · [ i, d, j ] .
Revised: ( I ( t ) ) of all pixels in a view inferred by the Depth Perception network based on visual features in the image I ( t ) , the 3D location of a pixel ˆ m ( t ) ( i,j ) at any coordinate ( i, j ) in the image, can be determined given the focal length d of the camera.

Original: Here, we take the coordinate of the center of an image as (0,0). On the other hand, with the inferred  ˆ x ( t ) k and ˆ x ( t − 1) k , the current and previous locations of the object k that the pixel ( i, j ) belongs to, from I ( t ) and I ( t − 1) respectively, we can estimate the instantaneous velocity of the  object ˆ v ( t ) k
Revised: We use the inferred current and previous locations, ˆ x ( t ) k and ˆ x ( t − 1) k to estimate the instantaneous velocity of the K th object ˆ v ( t ) k

Original: = ˆ x ( t ) k − ˆ x ( t − 1) k .
Revised: = ˆ x ( t ) k − ˆ x ( t − 1) k for all pixels ( i, j ) attributed to that K th object.

Original: Similarly, with the inferred the current and previous pose probabilities of the object, p ( t ) φ k and p ( t − 1) φ k , we can obtain the likelihood of its angular velocity p ( φ ( t ) k , φ ( t [Equation] 1) k 
Revised: Similarly, with the inferred the current and previous pose probabilities of the object, p ( t ) φ k and p ( t − 1) φ k , we can obtain the likelihood of its angular velocity p ( φ ( t ) k , φ ( t − 1) k | ω ( t ) k

Original: [Equation]
Revised: = ω ) .

Original: By additionally imposing a prior distribution (we use Von Mises distribution) over ω ( t ) k that favors slow rotation, we can obtain the posterior distribution of the object’s angular velocity p ( ω ( t ) k | φ ( t ) k , φ ( t − 1) k ) , andeventually the posterior distribution of the object’s next pose p ( φ ( t +1) k | φ [Equation]
Revised: We obtain the posterior distribution of the object’s next pose p ( φ ( t +1) k | φ ( t ) k , φ ( t − 1) k ) with a Von Mises prior. More details about warping and pose estimation are added in the Appendix.

Original: k , φ [Equation] ) k ) . Assuming a pixel ( i, j ) belongs to object k , using the estimated motion information ˆ v ( t ) k and p ( ω ( t ) k | φ ( t ) k , φ ( t − 1) k ) of the object, together with the current location and pose of the object and the current 3D location ˆ m ( t ) ( i,j ) of the pixel, we can predict the 3D location m (cid:48)
Revised: Assuming a pixel ( i, j ) belongs to object k , we use the estimated motion information ˆ v ( t ) k and p ( ω ( t ) k | φ ( t ) k , φ ( t − 1) k ) of the object, together with the current location and pose of the object and the current 3D location ˆ m ( t ) ( i,j ) of the pixel, to predict the 3D location m (cid:48)

Original: ( t +1) k, ( i,j ) of the pixel at the next moment as m (cid:48)
Revised: ( t +1) k, ( i,j ) of the pixel at the next moment as m (cid:48)

Original: ( t +1) k, ( i,j ) =
Revised: ( t +1) k, ( i,j ) =

Original: M ( t ) −
Revised: M ( t ) −

Original: ω obs
Revised: ω obs

Original: [ M ( t )
Revised: [ M ( t )

Original: ˆ ω k
Revised: ˆ ω k

Original: ( ˆ m ( t ) ( i,j ) − ˆ x ( t ) k )
Revised: ( ˆ m ( t ) ( i,j ) − ˆ x ( t ) k )

Original: + ˆ x ( t ) k
Revised: + ˆ x ( t ) k

Original: + ˆ v ( t ) k − v ( t ) obs ] , where M ( t ) − ω obs and M ( t )
Revised: + ˆ v ( t ) k − v ( t ) obs ] , where M ( t ) − ω obs and M ( t )

Original: ˆ ω k are rotational matrices due to the rotation of the observer and the object, respectively, and v ( t ) obs is the velocity of the observer (relative to its own reference frame at t ).
Revised: ˆ ω k are rotational matrices due to the rotation of the observer and the object, respectively, and v ( t ) obs is the velocity of the observer (relative to its own reference frame at t ).

Original: In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted.
Revised: In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted.

Original: If a pixel belongs to the background, ω K +1 = 0 and v K +

Revised: If a pixel belongs to the background, ω K +1 = 0 and v K +

Original: = 

Revised: =

Original: ( K +1 is the background’s index).
Revised: ( K +1 is the background’s index).

Original: Given the predicted 3D location, the target coordinate ( i (cid:48) , j (cid:48) )
Revised: Given the predicted 3D location, the target coordinate ( i (cid:48) , j (cid:48) )

Original: ( t +1) k of the pixel on the image and its new depth D (cid:48) k
Revised: ( t +1) k of the pixel on the image and its new depth D (cid:48) k

Original: ( i, j ) ( t +1) can be calculated.
Revised: ( i, j ) ( t +1) can be calculated.

Original: This prediction of pixel movement allows predicting the image I (cid:48) ( t +1) and depth D (cid:48) ( t +1) by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame, as explained below.
Revised: This prediction of pixel movement allows predicting the image I (cid:48) ( t +1) and depth D (cid:48) ( t +1) by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame.

Original: Because the object attribution of each pixel is not known but is inferred by f obj ( I ( t ) ) , it is represented for every pixel as a probability of belonging to each object and the background π ( t ) k , k = 1 , 2 , · · · , K + 1 .
Revised: Considering that object attribution of each pixel has to be inferred, we can write object attribution as a probability of belonging to each object, across all pixels, π ( t ) k , k = 1 , 2 , · · · , K +1 .

Original: Therefore, the predicted motion of each pixel should be described as a probability distribution over K + 1 discrete target locations p ( x (cid:48)(
Revised: The predicted motion of each pixel should be described as a probability distribution over K + 1 discrete target locations p ( x (cid:48)

Original: [Equation]
Revised: ( t +1)

Original: [Equation] [Equation] [Equation]
Revised: ( i,j ) )

Original: ) = [Equation]
Revised: = (cid:80) K

Original: i,
Revised: +1 k

Original: [Equation] ) ) , i.e., pixel ( i, j ) has a probability of π ( t ) kij to move to location x (cid:48)
Revised: =1 π ( t ) kij · δ ( x (cid:48) ( t +1) k, ( i,j ) ) , i.e., pixel ( i, j ) has a probability of π ( t ) kij to move to location x (cid:48)

Original: ( t +1) k, ( i,j )at the next time point, for k = 1 , 2 , · · · , K + 1 .
Revised: ( t +1) k, ( i,j ) .

Original: With such probabilistic prediction of pixel movement for all visible pixel ( i, j ) ( t ) , wecan partially predict the colors ofthe next image at the pixel grids where some original pixels from the currentview will landnearby by weighting their contribution: [Equation]
Revised: Using probabilistic prediction of pixel movements, we partially predict the color of next image at the pixel grids where some original pixels from the current will land by weighting their contribution I (cid:48)

Original: otherwise ( [Equation]
Revised: ( t +1) Warp ( p, q ) .

Original: We deﬁne the weight of the contribution from any source pixel ( i, j ) to  atarget pixel ( p, q ) as w k ( i, j, p, q [Equation] = [Equation]
Revised: This contribution term is used to calculate a weight for effect of a pixel (i,j) on position (p,q) on the grid, and is written as w k ( i, j, p, q ) .

Original: The ﬁrst term incorporates the uncertainty of which object a pixel belongs to. The second term e − β · D (cid:48) ( t +1) k ( i,j ) resolves the issue of occlusion when multiple pixels are predicted to move close to the same pixel grid by down-weighting the pixels predicted to land farther from the camera. This last two terms mean that only the source pixels predicted to land within a square of of 2 ×pixels centered at any target location ( p, q ) will contribute to the color I (cid:48) ( t +1) Warp ( p, q ) .
Revised: More details in the appendix.

Original: The depth map D (cid:48) ( t +1) Warp can be predicted by the same weighting scheme after replacing I ( t ) ( i, j ) with each predicted depth D (cid:48) ( t +1) k ( i, j ) assuming the pixel belongs to object k . 
Revised: . I MAGINATION

Original: For the regions not fully predictable by warping current image with  equation (2), i.e., for ( p, q ) where (cid:80) k,i,j w k ( i, j, p, q ) < 1 , we learn a function g that “imagines” the appearance I (cid:48)
Revised: For the regions not fully predictable by warping current image with appendix equation (7), i.e., for ( p, q ) where (cid:80) k,i,j w k ( i, j, p, q ) < 1 , we learn a function g that “imagines” the appearance I (cid:48)

Original: ( t +1)
Revised: ( t +1)

Original: k
Revised: k

Original: Imag ∈ R w × h ×and the pixel-wise depth D (cid:48) ( t +1)
Revised: Imag ∈ R w × h × 3 and the pixel-wise depth D (cid:48) ( t +1)

Original: k
Revised: k

Original: Imag ∈ R w × h of the object or background k in the next frame, and the predicted probabilities that each pixel in the next frame belongs to each object or the background π (cid:48)
Revised: Imag ∈ R w × h of the object or background k in the next frame, and the predicted probabilities that each pixel in the next frame belongs to each object or the background π (cid:48)

Original: ( t +1)
Revised: ( t +1)

Original: k
Revised: k

Original: Imag ∈ R w × h .
Revised: Imag ∈ R w × h .

Original: The function takes as input portion of the current image corresponding to each object I ( t ) (cid:12) π ( t ) k and the inferred depth D ( t ) (cid:12) π ( t ) k , both extracted by element-wise multiplying with the probabilistic segmentation mask π ( t ) k , the information of the camera’s self motion, and the location and motion of that object:
Revised: The function takes as input portion of the current image corresponding to each object I ( t ) (cid:12) π ( t ) k and the inferred depth D ( t ) (cid:12) π ( t ) k , both extracted by element-wise multiplying with the probabilistic segmentation mask π ( t ) k , the information of the camera’s self motion, and the location and motion of that object:

Original: [Equation]
Revised: [Equation]

Original: The “imagination” speciﬁc for each object and the background can then be merged using the weights prescribed by π (cid:48) (t +
Revised: The “imagination” speciﬁc for each object and the background can then be merged using the weights prescribed by π (cid:48) (

Original: [Equation]
Revised: [Equation]

Original: 1: K Imag : I [Equation] ( t +1) Imag = (cid:80)
Revised: : K

Original: [Equation]
Revised: [Equation]

Original: I
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ( t +
Revised: 1) k

Revised: Imag (cid:12) π (cid:48) ( t

Original: [Equation]
Revised: [Equation]

Original: k Imag (cid:12) π (cid:48) ( t +1) k
Revised: 1) k

Original: Imag , and D (cid:48) ( t +1)
Revised: Imag , and D (cid:48) ( t +1)

Original: Imag = (cid:80) k D (cid:48)
Revised: Imag = (cid:80) k D (cid:48)

Original: ( t +1)
Revised: ( t +1)

Original: k
Revised: k

Original: Imag (cid:12) π (cid:48) ( t +1) k
Revised: Imag (cid:12) π (cid:48) ( t +1) k

Original: Imag .
Revised: Imag .

Original: The ﬁnal predicted image or depth map are weighted average of the prediction made by warping the current image or predicted depth map and the corresponding predictions by imagination:
Revised: The ﬁnal predicted image or depth map are weighted average of the prediction made by warping the current image or predicted depth map and the corresponding predictions by imagination:

Original: [Equation]
Revised: [Equation]

Original: Here, W Warp ∈ R w × h with each element W Warp ( p, q ) = max { (cid:80) k,i,j w ( i, j, p, q ) , 1 } .
Revised: Here, W Warp ∈ R w × h with each element W Warp ( p, q ) = max { (cid:80) k,i,j w ( i, j, p, q ) , 1 } .

Original: The intuition is that imagination is only needed when there is not sufﬁcient contribution for predicting a pixel by warping.
Revised: The intuition is that imagination is only needed when there is not sufﬁcient contribution for predicting a pixel by warping.

Original: The same weighting applies for generating the ﬁnal predicted depth D (cid:48)
Revised: The same weighting applies for generating the ﬁnal predicted depth D (cid:48)

Original: ( t +1) .
Revised: ( t +1) .

Original: In addition to the image, the states of the objects can also be predicted.
Revised: In addition to the image, the states of the objects can also be predicted.

Original: The location of object k at t + 1 can be predicted as x (cid:48)
Revised: The location of object k at t + 1 can be predicted as x (cid:48)

Original: ( t +1) k
Revised: ( t +1) k

Original: [Equation] x [Equation] t ) k
Revised: = M ( t ) − ω obs (ˆ x ( t ) k

Original: [Equation]

Original:  ˆ v ( t ) k − v ( t ) obs ) .
Revised: + ˆ v ( t ) k − v ( t ) obs ) .

Original: Its new pose probability can be predicted by p (cid:48)
Revised: Its new pose probability can be predicted by p (cid:48)

Original: [Equation] [Equation] [Equation] [Equation]
Revised: ( t +1) ( φ k + ω obs = γ 2 ) =

Original: [Equation]

Original: [Equation] p (ˆ ω [Equation] = ω 
Revised: (cid:80) γ 1 ,ω,γ 2 − γ 1 ∈ { ω − 2 π,ω,ω +2 π } p

Original: [Equation] p ( [Equation] t ) k
Revised: (ˆ ω ( t ) k = ω ) p ( φ ( t ) k

Original: = γ 1 ) for γ 2 equal to each ﬁxed yaw angle bin.
Revised: = γ 1 ) for γ 2 equal to each ﬁxed yaw angle bin.

Original: To obtain p (cid:48) ( t +1) ( φ k ) instead of p (cid:48) ( t +1)
Revised: To obtain p (cid:48) ( t +1) ( φ k ) instead of p (cid:48) ( t +1)

Original: ( φ k + ω obs ) at the same set of bins, the vector p (cid:48)
Revised: ( φ k + ω obs ) at the same set of bins, the vector p (cid:48)

Original: ( t +1) ( φ k + ω obs ) can be shifted by multiplication with a pre-computed matrix composed of a bank of shifted Von Mises distributions to “move” the probability mess on the angle bins.
Revised: ( t +1) ( φ k + ω obs ) can be shifted by multiplication with a pre-computed matrix composed of a bank of shifted Von Mises distributions to “move” the probability mass on the angle bins.

Original: There is one important issue of object-centric representation when making prediction for future images: in order to predict the spatial state of each object at t + 1 based on the views at t and t − 1 , the network needs to match the representation of an object at t from the representation of the same object at t − 1 .
Revised: There is one important issue of object-centric representation when making prediction for future images: in order to predict the spatial state of each object at t + 1 based on the views at t and t − 1 , the network needs to match the representation of an object at t from the representation of the same object at t − 1 .

Original: As the dimensions of features (e.g., shape, surface texture, size, etc.) grows, the number of possible objects grows exponentially.
Revised: As the dimensions of features (e.g., shape, surface texture, size, etc.) grows, the number of possible objects grows exponentially.

Original: Therefore, we cannot simply match object representations based on the order by which an LSTM extracts objects, as this requires learning a consistent order over enormous amount of objects.
Revised: Therefore, we cannot simply match object representations based on the order by which an LSTM extracts objects, as this requires learning a consistent order over enormous amount of objects.

Original: Instead, we take a soft-matching approach: we take a subset of the features in z ( t ) k extracted by f as an identity code for each object.
Revised: Instead, we take a soft-matching approach: we take a subset of the features in z ( t ) k extracted by f as an identity code for each object.

Original: For object k at time t , we calculate the distance between its identity code and those of all objects at t − 1 , and pass the distances through a radial basis function to serve as a matching score r kl indicating how closely the object k matches each of the previous objects.
Revised: For object k at time t , we calculate the distance between its identity code and those of all objects at t − 1 , and pass the distances through a radial basis function to serve as a matching score r kl indicating how closely the object k matches each of the previous objects.

Original: The scores are used weight all the estimated translational and rotational speeds for object k each assuming a different object l were the true object k at t − 1 .
Revised: The scores are used weight all the estimated translational and rotational speeds for object k each assuming a different object l were the true object k at t − 1 .

Original: We additionally introduce a ﬁxed identity code z K +

Revised: We additionally introduce a ﬁxed identity code z K +

Original: = 

Revised: =

Original: for the background and set the predicted motion of background to zero.
Revised: for the background and set the predicted motion of background to zero.

Original: 2.3. L EARNING OBJECTIVE
Revised: 2.4. L EARNING OBJECTIVE

Original: Above, we have explained how the next image input I (cid:48)
Revised: Above, we have explained how the next image input I (cid:48)

Original: ( t +1) , the depth map D (cid:48)
Revised: ( t +1) , the depth map D (cid:48)

Original: ( t +1) and the spatial states of each object, x (cid:48) ( t +1) k and p (cid:48) ( t +1)
Revised: ( t +1) and the spatial states of each object, x (cid:48) ( t +1) k and p (cid:48) ( t +1)

Original: φ k can be predicted based on object-centric representation extracted by a function f from the current and previous images I ( t ) i and I ( t − 1)
Revised: φ k can be predicted based on object-centric representation extracted by a function f from the current and previous images I ( t ) i and I ( t − 1)

Original: i , the depth D (cid:48) ( t ) extracted by a function h , combined with the prediction from object-based imagination function g that are all to be learned.
Revised: i , the depth D (cid:48) ( t ) extracted by a function h , combined with the prediction from object-based imagination function g that are all to be learned.

Original: Among the three prediction targets, only the ground truth of visual input
Revised: Among the three prediction targets, only the ground truth of visual input

Original: I ( t +1) is available, while the other can only be inferred by f and h from I ( t +1) .
Revised: I ( t +1) is available, while the other can only be inferred by f and h from I ( t +1) .

Original: Therefore, for the prediction targets other than I ( t +1) , we use the self-consistent loss between the predicted value based on t and t − 1 and the inferred value based on t + 1 as additional regularization terms to learn the functions f and g .
Revised: Therefore, for the prediction targets other than I ( t +1) , we use the self-consistent loss between the predicted value based on t and t − 1 and the inferred value based on t + 1 as additional regularization terms to learn the functions f and g .

Original: To learn the functions f , g and h , we approximate them with deep neural networks with parameters θ and optimize θ to minimize the following loss function:
Revised: To learn the functions f , g and h , we approximate them with deep neural networks with parameters θ and optimize θ to minimize the following loss function:

Original: [Equation]
Revised: [Equation]

Original: Here,
Revised: Here,

Original: L image =
Revised: L image =

Original: MSE ( I (cid:48)
Revised: MSE ( I (cid:48)

Original: ( t +1) , I ( t +1 ) is the image prediction error.
Revised: ( t +1) , I ( t +1 ) is the image prediction error.

Original: [Equation]
Revised: [Equation]

Original: MSE (log( D (cid:48)
Revised: MSE (log( D (cid:48)

Original: ( t +1) ) , log( ˆ D ( t +1) )) is the error between the predicted and inferred depth. 
Revised: ( t +1) ) , log( ˆ D ( t +1) )) is the error between the predicted and inferred depth. These provide major teaching signals. L spatial = (cid:80)

Revised: Kk =1 | (cid:80)

Original: [Equation]
Revised: K +1 l =1 r kl x (cid:48) ( t +1) l − ˆ x ( t +1) k | 2 − (cid:80)

Revised: Kk =1 min {| ˆ x ( t +1) rand − ˆ x ( t +1) k

Revised: | , δ } + (cid:80)

Revised: Kk

Original: {+=
+}
Revised: {+=
+}

Revised: | ˆ x ( t +1)

Revised: k − (cid:80) i,j ˆ m ( t +1)

Original: {+i,j π kij | 
+}
Revised: {+i,j π kij | 
+}

Revised: + (cid:80)

Original:  ) is the self-consistent loss on spatial information prediction.
Revised: Kk =1 D KL (ˆ p ( t +1) φ k || (cid:80) K +1 l =1 r kl p (cid:48) ( t +1) φ l ) is the self-consistent loss on spatial information prediction.

Original: The ﬁrst term is the error between inferred and predicted location of each object, while the calculation of the predicted location incorporates soft matching between objects in consecutive frames.
Revised: The ﬁrst term is the error between inferred and predicted location of each object, while the calculation of the predicted location incorporates soft matching between objects in consecutive frames.

Original: The second term is the negative term of contrastive loss, which  prevents the network from reaching a local minimum where all objects are inferred at the same location relative to the camera.
Revised: The second term is the negative term of contrastive loss, which we found empirically prevents the network from reaching a local minimum where all objects are inferred at the same location relative to the camera (and covering minimal regions of the picture).

Revised: ˆ x rand is the inferred object location from a random sample within the same batch.

Original: The third term penalizes the discrepancy between the inferred object location and the average location of pixels in its segmentation mask.
Revised: The third term penalizes the discrepancy between the inferred object location and the average location of pixels in its segmentation mask.

Original: The last term is the KL-divergence between the predicted and inferred pose for each object at t +1 .
Revised: The last term is the KL-divergence between the predicted and inferred pose for each object at t +1 .

Original: L map =
Revised: L map =

Original: ReLu (10 − 4 − π 1:K )+ π k · π l , for k (cid:54) =
Revised: ReLu (10 − 4 − π 1:K )+ π k · π l , for k (cid:54) =

Original: l avoids loss of gradient due to zero probability of object belonging and discourages overlap between maps of different objects.
Revised: l avoids loss of gradient due to zero probability of object belonging and discourages overlap between maps of different objects.

Original: 2. D ATASET
Revised: 2. D ATASET

Original: We procedurally generated  dataset composed of 306445 triplets of images captured by a virtual camera with ﬁeld of view 
Revised: We procedurally generated a dataset composed of 306445 triplets of images captured by a virtual camera with ﬁeld of view of 90 degrees in a square room.

Original: of 90 degrees in a square room.

Original: The camera translates horizontally and pans with random small steps between consecutive frames to facilitate the learning of depth perception.
Revised: The camera translates horizontally and pans with random small steps between consecutive frames to facilitate the learning of depth perception.

Original:  objects with random shape, size, surface color or textures are spawned at random locations in the room and each move with a randomly selected constant velocity and panning speed.
Revised: 3 objects with random shape, size, surface color or textures are spawned at random locations in the room and each move with a randomly selected constant velocity and panning speed.

Original: The translation and panning of the the camera is known to the networks.
Revised: The translation and panning of the the camera is known to the networks.

Original: No other ground truth information is provided.
Revised: No other ground truth information is provided.

Original: The ﬁrst two

Original: frames

Original: serve

Original: as

Original:  data and the last frame serve as the prediction target 
Revised: The ﬁrst two frames serve as data and the last frame serve as the prediction target at t

Original: at

Original: t + 1 .
Revised: + 1 .

Original: An important difference between this dataset from the commonly used synthetic datasets for OCRL is that more complex and diverse textures are used on both the objects and the background.
Revised: An important difference between this dataset and other commonly used synthetic datasets for OCRL is that more complex and diverse textures are used on both the objects and the background.

Revised: We further evaluated on a richer version of the Trafﬁc dataset (Henderson & Lampert, 2020) in A.6.

Original: 2. C OMPARISON WITH OTHER WORKS
Revised: C OMPARISON WITH OTHER WORKS

Original: [Table] To compare our work with the statesof-art models of unsupervised objectcentric representation learning, we trained MONet (Burgess et al., 2019), slot-attention (Locatello et al., 2020) and GENESIS 2 (Engelcke et al., 2021) on the same dataset. 
Revised: To compare our work with the states-of-the-art models of unsupervised object-centric representation learning, we trained MONet (Burgess et al., 2019), slot-attention (Locatello et al., 2020) and GENESIS 2 (Engelcke et al., 2021) on the same dataset. Although these models are trained on single images, all images of each triplets are used for training.

Revised: [Table]

Original: To address the concern that the original conﬁgurations of the models are not optimized for more difﬁcult dataset, we trained variants of some of the models with large network size.
Revised: To address the concern that the original conﬁgurations of the models are not optimized for more difﬁcult dataset, we trained variants of some of the models with large network size.

Original: For MONet, we tested channel numbers of [32, 64, 128, 128] (MONet-128) and [32, 64, 128, 256, 256] (MONet-128-bigger) for the hidden layers of encoder of the component VAE instead of [32, 32, 64, 64] and adjusted decoder layers sizes accordingly, and increased the base channel from 64 to 128 for the attention network.
Revised: For MONet, we tested channel numbers of [32, 64, 128, 128] (MONet-128) and [32, 64, 128, 256, 256] (MONet-128-bigger) for the hidden layers of encoder of the component VAE instead of [32, 32, 64, 64] and adjusted decoder layers sizes accordingly, and increased the base channel from 64 to 128 for the attention network.

Original: For slot attention, we tested a variant which increased the number of features in the attention component from 64 to(slot-attention-128).
Revised: For slot attention, we tested a variant which increased the number of features in the attention component from 64 to 128 (slotattention-128).

Original: Slot numbers were chosen as 4 except for GENESIS.
Revised: Slot numbers were chosen as 4 except for GENESIS.

Original:  R ESULTS
Revised:  R ESULTS

Original: After training the networks, we evaluate them on 4000 test images unused during training but generated randomly with the same procedure, thus coming from the same distribution.
Revised: After training the networks, we evaluate them on 4000 test images unused during training but generated randomly with the same procedure, thus coming from the same distribution.

Original: We compare the performance of different models mainly on their segmentation performance.
Revised: We compare the performance of different models mainly on their segmentation performance.

Original: Additionally, we demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth of the scene.
Revised: Additionally, we demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth of the scene.

Original: The performance of depth perception is illustrated in the appendix.
Revised: The performance of depth perception is illustrated in the appendix.

Original: 3. O BJECT SEGMENTATION
Revised: 3. O BJECT SEGMENTATION

Original: Following prior works (Greff et al., 2019; Engelcke et al., 2019; 2021), we evaluated segmentation with the Adjusted Rand Index of foreground objects (ARI).
Revised: Following prior works (Greff et al., 2019; Engelcke et al., 2019; 2021), we evaluated segmentation with the Adjusted Rand Index of foreground objects (ARI).

Original: In addition, for each image, we matched ground-true objects and background with each of the segmented class by ranking their Intersection over Union (IoU) and quantiﬁed the average IoU over all foreground objects.
Revised: In addition, for each image, we matched ground-true objects and background with each of the segmented class by ranking their Intersection over Union (IoU) and quantiﬁed the average IoU over all foreground objects 3 . The performance is summarized in table 2.6 .

Original: [Figure]
Revised: [Figure]

Original: Our model outperforms all compared models on both metrics.
Revised: Our model outperforms all compared models on ARI and is second to a slot-attentionin IoU.

Original: As shown in Figure 2, MONet and GENESIS appear to heavily rely on color to group pixels into the same masks.
Revised: As shown in Figure 2, MONet and GENESIS appear to heavily rely on color to group pixels into the same masks.

Original: Slotattention model appears to adopt some spatial stereotype to segment the scene.

Original: Even though some of these models almost fully designate pixels of an object to a mask, the masks lacks speciﬁcity in that they often include pixels with similar colors from other objects or background.
Revised: Even though some of these models almost fully designate pixels of an object to a mask, the masks lacks speciﬁcity in that they often include pixels with similar colors from other objects or background.

Original: Patterns on the backgrounds are often treated as objects as well.
Revised: Patterns on the backgrounds are often treated as objects as well.

Original: The reason of such drawbacks awaits further investigation but we postulate there may be fundamental limitation in the approach that learns purely from static discrete images.
Revised: The reason of such drawbacks awaits further investigation but we postulate there may be fundamental limitation in the approach that learns purely from static discrete images.

Original: Patches in the background with coherent color offer room to compress information similarly as objects with coherent colors do, and their shapes re-occur across images just as other objects.
Revised: Patches in the background with coherent color offer room to compress information similarly as objects with coherent colors do, and their shapes re-occur across images just as other objects.

Original: Our model is able to learn object-speciﬁc masks because these masks are used to predict optical ﬂow speciﬁc to each object.
Revised: Our model is able to learn object-speciﬁc masks because these masks are used to predict optical ﬂow speciﬁc to each object.

Original: A wrong segmentation would generate large prediction error even if the motion of an object is estimated correctly.
Revised: A wrong segmentation would generate large prediction error even if the motion of an object is estimated correctly.

Original: Such prediction error forces the masks to be concentrated on object surface.
Revised: Such prediction error forces the masks to be concentrated on object surface.

Original: They emerge ﬁrst at object boundaries where the prediction error is the largest and gradually grow inwards during training.
Revised: They emerge ﬁrst at object boundaries where the prediction error is the largest and gradually grow inwards during training.

Original: Figure 3A-D further compares the distribution  of IoU across models.
Revised: Figure 3A-D further compares the distribution [Figure] of IoU across models

Original: All models have IoU < 0. for about 1/4 of objects.
Revised: Most models have IoU < 0.01 for about 1/4 of objects.

Original: Only OPPLE shows a bi-modal distribution while other models’ IoU are more skewed towards 0.
Revised: Only OPPLE and slot-attention-128 show bi-modal distributions while other models’ IoU are more skewed towards 0.

Original: Figure 3G plots each object’s distance and size on the picture with colors corresponding to their IoUs in our model.
Revised: Figure 3G plots each object’s distance and size on the picture with colors corresponding to their IoUs in our model.

Original: Objects with poor segmentation (blue dots) are mostly far away from the camera and occupy few pixels.
Revised: Objects with poor segmentation (blue dots) are mostly far away from the camera and occupy few pixels.

Original: This is reasonably because motion of farther objects causes less shift on the images and thus provide weaker teaching signal for the network to assign their pixels as separate from the background.
Revised: This is reasonably because motion of farther objects causes less shift on the images and thus provide weaker teaching signal for the network to assign their pixels as separate from the background.

Original: For other models, blue dots are more spread even for near objects (not shown).
Revised: For other models, blue dots are more spread even for near objects (not shown).

Original: 3. O BJECT LOCALIZATION
Revised: 3. O BJECT LOCALIZATION

Original: The Object Extraction Network infers object location relative to the camera.
Revised: The Object Extraction Network infers object location relative to the camera.

Original: We convert the inferred locations to angles and distance in polar coordinate relative to the camera.
Revised: We convert the inferred locations to angles and distance in polar coordinate relative to the camera.

Original: Figure 3E-F plot the true and inferred angles and distance, color coded by objects’ IoUs.
Revised: Figure 3E-F plot the true and inferred angles and distance, color coded by objects’ IoUs.

Original: For objects well segmented (red dots), their angles are estimated high accurately (concentrated on the diagonal in E).
Revised: For objects well segmented (red dots), their angles are estimated high accurately (concentrated on the diagonal in E).

Original: Distance estimation is negatively biased for farther objects, potentially because the regularization term on the distance between the predicted and inferred object location at frame t + 1 favors shorter distance when estimation is noisy.
Revised: Distance estimation is negatively biased for farther objects, potentially because the regularization term on the distance between the predicted and inferred object location at frame t + 1 favors shorter distance when estimation is noisy.

Original: Note that the ability to explicitly infer object’s location is not available in other models compared.
Revised: Note that the ability to explicitly infer object’s location is not available in other models compared.

Original: 3. M EANINGFUL LATENT CODE
Revised: 3. M EANINGFUL LATENT CODE

Original: Because a subset of the latent code (10 dimensions) was used to calculate object matching scores between frames in order to soft-match objects, this should force the object embedding z to be similar for the same objects.
Revised: Because a subset of the latent code (10 dimensions) was used to calculate object matching scores between frames in order to soft-match objects, this should force the object embedding z to be similar for the same objects.

Original: We explored the geometry of the latent code by examining whether the nearest neighbours of each of the object in the test data with IoU > 0.5 are more likely to have the same property as themselves.
Revised: We explored the geometry of the latent code by examining whether the nearest neighbours of each of the object in the test data with IoU > 0.5 are more likely to have the same property as themselves.

Original: 772 out of 3244 objects’ nearest neighbour had the same shape (out of 11 shapes) and 660 objects’ nearest neighbour had the same color or texture (out of 15).
Revised: 772 out of 3244 objects’ nearest neighbour had the same shape (out of 11 shapes) and 660 objects’ nearest neighbour had the same color or texture (out of 15).

Original: These numbers are 28 to 29 times the standard deviation away from the means of the distribution expected if the nearest neighbour were random (Figure 3H).
Revised: These numbers are 28 to 29 times the standard deviation away from the means of the distribution expected if the nearest neighbour were random (Figure 3H).

Original: This suggests the latent code reﬂects meaningful features of objects.
Revised: This suggests the latent code reﬂects meaningful features of objects.

Original: However, texture and shape are not the only factors determining latent code, as we found the variance of code of all objects with the same shape and texture to still be big.
Revised: However, texture and shape are not the only factors determining latent code, as we found the variance of code of all objects with the same shape and texture to still be big.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is on the same tracks as two recent trends in machine learning: object-centric representation Locatello et al. and self-supervised learning Chen et al.
Revised: Our work is on the same tracks as two recent trends in machine learning: object-centric representation (Locatello et al., 2020) and self-supervised learning (Chen et al., 2020).

Original: We take the same logic as self-supervised learning that learning to predict part of the data based on another part forces a neural network to learn important structures in the data.
Revised: We take the same logic as self-supervised learning that learning to predict part of the data based on another part forces a neural network to learn important structures in the data.

Original: However, most of the existing works in self-supervised learning do not focus on object-based representation, but instead encode the entire scene as one vector.
Revised: However, most of the existing works in self-supervised learning do not focus on object-based representation, but instead encode the entire scene as one vector.

Original: Other works on object-centric representations overcome this by assigning one representation to each object, as we do.
Revised: Other works on object-centric representations overcome this by assigning one representation to each object, as we do.

Original: Although works such as MONet Burgess et al. IODINE Greff et al. (2019), and slot-attention Locatello et al. (2020) and GENESIS (Engelcke et al., 2019)  can also segment objects and some of them can ”imagine” complete objects based on codes extracted from occluded objects or draw objects in the correct order consistent with occlusion
Revised: Although works such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019), slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019) and PSGNet (Bear et al., 2020) can also segment objects and some of them can ”imagine” complete objects based on codes extracted from occluded objects or draw objects in the correct order consistent with occlusion, few works explicitly infer an object’s location in 3D space together with segmentation purely by self-supervised learning, with the exception of a closely related work O3V (Henderson & Lampert, 2020).

Revised: Both our works learn from videos.

Original: , our work is the ﬁrst to be able to explicitly infer an object’s location in 3D space purely by selfsupervised learning, together with segmentation, to the best of our knowledge.
Revised: One major distinction is that O3V interleaves spatial and temporal convolution, thus it still require video as input at test time. In contrast, our three major networks process each image independently. Therefore, once trained, our network can generalize to single images.

Revised: Another distinction from Henderson & Lampert (2020) and many other works is that our model learns from prediction instead of reconstruction.

Original: Contrastive-learning of structured world model Kipf et al. (2019) also learns object masks and predict their future states by linking each object mask with a node in a Graphic Neural Network (GNN).
Revised: Contrastivelearning of structured world model (Kipf et al., 2019) also learns object masks and predict their future states by linking each object mask with a node in a Graphic Neural Network (GNN).

Original: The order of mapping object slot to nodes of GNN is ﬁxed through time, and the actions to objects are coded with speciﬁc associated nodes.
Revised: The order of mapping object slot to nodes of GNN is ﬁxed through time, and the actions to objects are coded with speciﬁc associated nodes.

Original: This arrangement may become infeasible with combinatorial number of different possible objects as the order of assigning different objects to a limited number of nodes may not be consistent across scenes.
Revised: This arrangement may become infeasible with combinatorial number of different possible objects as the order of assigning different objects to a limited number of nodes may not be consistent across scenes.

Original: We solve this by a soft matching of object representation between different[Figure] time points, which does not require the RNN in the Object Extraction Network to learn a ﬁxed order of extracting different types of objects.
Revised: We solve this by a soft matching of object representation between different time points, which does not require the RNN in the Object Extraction Network to learn a ﬁxed order of extracting different types of objects.

Original: On the neuroscience side, our work is highly motivated by recent works on predictive learning (O’Reilly et al., 2021) which also yields view-invariance representation while self-motion signal is available.
Revised: On the neuroscience side, our work is highly motivated by recent works on predictive learning (O’Reilly et al., 2021) which also yields view-invariance representation while self-motion signal is available.

Original: O’Reilly et al.
Revised: O’Reilly et al.

Original: (2021) uses biologically plausible but less efﬁcient learning and was demonstrated with an easier dataset with objects without background, and did not learn object localization.
Revised: (2021) used biologically plausible but less efﬁcient learning and applied their model to an easier dataset with objects without background, and did not learn object localization.

Original: Depth perception was not pursued in most previous studies of object-centric representation.

Original: We should note that explicit spatial localization and depth perception was not pursued in previous works on self-supervised object-centric learning, and the images in our dataset have signiﬁcantly richer texture information than those demonstrated in previous works Burgess et al. ; Kipf et al.
Revised: We should note that explicit spatial localization and depth perception were not pursued in previous works on self-supervised object-centric learning, and the images in our dataset have signiﬁcantly richer texture information than those demonstrated in previous works (Burgess et al., 2019; Kipf et al., 2019), making the task more challenging.

Original: (2019), making the task more challenging.
Revised: Although view synthesis is not our central goal, the principle illustrated here can be combined with recent advancement in 3D-aware image synthesis (Wiles et al., 2020).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We provide a new approach to learn object-centric representation that includes explicit spatial localization of objects, object segmentation from image, automatic matching the same objects across scene based on a learned latent code and depth perception as a by-product.
Revised: We provide a new approach to learn object-centric representation that includes explicit spatial localization of objects, object segmentation from image, automatic matching the same objects across scene based on a learned latent code and depth perception as a by-product.

Original: All of the information extracted by our networks are learned without supervision and no pre-training on other tasks is involved.
Revised: All of the information extracted by our networks are learned without supervision and no pre-training on other tasks is involved.

Original: The only additional information required is that of observer’s self-motion, which is availableboth in the brain as efference copy and easily accessible in vehicles.
Revised: The only additional information required is that of observer’s self-motion, which is available in the brain as efference copy.

Original: This demonstrate the possibility of learning rich embodied information of object, one step toward linking neural networks with symbolic representation in general.
Revised: This demonstrate the possibility of learning rich embodied information of object, one step toward linking neural networks with symbolic representation in general.

Original: We expect future works to develop self-supervised learning model for natural categories beyond simple object identity, building on our work.
Revised: We expect future works to develop self-supervised learning model for natural categories beyond simple object identity, building on our work.

Original: The work demonstrates that the notion of object can emerge as a necessary common latent cause of the pixels belonging to the object for the purpose of efﬁciently explaining away the pixels’ coherent movement across frames.
Revised: The work demonstrates that the notion of object can emerge as a necessary common latent cause of the pixels belonging to the object for the purpose of efﬁciently explaining away the pixels’ coherent movement across frames.

Original: In our experiment, object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation.
Revised: In our experiment, object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation.

Original: As a limitation, almost all existing object-centric representation works, including ours, focus on rigid bodies and simple environment.
Revised: As a limitation, almost all existing object-centric representation works, including ours, focus on rigid bodies and simple environment.

Original: Future works need to explore how to learn object representation for deformable objects, objects with more complex shapes and lighting conditions, and more cluttered environment, towards more realistic application.
Revised: Future works need to explore how to learn object representation for deformable objects, objects with more complex shapes and lighting conditions, and more cluttered environment, towards more realistic application.

Original: There is important implication for learning 3D-aware object-based representation, because for such representation to be useful eventually for robotics and RL, agents need to understand an object’s spatial relation to itself based on vision.
Revised: There is important implication for learning 3D-aware object-based representation, because for such representation to be useful eventually for robotics and RL, agents need to understand an object’s spatial relation to itself based on vision.

Original: Learning object-centric representation with explicit 3D information is an important steps toward embodied representation of the environment.
Revised: Learning object-centric representation with explicit 3D information is an important step toward embodied representation of the environment.

Original: A A PPENDIX A. P SEUDO CODE OF THE OPPLE FRAMEWORK
Revised: A A PPENDIX A. P SEUDO CODE OF THE OPPLE FRAMEWORK

Original:  Initialize: network parameters θ
Revised: Init Initialize: Network parameters θ

Original: Input: images I ( t − 1) , I ( t ) ∈
Revised: Input: images I ( t − 1) , I ( t ) ∈

Original: R w × h × 3 , self-motion v ( t − 1) obs , ω ( t − 1) obs , v ( t ) obs , ω ( t ) obs
Revised: R w × h × 3 , self-motion v ( t − 1) obs , ω ( t − 1) obs , v ( t ) obs , ω ( t ) obs

Original: Output: prediction I (cid:48) ( t +1) , segmentation π ( t − 1) 1: K +1 , π ( t ) 1: K +1 , objects’ codes z ( t − 1) 1: K , z ( t ) 1: K , objects’ locations and poses ˆ x ( t − 1) 1: K , p ( t − 1) φ 1: K , ˆ x ( t ) 1: K , p ( t ) φ 1: K for τ = { t − 1 , t } do scene code e ( τ ) ← U-NetEncoder f
Revised: Output: prediction I (cid:48) ( t +1) , segmentation π ( t − 1) 1: K +1 , π ( t ) 1: K +1 , objects’ codes z ( t − 1) 1: K , z ( t ) 1: K , objects’ locations and poses ˆ x ( t − 1) 1: K , p ( t − 1) φ 1: K , ˆ x ( t ) 1: K , p ( t ) φ 1: K for τ = { t − 1 , t } do scene code e ( τ ) ← U-NetEncoder f

Original: θ ( I ( τ ) ) object code z ( τ ) 1: K , location ˆ x ( τ ) 1: K , pose p ( τ ) φ 1: K ← LSTM f θ ( e ( τ ) ) background code
Revised: θ ( I ( τ ) ) object code z ( τ ) 1: K , location ˆ x ( τ ) 1: K , pose p ( τ ) φ 1: K ← LSTM f θ ( e ( τ ) ) background code

Original: z K +1 =depth D ( τ ) ← h
Revised: z K +1 =depth D ( τ ) ← h

Original: θ ( I ( τ ) ) segmentation mask π
Revised: θ ( I ( τ ) ) segmentation mask π

Original: ( τ ) 1: K +1 ← Softmax (U-NetDecoder
Revised: ( τ ) 1: K +1 ← Softmax (U-NetDecoder

Original: f θ ( I ( τ ) , z ( τ ) 1: K ) , 0) end for object matching scores r kl ← RBF ( z ( t ) k , z ( t − 1) l ) , k, l ∈ 1 : K +for k ← 1 to K do object motion ˆ v 1: K , ω 1: K ← r k,l , ˆ x ( t ) k , ˆ x ( t − 1) l , p ( t ) φ k , p ( t − 1) φ l , l = 1 : K +onject-speciﬁc optical ﬂow k ← ˆ v 1: K , ω 1: K , v obs , ω obs , D ( t ) end for
Revised: f θ ( I ( τ ) , z ( τ ) 1: K ) , 0) end for object matching scores r kl ← RBF ( z ( t ) k , z ( t − 1) l ) , k, l ∈ 1 : K +for k ← 1 to K do object motion ˆ v 1: K , ω 1: K ← r k,l , ˆ x ( t ) k , ˆ x ( t − 1) l , p ( t ) φ k , p ( t − 1) φ l , l = 1 : K +onject-speciﬁc optical ﬂow k ← ˆ v 1: K , ω 1: K , v obs , ω obs , D ( t ) end for

Original: I (cid:48)
Revised: I (cid:48)

Original: ( t +1) warp ← Warp ( I ( t ) , optical ﬂow 1: K +1 )
Revised: ( t +1) warp ← Warp ( I ( t ) , optical ﬂow 1: K +1 )

Original: I (cid:48)
Revised: I (cid:48)

Original: ( t +1) imagine ← g
Revised: ( t +1) imagine ← g

Original: θ ( I ( t ) (cid:12) π ( t ) 1: K +1 , log ( D ( t ) )
Revised: θ ( I ( t ) (cid:12) π ( t ) 1: K +1 , log ( D ( t ) )

Original: (cid:12) π ( t ) 1: K +1 , v obs , ω obs , ˆ v 1: K +1 , ˆ x 1: K ) ﬁnal image prediction: I (cid:48)
Revised: (cid:12) π ( t ) 1: K +1 , v obs , ω obs , ˆ v 1: K +1 , ˆ x 1: K ) ﬁnal image prediction: I (cid:48)

Original: ( t +1) ← I (cid:48)
Revised: ( t +1) ← I (cid:48)

Original: ( t +1) warp , I (cid:48)
Revised: ( t +1) warp , I (cid:48)

Original: ( t +1) imagine , warping weights update parameters: θ ← θ − γ ∇ θ [ | I (cid:48)
Revised: ( t +1) imagine , warping weights update parameters: θ ← θ − γ ∇ θ [ | I (cid:48)

Original: ( t +1) − I ( t +1) | 2 + regularization loss ]
Revised: ( t +1) − I ( t +1) | 2 + regularization loss ]

Original: A. PERFORMANCE ON DEPTH PERCEPTION
Revised: A. PERFORMANCE ON DEPTH PERCEPTION

Original: We demonstrate a few example images and the inferred depth.
Revised: We demonstrate a few example images and the inferred depth.

Original: Our network can capture the global 3D structure of the scene, although details on object surfaces are still missing.
Revised: Our network can capture the global 3D structure of the scene, although details on object surfaces are still missing.

Original: Because background occurs in every training sample, the network appear to bias the depth estimation on objects towards the depth of the walls behind, as is also shown in the scatter plot.
Revised: Because background occurs in every training sample, the network appears to bias the depth estimation on objects towards the depth of the walls behind, as is also shown in the scatter plot.

Original: [Figure]
Revised: [Figure]

Original: A. N ETWORK TRAINING AND DATASET
Revised: A. N ETWORK TRAINING AND DATASET

Original: We trained the three networks jointly using ADAM optimization Kingma & Ba (2014) with a learning rate of 3 e − 4 , (cid:15) = 1 e − 6 and other default setting in PyTorch, with a batch size of 24.
Revised: We trained the three networks jointly using ADAM optimization Kingma & Ba (2014) with a learning rate of 3 e − 4 , (cid:15) = 1 e − 6 and other default setting in PyTorch, with a batch size of 24.

Original: 40 epochs were trained on the dataset.
Revised: 40 epochs were trained on the dataset.

Original: We set λ spatial = 1 . 0 , λ depth = 0 .
Revised: We set λ spatial = 1 . 0 , λ depth = 0 .

Original: 1 and L map = 0 .
Revised: 1 and L map = 0 .

Original: 005 Images in datasets are rendered in Unity environment and downsampled to 128 × 128 resolution for training.
Revised: 005 Images in datasets are rendered in Unity environment and downsampled to 128 × 128 resolution for training.

Original: Images are rendered in sequence of 7 steps each time in a room with newly selected texture and object.
Revised: Images are rendered in sequence of 7 steps each time in a room with newly selected texture and object.

Original: Camera moves with random steps with random rotation between consecutive frames.
Revised: Camera moves with random steps and random rotation between consecutive frames.

Original: All possible triplets equally spaced by 1, 2, and 3 frames form training samples.
Revised: All possible triplets equally spaced by 1, 2, and 3 frames form training samples.

Original: In total 250K triplet samples are available for training.

Original: The model was implemented in PyTorch and trained on NVidia RTX 6000.
Revised: The model was implemented in PyTorch and trained on NVidia RTX 6000.

Original: We will release the code and dataset upon publication of the manuscript.
Revised: We will release the code and dataset upon publication of the manuscript.

Revised: A. M ETHOD A.4. P REDICTION BY WARPING

Revised: We ﬁrst describe the prediction of part of the next image by warping the current image.

Revised: Here we consider only rigid objects and the fates of all visible pixels belonging to an object.

Revised: With depth

Revised: D ( t ) ∈

Revised: R w × h

Revised: = h θ

Revised: ( I ( t ) ) of all pixels in a view inferred by the Depth Perception network based on visual features in the image I ( t ) , the 3D location of a pixel at any coordinate ( i, j ) in the image, where | i | ≤ w − 1 2 , | j | ≤ h − 12 , can be determined given the focal length d of the camera as ˆ m ( t ) ( i,j ) =

Revised: D ( t ) ( i,j ) √ i 2 + j 2 + d 2 · [ i, d, j ] .

Revised: Here, we take the coordinate of the center of an image as (0,0).

Revised: On the other hand, with the inferred ˆ x ( t ) k and ˆ x ( t − 1) k , the current and previous locations of the object k that the pixel ( i, j ) belongs to, from I ( t ) and I ( t − 1) respectively, we can estimate the instantaneous velocity of the object ˆ v ( t ) k

Revised: = ˆ x ( t ) k − ˆ x ( t − 1) k .

Revised: Similarly, with the inferred the current and previous pose probabilities of the object, p ( t ) φ k and p ( t − 1) φ k , we can obtain the likelihood of its angular velocity p (

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: By additionally imposing a prior distribution (we use Von Mises distribution) over ω ( t ) k that favors slow rotation, we can obtain the posterior distribution of the object’s angular velocity p ( ω ( t ) k | φ ( t ) k , φ ( t − 1) k ) , and eventually the posterior distribution of the object’s next pose p ( φ ( t +1) k | φ ( t ) k , φ ( t − 1) k ) .

Revised: Assuming a pixel ( i, j ) belongs to object k , using the estimated motion information ˆ v ( t ) k and p ( ω ( t ) k | φ ( t ) k , φ ( t − 1) k ) of the object, together with the current location and pose of the object and the current 3D location ˆ m ( t ) ( i,j ) of the pixel, we can predict the 3D location m (cid:48)

Revised: ( t +1) k, ( i,j ) of the pixel at the next moment as

Revised: [Equation]

Revised: 6) where M ( t ) − ω obs and M ( t ) ˆ ω

Revised: k are rotational matrices due to the rotation of the observer and the object, respectively, and v ( t ) obs is the velocity of the observer (relative to its own reference frame at t ).

Revised: In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted.

Revised: If a pixel belongs to the background, ω K +1 =

Original: {+and v K +
+}
Revised: {+and v K +
+}

Revised: =

Revised: ( K + 1 is the background’s index).

Revised: Given the predicted 3D location, the target coordinate ( i (cid:48) , j (cid:48) )

Revised: ( t +1) k of the pixel on the image and its new depth D (cid:48) k

Revised: ( i, j ) ( t +1) can be calculated.

Revised: This prediction of pixel movement allows predicting the image I (cid:48) ( t +1) and depth D (cid:48) ( t +1) by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame, as explained in Sec 2.4.2.

Revised: A.4. W ARPING CONTRIBUTION WEIGHT

Revised: As the object attribution of each pixel is not known but is inferred by f obj ( I ( t ) ) , it is represented for every pixel as a probability of belonging to each object and the background π ( t ) k , k = 1 , 2 , · · · , K + 1 .

Revised: Therefore, the predicted motion of each pixel should be described as a probability distribution over K + 1 discrete target locations p ( x (cid:48)

Revised: ( t +1)

Revised: ( i,j ) )

Revised: = (cid:80) K

Revised: +1 k

Revised: =1 π ( t ) kij · δ ( x (cid:48) ( t +1) k, ( i,j ) ) , i.e., pixel ( i, j ) has a probability of π ( t ) kij to move to location x (cid:48)

Revised: ( t +1) k, ( i,j ) at the next time point, for k = 1 , 2 , · · · , K +1 .

Revised: With such probabilistic prediction of pixel movement for all visible pixel ( i, j ) ( t ) , we can partially predict the colors of the next image at the pixel grids where some original pixels from the current view will land nearby by weighting their contribution:

Revised: [Equation]

Revised: otherwise

Revised: [Equation]

Revised: We deﬁne the weight of the contribution from any source pixel ( i, j ) to a target pixel ( p, q ) as w

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: | j (cid:48) ( t +1)

Revised: k − q | , 0 } (8)

Revised: The ﬁrst term incorporates the uncertainty of which object a pixel belongs to.

Revised: The second term e − β · D (cid:48) ( t +1) k

Revised: ( i,j ) resolves the issue of occlusion when multiple pixels are predicted to move close to the same pixel grid by down-weighting the pixels predicted to land farther from the camera.

Revised: These last two terms mean that only the source pixels predicted to land within a square of of 2 ×pixels centered at any target location ( p, q ) will contribute to the color I (cid:48) ( t +1) Warp ( p, q ) .

Revised: The depth map D (cid:48) ( t +1)

Revised: Warp can be predicted by the same weighting scheme after replacing I ( t ) ( i, j ) with each predicted depth D (cid:48)

Revised: ( t +1) k

Revised: ( i, j ) assuming the pixel belongs to object k .

Revised: A. DEPENDENCY OF SEGMENTATION PERFORMANCE ON OBJECT SIZE AND DISTANCE ACROSS MODELS

Revised: [Table]

Revised: IoU

Revised: A. I LLUSTRATION OF PREDICTION QUALITY

Revised: In the ﬁgure below, we display the ﬁrst, second image, one of the masked objects, the predicted third image, and the ground truth of the third image, both for our dataset, and for a richer version of the

Revised: Trafﬁc dataset used by (Henderson & Lampert, 2020).

Revised: We provide reference lines and some circles to aid the comparison between images and evaluate the warping quality.

Revised: Imagination quality can be insprected usually at one side of the predicted images (the camera motion is typically larger in our dataset).