SAVi ++ : Towards End-to-End Object-Centric Learning from Real-World Videos
Abstract
The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions.
Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided.
Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos.
In an effort to bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning.
We introduce
SAVi ++ , an object-centric video model which is trained to predict depth signals from a slot-based video representation.
By further leveraging best practices for model scaling, we are able to train SAVi ++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision.
Finally, we demonstrate that by using sparse depth signals obtained from LiDAR, SAVi ++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset. Project page: https://slot-attention-video.github.io/savi++/
Introduction
[Figure]
The natural world consists of distinct entities—people, dogs, cars, trees, etc.— and its complexity emerges from the combined, mostly independent, actions of the entities.
This compositional structure must be appreciated to predict future states of the world and to effect particular outcomes.
People have an intrinsic understanding of objects: objects have spatiotemporal coherence, they interact when in close proximity, and they [Table] [Table] [Table]
[Table] [Table] possess persistent, latent characteristics that determine their behavior over extended periods of time [31, 55]. Just as object-centric representations are critical to human understanding, they have the potential in machine learning to greatly improve sample efﬁciency, robustness, visual reasoning, and interpretability of learning algorithms [19, 41].
For example, consider the challenge faced by an autonomous vehicle operating in diverse surroundings (Figure 1).
Generalization across situations requires learning about recurring entities like cars, trafﬁc lights, and pedestrians, and the rules that govern interactions among these entities.
In human brains, the ability to organize edges and surfaces into unitary, bounded, and persisting object representations develops through experience and/or maturation from infancy and without explicit instruction via a ‘core system of object representation’ [55], i.e., a form of cognitive inductive bias.
In deep learning, such an inductive bias has been proposed in slot-based architectures which segregate knowledge about individual objects into nonoverlapping but interchangeable pools of neurons.
The resulting representational modularity can facilitate causal reasoning and prediction for downstream tasks [19, 53].
A grand challenge in computer vision has been to discover the compositional structure of real-world dynamic visual scenes in an unsupervised fashion.
By unsupervised, we mean no segmentation information is provided that speciﬁes which pixels belong together as part of a single object.
Initial efforts focused on single-frame, synthetic RGB images [17, 18, 42, 60], but extending this work to video and more complex scenes proved challenging.
A key insight to further progress was the realization that a color-intensity pixel array is not the only source of visual information readily available, at least not to human perceptual systems.
The human perceptual system extracts motion and depth cues early in the processing stream [11–13, 25, 47].
These cues are correlated with object identities, and can therefore bootstrap the formation of object-centric representations [54].
The recently introduced Slot Attention for Video (SAVi) model [37] leveraged optical ﬂow (frameto-frame motion) as a prediction target to obtain object-centric representations of dynamic scenes involving complex 3D scanned objects and real-world backgrounds.
However, motion prediction alone is insufﬁcient to learn about the distinction between static objects and the background.
Further, in real-world application domains such as self-driving cars, cameras themselves are subject to movement, which globally affects frame-to-frame motion as a prediction signal in non-trivial ways.
In the present work, we describe an enhanced slot-based model for video, referred to as SAVi++ (Figure 2), which obtains qualitative improvements in object-centric representations by exploiting depth signals readily available from RGB-D cameras and LiDAR sensors.
SAVi++ is the ﬁrst slotbased, end-to-end trained model that successfully segments complex objects in naturalistic, real-world video sequences without using direct segmentation or tracking supervision.
A summary of our contributions is as follows:
• We introduce SAVi ++ : an object-centric slot-based video model that makes several key improvements to SAVi [37] by utilizing depth prediction and by adopting best practices for model scaling in terms of architecture design and data augmentation .
• On the multi-object video (MOVi) benchmark containing synthetic videos of high visual and dynamic complexity [20], we ﬁnd that SAVi ++ is able to handle videos containing complex shapes and backgrounds, and a large number of objects per scene.
Improving on SAVi, our approach accommodates both static and dynamic objects and both static and moving cameras.
Finally, we demonstrate that SAVi ++ trained with sparse depth signals obtained from LiDAR enables emergent object decomposition and tracking in real-world driving videos from the Waymo Open dataset [56].
Related work
Object-centric learning A growing body of research is addressing the problem of end-to-end learning of object-centric representations from raw perceptual data without direct supervision.
Slotbased neural networks such as IODINE [18], MONet [5], and Slot Attention [42] rely on a factorized latent space and independent per-object decoders as inductive bias to enable object discovery in a simple auto-encoding setup.
Architectures with stronger inductive biases using ﬁxed object size, presence, or propagation priors have been explored in works such as SQAIR [39] and SCALOR [29], but generally these methods have faced challenges scaling to more complex real-world data when relying on auto-encoding alone.
Our work primarily builds on recent advances in object-centric generative models for video sequences [30, 37, 60, 62, 70].
Different from our approach, these methods have so far been unable to scale to complex real-world multi-object video data.
An alternative class of methods using contrastive learning for object discovery [24, 36, 44, 67], most notably GroupViT [67] and ODIN [24], has recently achieved some success in discovering semantic groupings in real-world images.
However, neither GroupViT nor ODIN model dynamics and typically fail to separate semantically similar object instances in close proximity.
In our work, we follow a generative approach, but instead of tasking the decoder to generate complex visual RGB pixel data, we utilize depth information to bootstrap object-centric learning without direct supervision.
Object discovery in driving scenes A range of recent methods [1, 21, 58, 63] use a multi-stage pipeline of 1) obtaining pseudo ground truth (PGT) segmentation or detection labels via some heuristic, and 2) training a model in a supervised fashion on PGT labels.
While this class of methods achieves some success in discovering and tracking objects in real-world driving scenes, it crucially hinges on the quality of the PGT labels, requiring carefully engineered task-speciﬁc heuristics to extract objects.
Earlier methods solely use clustering heuristics to extract approximate segmentation masks directly from motion trajectories for moving objects [4, 48].
In our work, we instead demonstrate that object segmentation and tracking can emerge in an end-to-end setting on complex real-world data without relying on PGT label generation.
Cross-modal learning For self-supervised object-centric learning from visual data, a range of target modalities and training signals have been explored in the literature.
By using motion cues from optical ﬂow as prediction targets, several recent methods [37, 68] were able to overcome limitations of purely RGB pixel-level generative models, which frequently failed in the presence of complicated textures [33].
However, this advantage is primarily limited to discovery of moving objects.
Utilizing depth targets from a simulator [2] or from sparse LiDAR [21, 58, 63], has been explored in an effort to overcome these limitations.
Different from prior works utilizing multi-stage pipelines and handcrafted heuristics for extraction of pseudo-labels from LiDAR [21, 58, 63], we directly utilize the (sparse) depth signal as target and demonstrate that this can enable emergent object segmentation and tracking on real-world driving data without any additional regularizers or pseudo-labeling techniques.
Scaling strategies for vision models It is common practice to scale architectural capacity with dataset complexity and size, while making use of strong data augmentation when addressing various supervised computer vision tasks [8, 10, 22, 38].
Nonetheless, self-supervised methods for end-toend object discovery have primarily been relying on overly simplistic and low-capacity backbone architectures [18, 42, 70], likely due to the simplicity of datasets and tasks considered in prior work.
By scaling object-centric methods to larger, visually more complex datasets, we ﬁnd that utilizing stronger visual backbone architectures—in combination with data augmentation—can provide substantial beneﬁts.
For simplistic datasets with lower visual complexity (and same number of examples), we found anecdotal evidence in preliminary experiments for the opposite effect: both architecture scaling and data augmentation can negatively affect object discovery performance, likely explaining why prior works have not explored these strategies.
Depth estimation Recent advances in supervised monocular depth estimation (see Ming et al.
[46] for a review) could be combined with our method in future work, for instance using ordinal regression losses [14], transformer architectures [52], or more complex instance-wise decoder architectures [64].
Methods
We begin by providing a brief introduction to Slot Attention for Video (SAVi), which is the starting point for our exploration.
With SAVi++, we introduce several simple yet crucial improvements, which allow us to bridge the gap to complex real-world data.
Our framework is summarized in Figure 2.
3.1 Background
Slot Attention for Video (or SAVi) is a recent state-of-the-art architecture for learning object-centric representations from video with minimal supervision.
We brieﬂy highlight some of its key components below and refer the reader for complete details to Kipf et al.
SAVi can be viewed as an autoregressive encoder-decoder video model with a structured latent state composed of K object slots.
At a given time-step, an encoder ﬁrst encodes the observed video frame [Figure] t i m e to yield high-level image features that are useful for learning about objects.
This is followed by
Slot Attention [42] (the ‘corrector’ ), which updates the slots using these features and encourages individual slots to specialize to different parts of the observation.
The content of each slot is decoded separately using a decoder , which additionally outputs a pixel-level alpha mask to indicate how the decoded values for each slot should be combined.
Together, the mask and decoded slots determine the output of the model at the current time-step from which a loss is computed, e.g., to train the model to predict frame-to-frame motion (optical ﬂow) for this frame.
Slots for the next time-step (for the corrector to update) are obtained by applying a predictor , which can model interactions between slots and learn about object dynamics to predict their future state.
In addition to optical ﬂow prediction, SAVi introduces conditioning that helps reduce uncertainty about the part-whole division into objects by pointing the model to speciﬁc locations. Indeed, in the absence of a speciﬁc downstream task, scene decomposition can be ambiguous and providing additional information as a conditioning signal may help alleviate this.
The conditioning takes place via the slot initializer , which initializes the slots used in the initial video frame.
The initialization may be learned in an unconditional setting (i.e., learn the initial slot states) or obtained by conditioning the initial state on high-level cues such as bounding boxes of objects of interest in the ﬁrst video frame.
This direction of attention or input conditioning helped SAVi to succeed in decomposing more complex visual scenes.
3.2 SAVi++
As SAVi relies on optical ﬂow prediction as its main training signal for object discovery, its application is primarily limited to settings where all objects in a scene have independent motion.
In addition,
SAVi struggled to generalize to scenes with a moving camera, even though the optical ﬂow ﬁeld encodes information about (static) scene geometry in this case.
Here, we identify two key directions for improving SAVi and bridging its capabilities to real-world video data, while preserving its core foundation for learning object representations from video: (1) exploiting depth as a prediction signal, which is readily available in many real-world settings, and (2) utilizing model scaling strategies in terms of encoder improvements and data augmentation, which, despite being commonly used for classic vision problems, are generally underutilized for objectcentric learning.
Our improved approach, called SAVi ++ , successfully segments complex objects in naturalistic, real-world video sequences without using direct segmentation or tracking supervision.
Exploiting depth information Training object-centric models solely using RGB image or video frame reconstruction proves challenging in the presence of complex visual textures, frequently leading to failure modes such as clustering by color or into object-agnostic spatial regions [18, 21].
In SAVi, optical ﬂow was proposed as a prediction signal to mitigate this issue, while still operating on visual
RGB inputs [37].
However, relying solely on optical ﬂow as a prediction target for learning about objects has a clear disadvantage: static objects, which make up the vast majority of visual entities we encounter on a daily basis, are not captured in this signal unless the observer or the entire scene is in
Video
Flow
Depth
M O V iC
M O V iD
M O V i - E (
[Figure]
datasets.
Video
Sparse depth (
[Figure]
C o m p l e x t e x t u r e
M ov i ng ob j ec t s
S t a ti c ob j ec t s
M ov i ng ca m e r a # fr a m e s ( t r a i n )
MOVi-C    234k MOVi-D    234k MOVi-E    234k WaymoOpen    159.6k (
[Figure]
Dataset details and statistics. motion. As a consequence, SAVi fails to represent objects that are at rest, and similarly struggles with scenes observed from a moving camera, as optical ﬂow can prove challenging to model in this case.
Here, we explore depth as a target signal, used in conjunction with ﬂow or even in isolation. Depth estimation has received little attention in slot-based models, yet does not suffer from the limitation of optical ﬂow in datasets with static objects and camera movement. We thus hypothesize that depth may greatly beneﬁt obtaining emergent object decompositions of complex videos.
In terms of practical applicability, we note how depth is a readily-available signal in many real-world settings thanks to the prevalence of RGB-D cameras and LiDAR in settings like self-driving cars [56]. Even in the absence of depth sensing capabilities, this signal can be cheaply estimated from multi-camera systems [40]. In our implementation, we represent the depth signal in image space, which we encode using a log transformation log(1 + d ) , where d is the distance of a pixel to the camera (see Figure 3a). This log-transform puts a stronger emphasis on close-by objects and—in early experiments—we found this form of normalization crucial for reliably training object-centric models using depth targets.
SAVi ++ is then trained to minimize the squared difference between the decoder output and this target signal.
In case of multiple available targets, such as depth and ﬂow, we concatenate the target images along the channel dimension and predict them using an otherwise unchanged model.
For sparse targets such as depth obtained from LiDAR, we ignore any points in the image space for which no signal is present in the computation of the loss.
For LiDAR speciﬁcally, we obtain the x, y, z coordinates of all the LiDAR points in the self-driving car (SDC) world and compute the distance of each of the points from the LiDAR sensor.
We then use the camera and LiDAR calibration parameters to project the LiDAR point distances from the SDC domain to the camera frame.
This projection represents a very sparse approximation of the ground-truth depth signal (Figure 3b).
Scaling strategies Visual complexity present in real-world videos necessitates a different class of encoders than those used for simple synthetic datasets.
Inspired by successful visual backbone architectures for set-based supervised object detection models [7, 32], we use a more capable encoder that utilizes the ResNet34 [22] architecture followed by a transformer encoder [61] (with 4 layers, unless otherwise mentioned).
To avoid computation of batch and/or temporal statistics, we replace the typical batch normalization in ResNet34 with group normalization [65].
We use a stride 1 convolution and use no max-pooling in the ResNet root block.
This results in an overall backbone stride of 8 (as opposed 32 ), which was found to be important for retaining object decomposition capabilities.
Please see the appendix for further architectural details.
Drawing inspiration from training schemes commonly used for real-world vision models [57], we further apply Inception-style cropping as data augmentation.
In particular, we randomly crop a region of each frame with aspect ratio 2 [0 .
33] such that enough of the frame is retained after cropping.
The same crop is applied consistently across all frames and the resulting video is resized to the original resolution.
Flow ﬁelds and depth maps are adjusted accordingly to keep them accurate and spatially aligned with the video frame.
Experiments
The goal of our experimental evaluation is twofold: 1) on synthetic video data of varying complexity we would like to analyze the potential advantages of utilizing a depth signal and model scaling strategies for learning emergent segmentation and tracking, and 2) we would like to investigate whether these improvements enable bridging the gap to complex real-world video data.
Section 4.1 covers both qualitative and quantitative comparisons of SAVi++ against baselines on the synthetic MOVi datasets.
In Section 4.2, we perform an ablation study on SAVi++.
Finally, in
Section 4.3 we demonstrate and analyze results for a SAVi++ model applied to real-world driving videos from the Waymo Open [56] dataset.
Datasets As basis for our experiments, we use videos of different scene and camera complexities (Figure 3c).
We use three synthetic Multi-Object Video (MOVi) datasets (Figure 3a) introduced in
Kubric [20], which are created by simulating rigid body dynamics.
We narrow our investigation to
MOVi datasets with complex naturalistic backgrounds and 3D-scanned everyday objects (variants
C, D, and E).
MOVi-C is generated using a static camera, and all objects (max. 10) are initialized to move independently.
MOVi-D introduces more objects, some of which are dynamic (1-3) and the majority rests statically in the scene (10-20).
Finally, MOVi-E introduces random, linear camera movement.
Each video contains 24 frames sampled at 12 frames per second (fps).
We also train and evaluate SAVi ++ in a real-world driving setting using the Waymo Open dataset (Figure 3b).
Waymo Open is comprised of high resolution video data of 1280 ⇥ 1920 original resolution from a multi-camera system collected by Waymo vehicles [56].
The dataset consists of 798 train and 202 validation scenes of 20s video each, sampled at 10 fps.
We subsample the dataset at 5 fps both for training and validation.
The dataset also includes LiDAR signals that we use to compute sparse depth maps as discussed in Section 3.
Training setup For all our experiments, unless stated otherwise, we resize frames to a height ofpixels while keeping the aspect ratio ﬁxed, resulting in a 128 ⇥ 128 resolution for MOVi datasets, and a resolution of 128 ⇥ 192 for Waymo Open.
We train SAVi ++ for 500k steps on Tensor Processing
Unit (TPU) accelerators with a batch size of 64 using Adam [35].
We train on randomly sampled sub-sequences of only 6 frames using 24 slots for MOVi and 11 slots for Waymo Open.
See appendix for further training details and hyperparameters.
4.1 SAVi ++ improves object-centric learning on complex synthetic video data
We investigate whether the key changes introduced to SAVi [37], which constitute our improved
SAVi++ model, allow us to overcome limitations of SAVi and address the most challenging synthetic multi-object video (MOVi) benchmarks introduced in Kubric [20].
Setup We train all models independently on each dataset variant.
Both SAVi and SAVi++ are trained in a conditional setting where we initialize slots using ground-truth bounding box information in the ﬁrst frame.
We report the same segmentation metrics as in prior work, i.e. Foreground Adjusted
Rand Index (FG-ARI) [27, 51] and Mean Intersection over Union (mIoU).
FG-ARI is a permutationinvariant clustering similarity metric frequently used for evaluating scene decomposition quality.
It compares discovered segmentation masks with ground-truth masks while ignoring any pixels that belong to the background.
It is sensitive to temporal consistency of masks, but insensitive to their ordering.
The mIoU metric is a standard segmentation metric, here adapted for video as in [6].
We note that this implementation is sensitive to the correct ordering of masks, i.e. it also measures whether models used the conditioning signal (here, ﬁrst-frame bounding boxes) correctly.
Baselines Besides comparing to SAVi [37], the most representative prior method for the task we are interested in, we compare against a range of baselines aimed at establishing the difﬁculty of the unsupervised, bounding box-conditioned video object segmentation task: 1) a bounding box copy ( BBox copy ) baseline, which simply repeats the ﬁrst-frame boxes throughout the video, 2) a learned
BBox propagation baseline that does not receive visual inputs, to test for easily exploitable biases in the datasets, 3) k-Means clustering baselines, that cluster the ﬂow and/or depth signal across the video sequence (initialized using the ground-truth object centers in the ﬁrst frame), and 4) a label propagation baseline, that uses visual features to propagate the initial boxes (rendered as rectangular masks) across the video, based on Contrastive Random Walks (CRW) [28].
See appendix for further details.
[Table] be seen in Table 1 and qualitative results on MOVi-E in Figure 4a.
The BBox copy method serves as a trivial baseline which a learning-based approach should outperform.
While the original SAVi model does so on MOVi-C, it clearly fails to model the more complex MOViD and -E datasets.
The BBox copy baselines is—perhaps unsurprisingly—strongest on MOVi-D, where most objects are static.
SAVi++ outperforms this baseline on all datasets, indicating that it learns non-trivial segmentation and tracking capabilities.
Indeed, this advantage does not solely come from ﬁtting certain biases in the datasets, as a learned BBox propagation baseline (using the same predictor as in SAVi++) that does not receive visual input, fails to generalize to unseen evaluation videos.
It is worth noting that neither of the MOVi tasks can easily be solved by simply clustering the target signals, as the results for the k-Means baselines demonstrate.
Compared to CRW it can be seen how SAVi ++ yields markedly better mIoU on MOVi-C and
D, while performance on MOVi-E is similar.
Note that, unlike SAVi ++ , CRW is merely capable of propagating pixel-level annotations across frames in a video and does not by itself produce instance-level object segmentations or corresponding object-representations that could be used for down-stream tasks.
Finally, comparing SAVi++ and SAVi directly, we see that SAVi++ overcomes the primary limitations of SAVi on the harder MOVi-D and -E datasets, both quantatively (Table 1) and qualitatively (Figure 4a), while also improving performance on MOVi-C.
Discussion It is evident that a small number of critical changes to SAVi [37], namely utilizing depth targets, a stronger architecture, and data augmentation, can have dramatic consequences on the ability of this slot-based model to learn emergent object segmentation and tracking in complex video sequences.
The difference between SAVi ++ and SAVi is especially evident for the more complex datasets in our study (e.g., improving the mIoU score on MOVi-E from 30.7% to 47.1%; see also
Figure 4a).
These results demonstrate that SAVi ++ is better suited for various data complexities in terms of object dynamics and camera movement, which are likely to exist in real-world data.
4.2 Ablation study
In this section, we report results of an ablation study to gauge the contribution of the different components of SAVi ++ .
The three main ingredients of SAVi ++ are 1) the use of depth as training target, 2) the extra capacity added to SAVi by including a transformer encoder, and 3) the use of data augmentation.
Figure 4b shows a systematic ablation of each of those components.
Removing the transformer encoder reduces object segmentation quality, yet the degradation in performance is relatively limited.
While data augmentation only has a mild effect on the simpler MOVi-C dataset, it makes a substantial difference on the more challenging datasets, MOVi-D and E. Finally, removing depth targets reduces performance further and is particularly catastrophic on MOVi-E.
In fact, we ﬁnd that training solely using depth targets without relying on predicting optical ﬂow as well (see w/o Flow in Figure 4b) still allows the model to accurately segment and track objects, especially on the more complex MOVi-D and E datasets.
This result is particularly strong on MOVi-E where jointly predicting optical ﬂow presents a difﬁcult task for scenes with camera movement. Further, training on depth targets was very crucial to obtain good performance on the most complex [Figure] [Figure]
[Figure]
synthetic data MOVi-E as demonstrated with the large drop in mIoU when depth only on
4.3 SAVi ++ enables emergent segmentation on real-world driving data
In the previous section we found that solely using depth as a training target can be sufﬁcient to learn emergent object segmentation and tracking.
This ﬁnding provides a strong motivation for scaling this class of methods to real-world data, where the availability of optical ﬂow relies on approximate and potentially inaccurate ﬂow estimation methods, whereas depth can be accurately measured using technologies like LiDAR.
To investigate this possibility, we use the Waymo Open dataset [56], which includes videos obtained from cameras mounted on cars in various trafﬁc environments.
Setup To obtain a depth signal, we project 3D LiDAR points into the camera frame, resulting in a very sparse depth image for each time step (see Figure 3b for examples).
We exclude pixels that do not have a valid LiDAR point when computing the L2 loss in image space.
We train SAVi ++ with 11 slots on 6 frames and evaluate the model on sequences of 10 frames.
Due to the absence of ground-truth segmentation labels in Waymo Open, we quantitatively measure performance compared to ground-truth bounding boxes using three metrics.
The Center-of-Mass (CoM) distance measures the average Euclidean distance between the centroid of the predicted segmentation masks and the centers of the ground-truth bounding boxes.
We report the centroid distance normalized by the maximum achievable distance in the video frame.
Additionally, we separately measure the fraction t =t =t =t =t =t = 25 Conditioning of cases where any sort of segment is predicted when a valid ground-truth box exists, denoted as bounding box recall (B. Recall).
The Bounding Box mIoU (B. mIoU) is analog to mIoU using predicted and ground-truth bounding boxes.
The former are obtained by training a readout MLP to predict bounding boxes from the slot representations.
See appendix for further details.
Baselines We quantitatively compare to the subset of previous baselines that work with (sparse) depth.
Further, we report qualitative results for SAVi++ in the unconditional setting, i.e. without providing ﬁrst-frame bounding boxes to the model to initialize slots, and compare to SIMONe [30] as a representative object-centric video model baseline from the literature.
Results Quantitative results can be seen in Table 2 and qualitative results in Figures 5–6.
We ﬁnd that SAVi ++ markedly outperforms the BBox copy and propagation baselines, as well as the clustering baseline in terms of object tracking.
Further, the bounding box recall is high indicating that valid objects are rarely ignored.
The qualitative results in Figures 5–6 even better reﬂect the signiﬁcance of SAVi ++ ’s performance as well as its potential utility for object-centric representation learning from real-world videos (for SAVi ++ results divided per object category see Table 4).
Our results using sparse depth targets suggest that SAVi ++ does not need complete (i.e. dense) depth supervision. To investigate how accurate this signal needs to be, we explored the degree of sensitivity of SAVi ++ to noise in the depth signal. We trained SAVi ++ with noisy depth targets by applying additive Gaussian noise to the ground-truth sparse LiDAR depth signals with standard deviations of 10cm, 20cm and 40cm. We found that SAVi ++ was able to retain its emergent tracking performance even at the highest considered noise scale of 40cm (see Table 5 in Appendix).
error in %, 3 seeds) from evaluating models on sequences of 10 frames. SAVi ++ HR is a variant trained on higher-resolution ( 256 ⇥ 384 ) video frames.
[Table]
We additionally experimented with removing the bounding box conditioning in SAVi ++ in the initial frame.
Removing this conditioning signal and using a learned initialization together with a simpliﬁed encoder also yielded good object decompositions (see SAVi ++ (unconditional) in Figure 5).
Compared to using plain
SIMONe [30], we observe that SAVi ++ (unconditional) performs markedly better.
Interestingly, modifying the non-autoregressive SIMONe baseline similar to SAVi ++ by predicting sparse depth instead of RGB also showed improvement in object emergence.
This gives further evidence that using depth is suitable for learning object-centric representations from real videos.
Quantitatively, SAVi++ achieves a CoM distance of 6 . 9 ± 0 .
5 while SIMONe (with depth loss) achieves 7 . 4 ± 0 .
2 1 over a sequence of 12 frames at test time, evaluated using Hungarian matching.
We show qualitative results for longer sequences in Figure 6 and in video-format in the supplementary material.
It is worth noting that SAVi ++ was only trained on 6 frames and did not receive any tracking supervision.
Interestingly, we ﬁnd that objects are often consistently tracked until the moment they leave the scene.
At this stage, slots are freed up again and tend to bind to previously unexplained or new objects.
This behaviour indicates that our reported tracking metrics are an underestimation of the capabilities of the model, as such re-binding is not accounted for.
It is, however, conceivable that re-binding events could be identiﬁed post-hoc if one were to use the representations learned by
SAVi ++ for downstream tasks, which is an interesting avenue for future work.
4.4 Limitations
With SAVi ++ , we demonstrated the ﬁrst proof of concept that an emergent object-centric decomposition of real-world complex videos is possible with an end-to-end slot-based approach.
Yet, there is still a lot of room for improvement.
Reliance on conditioning We focused our exploration on the conditional setup where we provided cues in the form of bounding boxes of objects in the ﬁrst frame.
Although the use of such “object hints” may share some similarity to how human visual attention (and how humans parse a visual scene) can be directed via external signals (e.g., via gestures such as pointing), it ultimately limits the practical applicability of our approach.
Preliminary results with unconditional SAVi ++ suggest that this information may not be strictly necessary and could be removed in future research.
Reliance on ground-truth target signals In a similar vein, the reliance of SAVi ++ on ground-truth target signals for training is a limitation that may affect its practical applicability.
Fortunately, LiDAR sensors for depth estimation are readily available in many application domains (such as in robotics and self-driving), and there is also a rich literature on monocular depth estimation.
While estimated, depth (or ﬂow) are expected to be noisier compared to the signals considered in our experiments, our experiment with “noisy depth” offers an initial sign that this may not affect performance much.
Gap to videos recorded in the wild It is also important to point out that although Waymo Open offers a challenging real-world benchmark for learning about objects, its videos are relatively structured compared to real-world videos recorded “in the wild”, and especially heavy on cars, roads, trafﬁc signs, pedestrians, etc.
Other datasets, such as DAVIS [49] or Kinetics [34] offer greater complexity in that regard and it is foreseeable that further development of SAVi ++ will be needed to truly support these. An example of this is that objects in Waymo Open usually do not re-appear, which is an aspect that is currently not explicitly modeled in SAVi ++ (e.g. to ensure that the same object is re-captured by the same slot). More generally, there is substantial headroom to improve the modeling of disappearing and reappearing objects in future work, such as by explicitly modeling object presence [39], or by explicitly attending to past latent states [69]. Gap to supervised approaches Finally, we note how both in the conditional and the unconditional setting, the segmentation and tracking performance, though impressive given the minimal amount of supervision the model receives, still qualitatively lags behind supervised approaches.
Improving on the temporal consistency of object tracks, especially in the unconditional setting, is another promising direction for future work.
Conclusion
We demonstrate that object tracking and segmentation can emerge from utilizing information about scene geometry in the form of depth signals in complex video data with slot-based neural architectures.
We utilize a series of synthetic multi-object video benchmarks with increasing complexity to ﬁnd a simple yet effective set of changes to an existing state-of-the-art object-centric video model (SAVi), allowing us to bridge the gap from synthetic to complex real-world driving videos.
Our work marks a ﬁrst step towards building end-to-end trainable systems that learn to perceive the world in an object-centric, decomposed fashion without relying on detailed human supervision.
While many open challenges remain, this result evidences that object-centric deep neural networks are not inherently limited to simple synthetic environments, and we are excited about the potential for this class of methods to radically reduce the need for human supervision in building scalable perceptual systems for the real world.
Acknowledgements
We would like to thank Ben Caine, Alex Bewley and Pei Sun for assistance with self-driving data.
We are grateful to Jie Tan, Daniel Keysers, David Fleet, Matthias Minderer, Mehdi Sajjadi and Mario
Luˇci´c for general advice and feedback.