Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns
Abstract
We propose a new approach to learn to segment multiple image objects without manual supervision.
The method can extract objects form still images, but uses videos for supervision.
While prior works have considered motion for segmentation, a key insight is that, while motion can be used to identify objects, not all objects are necessarily in motion: the absence of motion does not imply the absence of objects.
Hence, our model learns to predict image regions that are likely to contain motion patterns characteristic of objects moving rigidly.
It does not predict specific motion, which cannot be done unambiguously from a still image, but a distribution of possible motions, which includes the possibility that an object does not move at all.
We demonstrate the advantage of this approach over its deterministic counterpart and show state-of-the-art unsupervised object segmentation performance on simulated and real-world benchmarks, surpassing methods that use motion even at test time.
As our approach is applicable to variety of network architectures that segment the scenes, we also apply it to existing image reconstruction-based models showing drastic improvement.
Project page and code: https://www.robots.ox.ac.uk/~vgg/research/ppmp .
Introduction
Humans have an innate ability to segment individual objects in a picture, but learning this capability with an algorithm usually relies on manual supervision.
In this paper, we consider the problem of learning to segment objects from visual data only — without externally provided labels.
Algorithms for this task usually assume that objects are seen in different configurations and in front of different backgrounds.
They then exploit cues such as the visual consistency and the co-occurrence of characteristic object parts to learn to discover and segment individual object instances.
Most such methods use still images as input and train with a reconstruction objective [18, 22, 36].
They work well on simple synthetic scenes, but they struggle in scenes with more complex visual appearance [26].
This has motivated the development of algorithms that use videos as input and can thus observe the motion of the objects as evidence of their presence.
A common way of using motion for unsupervised learning is to seek for a compact representation to reconstruct the video itself [23, 25, 32, 52].
Effectively, such methods seek for a compressed representation of appearance, but do not sidestep entirely the difficult task of modeling it.
This has motivated authors to look instead at reconstructing the video’s optical flow [27, 56].
In fact, the optical flow measures the motion of the objects directly and is much simpler to model than the objects’ appearance.
In this paper, we propose a method that lies in-between these two classes, i .
e .
single image-based and video-based.
Our model learns to segment objects in still images, and is thus based on appearance, but learns to do so using video as a learning signal , in an unsupervised manner.
The learning process can be summarized as follows.
Given an image, each pixel is assigned to a slot that represents a certain object.
The quality of the assignments is then measured by the coherence of the (unobserved) optical flow within the extracted regions.
Because predicting optical flow from a still image is intrinsically ambiguous, the method models distributions of probable flow patterns within each region.
The idea is that (rigid) objects generate characteristic flow patterns that can be used to distinguish between them.
Note that, because the segmentation network is based on a single image, it will learn to partition all objects contained in the scene, not just the ones that actually move in the video, i .
e .
solving an object instance segmentation problem, rather than motion segmentation.
We derive closed-form distributions for the flow field generated by rigid objects moving in the scene.
We also derive efficient expressions for the calculation of the flow probability under such models.
The problem of decomposing the image into a number of regions is cast as a standard image segmentation task and an off-the-shelf neural network can be used for it.
As our method uses videos to train an image-based model, we introduce two new datasets which are straightforward video extensions of the existing image datasets CLEVR [24] and C LEVR T EX [26].
These datasets are built by animating the objects with initial velocities and using a physics simulation to generate realistic object movement.
Our datasets are constructed with the realistic assumption that not all objects are moving at all times.
This means that motion alone cannot be used as the sole cue for objectness and reflects scenarios such as a workbench where a person only interacts with a small number of objects at a time.
Empirically, we validate our model against several ablations and baselines.
We compare our approach to existing unsupervised multi-object segmentation methods achieving state-of-the-art performance.
We demonstrate particularly strong performance in visually complex scenes even with unseen objects and textures at test time.
Our experiments in comparison to image-based models and, in particular, adding our motion-aware formulation to existing models shows substantial improvements, confirming that motion is an important cue to learn objectness.
Furthermore, we show that our learned segmenter, which operates on still images, produces better segmentation results than current video-based methods that use motion information at test time. Finally, we also apply our method to real-world self-driving scenarios where we show superior performance to prior work.
Related Work
Multi-object decomposition.
Learning unsupervised object segmentation for static scenes is a well-researched problem in computer vision [5, 8, 10, 11, 12, 13, 16, 17, 22, 33, 36, 44, 45, 55].
These methods aim to decompose the scene into constituent parts, e .
g .
the different foreground objects and the background.
Glimpse-based methods [8, 13, 22, 33] find input patches (glimpses) that contain the objects in the scene.
These methods learn object descriptors that encode their properties ( e .
g .
position, number, size of the objects) using variational inference, composing glimpses into the final picture.
More related to ours are approaches that learn per-pixel object masks [5, 10, 11, 12, 36].
MoNet [5] and IODINE [18] employ multiple encoding-decoding steps to sequentially explain the scene as a collection of regions.
Slot Attention [36] uses a multi-step soft clustering-like scheme to find the regions simultaneously.
In all cases, learning is posed as an image reconstruction problem.
In order to align learnable slots with semantic objects, models have to make efficient use of a limited representation available for each region, such as learning to only explain visual appearance.
This principle, however, is difficult to extend to visually complex data [26] and relies on custom specialized architectures.
Instead, our method allows for any standard segmentation architecture to be used, which we train to predict regions that are most likely described by rigid motion patterns.
Video-based multi object decomposition.
Another line of work extends the unsupervised object decomposition problem to videos [20, 23, 25, 28, 29, 30, 32, 43, 51, 53, 54, 58].
Many of these methods work mainly with simpler datasets [29, 43, 58] and require sequential frames for training.
For example, SCALOR [23] is a glimpse-based method that discovers and propagates objects across frames to learn intermediate object latents.
SIMONe [25] processes the whole video at once, learning both temporal scene representation and time-invariant object representations simultaneously.
Slot
Attention for Video (SAVi) [27] poses the multi-object problem as optical-flow prediction using sequential frames as input.
The internal slot-attention mechanism drives the network to learn regions that move in a simple and consistent manner.
Different to our work, it does not assume a specific motion model but relies on directly regressing the flow.
It is computationally more expensive and struggles when only one or few frames are available.
Unsupervised video object segmentation.
Unsupervised video object segmentation (VOS) is a popular problem in computer vision [9, 14, 31, 37, 42, 48, 50, 56, 57], that focuses on extracting the most salient object in the scene.
Many of the approaches treat the problem as a motion segmentation task as the background typically shows a dominant motion independent of the salient object.
Motion
Grouping [56] employs the Slot Attention architecture to reconstruct optical flow from itself, avoiding appearance information entirely.
Another related line of work [7, 39, 40, 49] employs approximate motion models.
These approaches rely on a point estimate of the motion model parameters.
In contrast, we adopt a more principled probabilistic approach, placing a prior on the motion parameters and integrating them out.
To deal with flow outliers that do not conform to a rigid motion model,
Mahendran et al.
[39] use a histogram matching-based loss and GWM [7] over-segments the scene relying on spectral clustering to produce a binary segmentation during inference.
Instead, we model the noise in our formulation directly.
Finally, Meunier et al.
[40] rely on flow as input, limiting the method to videos only.
Method
Let a frame I ∈ R 3 × H × W of a video and its optical flow f ∈ R 2 × H × W be defined on the H × W lattice.
The optical flow is a local summary of the motion from one frame to the next.
We use it to supervise a network Φ that, given the (single) image I as input, predicts soft assignments of each pixel to up to K different image regions, outputting an H × W collection of probability vectors Φ( I ) ∈
ˆ∆
H × W K ⊂ [0 , 1] K × H × W , where ˆ∆ K is the K − 1 -dimensional simplex.
The quality of the regions is measured based on how likely they contain flow patterns typical of the motion of independent objects.
In more detail, we represent the predicted image regions by a hard K -way pixel assignment (mask) m ∈ ∆ H × W K ⊂ { 0 , 1 } K × H × W , where ∆ K is the space of K -dimensional one-hot vectors.
Each mask is a sample from the categorical distribution output by the network, i .
e .
m ∼ p Φ ( m | I ) =
Categorical[Φ( I )] .
Note that there is one categorical distribution for each pixel and that these are mutually independent.
We then assume that the flow depends only on the regions, in the sense that p Φ ( f , m | I )
=
p ( f | m ) p Φ ( m | I ) , where p ( f | m ) is a model of the distribution of the flow field given the regions.
The likelihood of the modeled Φ is bounded by:
[Equation]
Furthermore, inspired by ELBO, we regularize the model’s prediction p Φ ( m | I ) by taking its KL divergence from a uniform prior p 0 ( m ) , obtaining the learning objective
[Equation]
Next, we introduce the closed-form motion model p ( f | m ) in Eq.
(1) and then explain how the
Gumbel-Softmax trick can be used to train the network.
Approximate motion models for optical flow.
We now turn to describing the models of motion used in our work, which play a role in assessing the likelihood of optical flow p ( f | m ) .
Optical flow measures the coordinate change of pixels between neighboring frames, which arises due to the motion of the camera and objects.
We consider rigid-body motion of some object k .
Let x tk , y tk ∈ R n k be the spatial locations of the pixels belonging to region/object k at time t , where n k is the number of pixels in the region.
For convenience, we stack the coordinates in a single vector Ω tk = ( x tk , y tk ) ∈ R 2 n k .
The pixels comprising this object undergo coordinate change from Ω tk to Ω t +1 k , giving rise to the optical flow for this object as f k = Ω t
+1 k − Ω tk .
We assume this underlying 3D rigid-body motion can be approximated using a linear 2D parametric model Π θ with parameters θ , so that:
[Equation]
where ϵ captures the residual error of the approximation.
Several forms of models are available (see [1, 4] for an overview).
Here, we consider two such models: the translation of an object within the camera plane, and an affine motion, given respectively by linear functions: [Equation] [Equation]
[Equation]
 
[Equation]
where we use 1 n k is a vector of n k ones and matrix P k contains the coefficients of the model.
The affine model supports object rotation, scaling and shearing in addition to translation.
It is often a sufficient approximation to real-world optical flow, provided the objects are rigid, convex, and mainly rotating in-plane.
We can then use the motion equations (2) to construct the distribution p ( f | m ) by assuming a prior on the motion parameters and by marginalizing over it.
Specifically, denote by m k the k -th slice of the tensor m encoding the regions ( i .
e .
the mask of the k -th region).
We assume that regions are statistically independent and decompose the log-likelihood p ( f | m ) as:
[Equation]
Assuming that each object has i.i.d.
parameters θ k with a Gaussian prior N ( θ ; µ, Σ) , and assuming ϵ is a zero-mean noise with variance σ 2 , Eq.
(2) gives marginal optical flow likelihood for segment k :
[Equation]
where I is the identity matrix.
A practical issue with Eq.
(5) is that, if segment k contains n k = (cid:80) i ( m k ) i pixels, then the covariance matrix P k Σ P ⊤ k + σ
I has dimension 2 n k × 2 n k .
Inverting such a matrix in the evaluation of the Gaussian log-density is very slow except for very small regions.
Furthermore, it is not obvious how to relax Eq.
(4) to support gradient-based learning, e .
g .
through the Gumbel-Softmax approximation.
We solve these problems in the next section.
Expressions for the likelihood.
We now derive expressions for Eq.
(5) which are efficient and that lead to a natural relaxation for use in the Gumbel-Softmax sampling.
Given the definitions
F k = f k − Π µ (Ω k ) + Ω k and Λ = Σ − 1 , we can rewrite Eq.
(5) as:
[Equation]
where S k = 1 / σ 2 P ⊤ k P k +Λ .
The significant advantage of this form is that it involves the computation of the inverse and determinant of matrix S k , whose size is only 2 × 2 (for the translation model) or 6 × 6 (for the affine one), instead of the much larger 2 n k × 2 n k .
We can more explicitly introduce the dependency on the region assignments m by defining selector matrices R k ∈ { 0 , 1 } 2 n k × 2 n (with n = (cid:80) k n k = HW ) that extract the x and y coordinates of the pixels that belong to the corresponding region, i .
e .
Ω k
= R k Ω .
We can then also write F k = R k F and P k
= R k P .
Furthermore, the product of the selectors L k =
R ⊤ k R k ∈ { 0 , 1 } 2 n × 2 n can be written directly as a function of the assignment m as L k ( m ) =
diag( m k , m k ) .
Plugging these back in Eq.
(6), we obtain expressions involving L k only:
[Equation]
Translation-only model.
Further simplifications are possible for specific models.
For instance, for the translation-only model, assuming that Λ =
diag(1 /τ 2 , 1 /τ 2 ) then S k = diag( n k + 1 /τ 2 , n k + 1 /τ 2 ) and, after some calculations, we obtain the expression: − log p ( f | m ) = n log 2 πσ 2 + (cid:88)
[Equation]
Affine model.
For the affine model, the expression for − log p ( f | m ) does not simplify as much.
Still, by exploiting the structure of matrix P aff k , we can reduce the calculations to the computation of inverse and determinant of small 3 × 3 matrices, which can be implemented efficiently in closed form.
Please see the Appendix for the derivation.
Unless otherwise stated, the mean vector is set to µ = (1 0 0 0 1 0)
⊤ centering the prior on the no-motion point.
Gumbel-softmax.
In order to train the network using gradient descent, we need a differentiable version of loss (1).
To do so, we use the re-parametrizable Gumbel-softmax relaxation [21, 38].
The Gumbel-softmax relaxation replaces categorical samples m ∈ ∆ H × W K from the distribution p Φ ( m | I ) =
Categorical[Φ( I )] with continuous samples ˆ m ∈
ˆ∆
H × W K from the distribution
GumbelSoftmax[Φ( I )] .
We take N = 3 samples from this distribution to evaluate the expected negative log-likelihood, further reducing variance.
Then we simply replace ˆ m for m in Eq.
(1), leading to differentiable quantities.
Post-processing.
Eq.
(5) naturally encourages the model to form larger regions to explain parts of the scene that move in a consistent (under the assumed prior) manner.
However, we find that this can also lead to the model grouping together objects that only coincidentally move together ( e .
g .
all objects mostly falling due to gravity in one of the datasets).
Furthermore, optical flow is ambiguous around object edges and occlusions.
To address both the object grouping and occlusion boundary issue, we use a simple post-processing step.
We isolate connected components in the model output, selecting the K largest masks, discarding any that are smaller than 0.1% of the image area, and combining the left-over and discarded ones with the largest mask overall.
Warp loss.
Occasionally, the optical flow used to supervise our model can be noisy as it is estimated by other methods.
This noise is also unlikely to be isotropic as some surfaces are easier to estimate that others.
Rather than supporting heterogeneous noise and approximation error (Eq.
, we instead prioritize parts of the scene covered by higher-quality flow.
To this end, we introduce an additional loss term that simply enforces consistency between adjacent frames I 1 , I 2 .
In particular, it warps the predicted mask distributions Φ( I 1 ) , Φ( I 2 ) using the optical flow, weighted by the error of warping the frames themselves, as follows:
[Equation]
where f 1 ( · ) indicates warping by forward optical flow
f 1 (or backward b 2 ).
The symmetrized KL divergence, d ( · ) , measured agreement between predicted and warped mask distributions, weighted by the absolute error of the warped frames normalized in [0 , 1] .
While the use of this term is not central to our method, we include it to show how tolerance to noisy optical flow can be improved.
We do not use the warp loss (Eq.
(8)) in our experiments, unless otherwise indicated by (WL).
In that case, the final loss is simply sum of the two terms: L β + L warp .
Experiments
Our method lies in between image-based and video-based segmentation approaches, because it uses videos for supervision, but trains an image segmentation network that operates on still images only.
We thus evaluate our approach under a number of settings.
Firstly, we evaluate how well motion can be used to supervise an object instance segmenter that operates on still images.
Secondly, we compare such a segmenter to state-of-the-art object segmentation methods that use motion also at test time (and are thus advantaged compared to our model).
We conduct further analysis to validate our modelling assumptions and the model’s reliance on the quality of the optical flow used for supervision.
Finally, we apply our method to a real-world setting.
4.1 Experimental setup
Datasets.
We evaluate our method on video and still image datasets.
For video-based data, we use the Multi-Object Video (MOVi) datasets, released as part of Kubric [19].
Specifically, we employ
MOVi-{A,C,D,E} versions.
MOVi-A is similar to CLEVR [24] in terms of visual complexity and contains videos of 3–10 falling objects on a simple, gray background.
MOVi-C is significantly more challenging, as it features scanned, textured, common objects on top of backgrounds textured using
HDR images.
In MOVi-D, the number of objects is increased up to 23.
In MOVi-E, the camera is additionaly moving.
We use a resolution of 128 × 128 and the provided ground truth optical flow.
To evaluate our method on still images, we use CLEVR [24] and C LEVR T EX [26] benchmark suites.
Both consist of images depicting 3–10 objects.
CLEVR images are simpler with uniformly colored objects with metallic or rubbery materials.
C LEVR T EX features more diverse objects with complex textures applied.
We also use the OOD and CAMO test sets from C LEVR T EX benchmark.
OOD contains out-of-distribution shapes and textures.
CAMO has camouflaged objects where the same texture is sampled for objects and the background.
Since our method requires optical flow during training, we extend the implementation of [26] to generate video datasets of CLEVR and C LEVR T EX scenes, where a subset of objects contained in each scene are sliding, rolling and colliding based on a physics simulation (Fig. 1).
We generate 10k sequences for M OVING C LEVR T EX and 5k for M OVING CLEVR, where we retain 1000 andsequences, respectively, for validation.
Each sequence is 5 frames long.
Dataset details can be found in the Appendix.
The evaluation is performed on the original CLEVR and C LEVR T EX test sets.
We also evaluate our method on the real-world KITTI [15] benchmark which depicts street scenes captured from a moving car.
We follow the set up of [3], using 147 videos for training and evaluate on the instance segmentation subset which contains 200 annotated validation frames.
Metrics.
Following prior work [26, 27], we measure performance using two metrics.
FG-ARI is the Adjusted Rand Index measured on foreground pixels only (selected using the ground-truth segmentation).
Mean Intersection-over-Union (mIoU), is measured through Hungarian matching and averaged across the number of predicted or ground truth components, whichever is higher.
When evaluating on videos, we calculate these metrics per-frame.
Network architecture.
Our method can employ any image segmentation network architecture and train from scratch.
Unless otherwise specified, we use Mask2Former [6], using only its semantic segmentation.
Following prior work [27, 36], we use a 6-layer CNN backbone on synthetic datasets and ResNet-18 for KITTI.
We also experiment with Swin-tiny transformer [35] as the backbone.
We use 11 transformer queries which become K = 11 slots on CLEVR, C LEVR T EX , and MOVi-A/C.
On MOVi-D/E, we set K = 24 and K = 22 on KITTI.
The model takes approximately 48h to train on a single A30 24GB GPU.
All training details and hyper-parameters are included in the Appendix.
4.2 Unsupervised multi-object segmentation in images
In Table 1, we evaluate our method on the CLEVR [24] and C LEVR T EX [26] benchmarks and compare to prior work.
Our method outperforms image models based on appearance reconstruction on both metrics (mIoU and FG-ARI) and across all datasets.
The performance gap increases on the visually complex C LEVR T EX , OOD, and CAMO variants, demonstrating the strong inductive bias that motion provides during training, especially when the objects are camouflaged.
Note that, in this setting, our model is advantaged compared to the other models in Table 1, as it can observe (through the loss) the optical flow of the training scenes. For this reason, we also train the optical flow-based, unconditional SAVi [27] model. Nevertheless, we find that despite having access to motion information during training, SAVi does not surpass appearance-only models, likely due to only having access to single frames at test time.
[Table]
[Figure]
O u r
[Figure]
Post-processing helps improve results further.
As shown in Fig.
2, separating connected components in post-processing distinguishes objects that might be assigned to the same mask and suppresses boundary segments that tend to group difficult occlusion boundaries.
For a fairer comparison, we also test if this post-processing improves the results of other methods but only obtain mixed results.
4.3 Unsupervised multi-object segmentation in video
We now evaluate our approach on video segmentation, where motion is available at test time. We report the performance of our method in Table 2 compared to video-based models: SCALOR [23],
[Table]
[Figure]
[Table]
[Figure]
unconditional SAVi [27], and GWM [7]; the latter two also use optical flow supervision.
It is important to note that SAVi and SCALOR make predictions jointly over all frames of a video, that allows them to actually see the objects in motion at test time.
Despite this comparison being unfair to our approach, which operates on a single frame at a time, we achieve competitive results.
On the visually simpler MOVi-A, our method has more than 10% lead over SAVi in mIoU, but performs slightly worse in terms of FG-ARI.
On the more complex MoVi-C/D/E datasets our model shows strong performance, outperforming prior work on both metrics, with the performance gap again increasing with data complexity.
Finally, we experiment with a version of the model using a deeper backbone (Swin) and the warp loss (Eq.
Though not necessary to achieve state-of-art results, this drastically improves performance on all metrics and dataset versions.
In Fig.
3, we also compare these models qualitatively, demonstrating that the different objects are overall better captured by our model with more refined boundaries, which explains the higher mIoU.
[Figure]
[Figure]
[Table]
[Table]
4.4 Model ablations
Optical flow.
In Table 3a, we replace the ground-truth optical flow used so far in our experiments with the one estimated by SMURF [46].
The additional noise impacts our model’s accuracy, as evidenced by the significant drop in mIoU (but comparable FG-ARI scores).
Motion model.
In Table 3b, we compare our affine motion model to the simpler translation-only one. We observe that the ability to describe complex motion patterns with an affine model improves performance over the translation model, which can only represent translation in the camera plane.
Motion awareness.
Finally, we investigate the effectiveness of our objective in combination with existing appearance-based methods.
The goal of this experiment is to understand the advantage of using motion information during training (if available) and to decouple the effect of our formulation from the choice of architecture. To this end, we employ our objective on top of two models based on appearance reconstruction, GNM [22] and SA [36], with no other modifications.
We train GNM and SA with and without our loss on videos (M OVING C LEVR T EX ) and evaluate on the corresponding single-image test sets of the C LEVR T EX suite. In Table 3c, we compare these models respectively to the original methods trained on static images (C LEVR T EX ).
We find that when trained on video data (M OVING C LEVR T EX ), without our loss, both GNM and SA struggle.
We attribute this to the reduced number of scenes in M OVING C LEVR T EX compared to C LEVR T EX .
However, we note that using our loss significantly improves the performance of the appearance methods, suggesting the effectiveness of exploiting motion information through our formulation.
[Table] We now turn to assessing our model’s performance in a realworld setting.
We follow the setting of Bao et al.
[3] and evaluate on KITTI [15], using RAFT [47] to estimate optical flow and ResNet-18 as the backbone of our model, trained from scratch.
We lower the input resolution for our model from 368 × 1 , 248 [3] to 288 × 960 , which enables us to fit on a single GPU.
We evaluate at 96 × 320 resolution.
As we show in Table 4, our method outperforms prior work on the challenging real-world setting.
In the same table, we also consider our method with an additional warp loss term, which further boosts performance.
We also experiment with a transformer-based backbone (Swin) which is also pre-trained using self-supervision.
Although, not necessary to show state-of-art result, this significantly improves real-world performance.
4.6 Limitations
Motion is sometimes insufficient to distinguish different objects, for instance because they do not move or because they move similarly.
In principle, this should not matter if sufficient motion diversity is observed in the training data as a whole; in practice, our model occasionally merges different object at test time, which we address partly in post-processing.
Further improvements could be obtained by choosing a more informative prior p 0 ( m ) in Eq.
(1), to capture other desirable properties of objects, such as compactness and connectedness.
Figure 4 shows some failure cases where the affine motion model struggles to capture strong perspective effects caused by non-smooth depth changes in the object geometry.
This could be addressed by modeling the object geometry (depth) and the ensuing complex flow patterns.
Alternatively, this could be dealt with using a hierarchical segmentation model that can account for geometric discontinuities and self-occlusions.
Hierarchical segmentation would also help with pronounced non-rigid motion ( e . g . humans dancing or animals running) as motion could be explained at the level of object parts.
Conclusions
We have presented a method that bridges the gap between image-based and video-based scene decomposition, in that it requires only a single image as input, yet exploits motion cues available in videos during training.
In comparison to prior work on image-based multi-object segmentation, our approach shows that motion provides useful objectness cues, especially as the visual complexity of a scene increases.
Different from video-based approaches, however, our model operates on still images and does not rely on motion to detect or refine objects, which makes it more generally applicable.
Finally, we deviate from the common objective of image or flow reconstruction and, instead, model the problem by only predicting regions likely to contain affine flow patterns.
This does not require a specialized architecture, thus any segmentation network is suitable for this task.
Our approach achieves state-of-the-art performance on multiple image and video benchmarks, in simulated and real-world settings, validating the paradigm of training image models using motion.
Broader impact.
Our work introduces a principled method for unsupervised multi-object segmentation.
The work is mainly evaluated on 3D simulated datasets that do not contain people or personal information.
Additionally, we evaluate on KITTI, a real-world self-driving dataset, which occasionally contains images of pedestrians.
Consent cannot be obtained in this case, but we follow the KITTI terms of usage.
We build on top of open source projects, respecting licenses and release all code, trained models and datasets for research purposes.
Currently, the application of this approach is mainly limited to simulated imagery.
Although the results on KITTI show promise, the immediate broader impact of our work in real-world scenarios, beyond the research community, is limited.
Acknowledgements L. K. is funded by EPSRC Centre for Doctoral Training in Autonomous
Intelligent Machines and Systems EP/S024050/1.
S. C. is supported by a scholarship sponsored by Facebook.
I. L., C. R. and A. V. are supported by European Research Council (ERC) grant 2020CoG-101001212 UNION. I. L. and C. R. are also funded by EPSRC grant VisualAI EP/T028572/1.