B RIDGING
THE
G AP
TO
R EAL -W ORLD
O BJECT -
C ENTRIC
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans naturally decompose their environment into entities at the appropriate level of abstraction to act in the world.
Allowing machine learning algorithms to derive this decomposition in an unsupervised way has become an important line of research.
However, current methods are restricted to simulated data or require additional information in the form of motion or depth in order to successfully discover objects.
In this work, we overcome this limitation by showing that reconstructing features from models trained in a self-supervised manner is a sufficient training signal for object-centric representations to arise in a fully unsupervised way.
Our approach, DINOSAUR , significantly out-performs existing image-based object-centric learning models on simulated data and is the first unsupervised object-centric model that scales to real-world datasets such as COCO and PASCAL VOC.
DINOSAUR is conceptually simple and shows competitive performance compared to more involved pipelines from the computer vision literature.
I NTRODUCTION
Object-centric representation learning has the potential to greatly improve generalization of computer vision models, as it aligns with causal mechanisms that govern our physical world (Sch ¨ olkopf et al., 2021; Dittadi et al., 2022).
Due to the compositional nature of scenes (Greff et al., 2020), object-centric representations can be more robust towards out-of-distribution data (Dittadi et al., 2022) and support more complex tasks like reasoning (Assouel et al., 2022;
Yang et al., 2020) and control (Zadaianchuk et al., 2020; Mambelli et al., 2022; Biza et al., 2022).
They are in line with studies on the characterization of human perception and reasoning (Kahneman et al., 1992; Spelke & Kinzler, 2007).
Inspired by the seemingly unlimited availability of unlabeled image data, this work focuses on unsupervised object-centric representation learning.
Most unsupervised object-centric learning approaches rely on a reconstruction objective, which struggles with the variation in real-world data.
Existing approaches typically implement “slot”-structured bottlenecks which transform the input into a set of object representations and a corresponding decoding scheme which reconstructs the input data.
The emergence of object representations is primed by the probabilistic permutation symmetry (Bloem-Reddy & Teh, 2020) of the set bottleneck of models like Slot Attention (Locatello et al., 2020) that group together independently repeating visual patterns across a fixed data set.
While this approach was successful on simple synthetic datasets, where low-level features like color help to indicate the assignment of pixels to objects, those methods have failed to scale to complex synthetic or real-world data (Eslami et al., 2016; Greff et al., 2019;
Burgess et al., 2019; Locatello et al., 2020).
To overcome these limitations, previous work has used additional information sources, e.g. motion or depth (Kipf et al., 2022; Elsayed et al., 2022).
Like color, motion and depth act as grouping signals when objects move or stand-out in 3D-space.
Unfortunately, this precludes training on most real-world image datasets, which do not include depth annotations or motion cues.
Following deep learning’s mantra of scale, another appealing approach could be to increase the capacity of the Slot Attention architecture.
However, our experiments (Sec. 4.4) suggest that scale alone is not sufficient to close the gap between synthetic and real-world datasets.
We thus conjecture that the image reconstruction objective on its own does not provide sufficient inductive bias to give rise to object groupings when objects have complex appearance.
But instead of relying on auxiliary external signals, we introduce an additional inductive bias by reconstructing features that have a high level of homogeneity within objects.
Such features can easily be obtained via recent self-supervised learning techniques like DINO (Caron et al., 2021).
We show that combining such a perceptual similarity (Dosovitskiy & Brox, 2016) with existing grouping modules such as Slot Attention leads to models that significantly out-perform other image-based object-centric methods and bridge the gap to real-world object-centric representation learning .
The proposed architecture DINOSAUR ( DINO and S lot
A ttention U sing R eal-world data) is conceptually simple and highly competitive with existing unsupervised segmentation and object discovery methods in computer vision.
R ELATED W ORK
Our research follows a body of work studying the emergence of object-centric representations in neural networks trained end-to-end with certain architectural biases (Eslami et al., 2016; Burgess et al., 2019; Greff et al., 2019; Lin et al., 2020; Engelcke et al., 2020; Locatello et al., 2020; Niemeyer & Geiger, 2021; Singh et al., 2022a).
These approaches implicitly define objects as repeating patterns across a closed-world dataset that can be discovered e.g. via semantic discrete- or set-valued bottlenecks. As the grouping of low-level features into object entities is often somewhat arbitrary (it depends for example on the scale and level of detail considered), recent work has explored additional information sources such as video (Kosiorek et al., 2018; Crawford & Pineau, 2020; Jiang et al., 2020; Veerapaneni et al., 2020; Weis et al., 2021; Singh et al., 2022b; Traub et al., 2022), optical flow (Kipf et al., 2022; Elsayed et al., 2022; Bao et al., 2022), text descriptions of the scene (Xu et al., 2022) or some form of object-location information (e.g. with bounding boxes) (Kipf et al., 2022).
In contrast, we completely avoid additional supervision by leveraging the implicit inductive bias contained in the self-supervised features we reconstruct, which present a high level of homogeneity within objects (Caron et al., 2021). This circumvents the scalability challenges of previous work that rely on pixel similarity as opposed to perceptual similarity (Dosovitskiy & Brox, 2016) and enables object discovery on real-world data without changing the existing grouping modules. In the computer vision literature, the task of structuring natural scene without any human annotations has recently also enjoyed popularity. Ji et al.
; Ouali et al.
Cho et al.
; Gansbeke et al.
; Zadaianchuk et al.
Hamilton et al.
(2022) tackle unsupervised semantic segmentation , i.e. splitting the scene into semantically coherent regions.
These methods are typically separated into several independent steps: first, dense representations for each pixel in the image are refined from a pretrained self-supervised model; second, those representations are clustered on the whole dataset to assign a class category to each pixel.
Instead, we combine those steps into a simple method that groups semantically meaningful regions jointly with learning region representations.
Another line of work focuses on unsupervised object localization (Vo et al., 2020; 2021;
Simeoni et al., 2021; Melas-Kyriazi et al., 2022; Wang et al., 2022).
These methods often group pre-trained, self-supervised features of individual images into initial object proposals, and then aggregate and cluster these object proposals across images to assign them discrete labels.
Self-training can then be used on top to achieve multi-object detection on natural scenes.
In contrast, our method uses the reconstruction loss to discover objects and makes little to no assumption on the underlying features and how they represent objects.
M ETHOD
Our approach essentially follows the usual autoencoder-like design of object-centric models and is summarized in Figure 1: a first module extracts features from the input data (the encoder), a second module groups them into a set of latent vectors called slots , and a final one (the decoder) tries to reconstruct some target signal from the latents.
However, our method crucially differs from other approaches in that instead of reconstructing the original inputs, the decoder is tasked to reconstruct the encoder’s output .
We now describe our choices for each module, with additional details in App.
C.1.
ViT as Encoder We use a vision transformer (ViT) (Dosovitskiy et al., 2021) as the feature extractor.
ViTs process the image as a set of N square patches, where each patch is linearly mapped to an embedding vector.
After adding a learned positional encoding to each patch, the set of patches is transformed through a series of alternating self-attention (Vaswani et al., 2017) and MLP blocks.
The output tokens h ∈ R N × D feat of the last ViT block form the input to the Slot Attention stage.
[Figure]
Slot Attention Grouping The grouping stage of our model uses Slot Attention (Locatello et al., 2020) to turn the the set of ViT features h into a set of K slot vectors z ∈ R K × D slots .
This follows an iterative process where slots compete for input features using an attention mechanism, starting from randomly sampled initial slots.
We largely use the original Slot Attention formulation (including
GRU (Cho et al., 2014) and residual MLP modules), with one difference: we do not add positional encodings on the ViT features before Slot Attention, as we found the ViT’s initial position encodings to be sufficient to support spatial grouping of the features.
Additionally, we add a small one-hidden-layer
MLP that transforms each ViT feature before Slot Attention.
Self-Supervised Pre-Training with DINO
The architecture as described so far does not suffice for learning object-centric representations on real-world data.
Our intuition is that a randomly initialized feature space makes it difficult for the grouping module to find commonly occurring patterns, in turn sending poor gradients to the feature encoder.
Therefore, we start with a generic self-supervised pre-training step of the encoder.
Self-supervised learning techniques have been successful in learning powerful representations for vision tasks such as classification and object detection (Chen et al., 2020b; Grill et al., 2020;
He et al., 2022).
While we demonstrate that different self-supervised approaches work for the purpose of object-centric learning (Sec. 4.4), in this work we mainly apply the DINO method (Caron et al., 2021).
DINO applies a student-teacher distillation scheme for training, namely by encouraging representations resulting from multiple crops and distortions of the input image to match between student and teacher network.
The resulting features have been shown to be highly discriminative of objects (e.g. Amir et al., 2021), making DINO an attractive candidate for pre-training.
Training Signal We hypothesize that reconstruction on the pixel level produces too weak of a signal for object-centricness to emerge; the task focuses (at least initially) strongly on low-level image features such as color statistics.
This quickly decreases the reconstruction error, but the resulting model does not discover objects beyond datasets where objects are mostly determined by distinct object colors.
Instead, if we had a signal that required higher-level semantic information to reconstruct, there would be pressure on the slots to efficiently encode this information as well.
Luckily, we have such a signal available in the form of the features produced by the DINO pre-trained ViT.
Thus, instead of images, the slots are trained to reconstruct the input features well, by minimizing the following loss:
[Equation]
As we use the intermediate features h as the reconstruction target, training the full architecture would lead to training collapse (because of trivial solutions minimizing the loss).
We therefore freeze the pretrained ViT encoder and only optimize Slot Attention and the decoder.
This loss is akin to a perceptual similarity (Dosovitskiy & Brox, 2016), optimizing the slots in a space more semantic than pixel space.
Feature Decoding As we apply feature instead of image reconstruction as the training objective, we need a decoder architecture suitable for this purpose.
To this end, we consider two different designs: a MLP decoder that is applied independently to each slot, and a Transformer decoder (Vaswani et al., 2017) that autoregressively reconstructs the set of features.
We describe both options in turn.
The MLP decoder follows a similar design as the commonly used spatial broadcast decoder (Watters et al., 2019).
Each slot is first broadcasted to the number of patches, resulting in a set of N tokens for each slot.
To make the spatial positions of the tokens identifiable, a learned positional encoding is added to each token.
The tokens for each slot are then processed token-wise by the same MLP, producing the reconstruction ˆ y k for slot k , plus an alpha map α k that signifies where the slot is active.
The final reconstruction y ∈ R N × D feat is formed by taking a weighted sum across the slots:
[Equation]
The advantage of this simple design is its computational efficiency: as the MLP is shared across slots and positions, decoding is heavily parallelizable.
The Transformer decoder (Vaswani et al., 2017) reconstructs features y jointly for all slots in an autoregressive manner.
In particular, the feature at position n is generated while conditioning on the set of previously generated features y <n and the set of slots z : y n =
Decoder( y <n ; z ) .
This decoder design is more powerful than the MLP decoder as it can maintain global consistency across the reconstruction, which might be needed on more complex data.
In exchange, it loses the property of independent decoding of slots, often thought to be an architectural bias necessary for object-centric learning.
However, we find that object-centric slots still emerge in our experiments.
We note that Transformer decoders have also been previously explored by SLATE (Singh et al., 2022a) and STEVE (Singh et al., 2022b), but to reconstruct the discrete token map of a VQ-VAE (van den Oord et al., 2017).
Evaluation Object-centric methods are commonly evaluated by inspecting masks associated with each slots.
Previous approaches reconstructing to image-level typically use the decoder’s alpha mask for this purpose; for the MLP decoder, we also make use of this option.
The Transformer decoder does not produce an alpha mask.
Instead, we have two options: the attention masks of Slot Attention (used by SLATE), or the decoder’s attention mask over the slots.
We found that the latter performed better (see Sec. B.5), and we use it throughout.
As the masks from feature reconstruction are of low resolution, we bilinearly resize them to image resolution before comparing them to ground truth masks.
E XPERIMENTS
Broadly, we pursue two goals: 1) demonstrating that our approach significantly extends the capabilities of object-centric models towards real-world applicability (Sec. 4.1), and 2) showing that our approach is competitive with more complex methods (Sec. 4.2 and Sec. 4.3).
Additionally, we ablate key model components to find what is driving the success of our method in Sec. 4.4.
Datasets In this work, we consider two synthetic and two real-world image datasets.
As synthetic datasets, we use the MOVi datasets (Greff et al., 2022), recently introduced as challenging testbeds for object-centric methods.
In particular, we use the variants MOVi-C and MOVi-E, which contain around 1 000 realistic 3D-scanned objects on HD backgrounds.
For our purposes, the main difference is that MOVi-C contains 3–10, and MOVi-E 11–23 objects per scene. Note that we treat the video-based
MOVi datasets as image datasets by randomly sampling frames. As real-world datasets, we use PASCAL VOC 2012 (Everingham et al., 2012) and MS COCO 2017 (Lin et al., 2014), commonly used for object detection and segmentation.
Whereas PASCAL VOC contains many images with only a single large object, COCO consists of images with at least two and often dozens of objects.
Both datasets represent a significant step-up in complexity to what object-centric models have been tested on so far.
Tasks We evaluate on the following tasks: object discovery (Sec. 4.1), i.e. finding instances of objects by inferring pixel masks around them, object localization 1 (Sec. 4.2), i.e. finding object location and size by predicting bounding boxes, and unsupervised semantic segmentation (Sec. 4.3), separating the image into semantically consistent labeled regions.
For the latter, we consider two variations: object segmentation, where only foreground objects should get segmented and labeled, and scene decomposition, where each pixel of the image has to be labeled with a semantic class.
To be comparable to prior work, each task follows a slightly different evaluation protocol (see App. D).
As such, numbers between the different tasks are not directly comparable.
[Table]
Training Details We train DINOSAUR using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 4 · 10 − 4 , linear learning rate warm-up of 10 000 optimization steps and an exponentially decaying learning rate schedule.
Further, we clip the gradient norm at 1 in order to stabilize training and train for 500k steps for the MOVI and COCO datasets and 250k steps for PASCAL VOC.
The models were trained on 8 NVIDIA V100 GPUs with a local batch size of 8 , with 16-bit mixed precision.
For the experiments on synthetic data, we use a ViT with patch size 8 and the MLP decoder.
For the experiments on real-world data, we use a ViT with patch size 16 and the Transformer decoder.
We analyze the impact of different decoders in Sec. 4.4.
The main results are averaged over 5 random seeds; other experiments use 3 seeds.
Further model details can be found in App.
C.1.
4. U NSUPERVISED O BJECT D ISCOVERY
The task of object discovery – splitting an image into a set of masks that cover semantically related parts of the image – is the most frequent way object-centric models are evaluated (Burgess et al., 2019; Locatello et al., 2020).
Our goal in this section is two-fold: 1) demonstrating that previous object-centric methods fail to produce meaningful results on real-world datasets and struggle even on synthetic datasets, and 2) showcase how our approach of incorporating strong pre-trained models results in a large step forward for object-centric models on both kinds of datasets.
Metrics
As common in the object-centric literature, we evaluate this task using foreground adjusted rand index (FG-ARI), a metric measuring cluster similarity.
Additionally, we compute a metric based on intersection-over-union (IoU), the mean best overlap (mBO) (Arbel ´ aez et al., 2014).
mBO is computed by assigning each ground truth mask the predicted mask with the largest overlap, and then averaging the IoUs of the assigned mask pairs.
In contrast to ARI, mBO takes background pixels into account, thus also measuring how close masks fit to objects.
On datasets where objects have a semantic label attached (e.g. on COCO), we can evaluate this metric with instance-level (i.e. object) masks, and semantic-level (i.e. class) masks.
This allows us to find model preferences towards instance- or semantic-level groupings.
Baselines
We compare our approach to a more powerful version of Slot Attention (Locatello et al., 2020) which is based on a ResNet encoder and has been shown to scale to more complex data (Elsayed et al., 2022).
Further, we compare with SLATE (Singh et al., 2022a), a recent object-centric model that uses a discrete VQ-VAE (van den Oord et al., 2017) as the feature extractor and a Transformer as the decoder.
We refer to App.
C.2 for details about baseline configurations.
As it can be hard to gauge how well object-centric methods perform on new datasets solely from metrics, we add one trivial baseline: dividing the image into a set of regular block masks.
This block pattern (see Fig. 12) thus shows the performance of a method that only follows a geometric strategy to group the data, completely ignoring the semantic aspects of the image.
Familiar to practitioners, this is a common failure mode of object-centric methods, particularly of Slot Attention
Table 1 and Fig.
on the challenging MOVi datasets.
Our model achieves good performance on both MOVI-C and
MOVi-E. Visually, the quality is on par with reports from more involved methods that use temporal information and weak supervision (Elsayed et al., 2022; Singh et al., 2022b).
Results on Real-World Datasets (Table 2 and Fig.
3) As expected, Slot Attention and SLATE can not handle the increased complexity of real-world data and degrade to non-semantic grouping patterns.
In contrast, our method captures a variety of objects of different size, appearance and shape.
MOVi-C
MOVi-E
I m a g e
DINOSAUR
S l o t
A tt e n ti on
G r ound
T r u t h
DINOSAUR
S l o t
A tt e n ti on
To the best of our knowledge
[Table]
we are the first to show a successful version of an object-centric model on unconstrained real-world image data in the fully unsupervised setting. In fact, even more restricted real-world datasets such as for autonomous driving have so far only been tackled using additional supervision signals (Elsayed et al., 2022). Our result represents a significant step-up in complexity of what object-centric methods can handle
[Table] [Table] In the computer vision literature, the task of object localization refers to finding the location of one or more objects that repeatedly appear in a collection of images (Cho et al., 2015).
In particular,
We report CorLoc and detection rate and compare with rOSD
(Vo et al., 2020), DINO-Seg (Simeoni et al., 2021), LOST (Simeoni et al., 2021), DeepSpectral (Melas-Kyriazi et al., 2022) and
TokenCut (Wang et al., 2022)
[Table]
objects have to be discovered by proposing bounding boxes that “correctly localize” them.
An object counts as correctly localized if its bounding box has an IoU-overlap of at least 50% with a proposed bounding box.
Like prior work (Vo et al., 2020; Melas-Kyriazi et al., 2022), we report the fraction of images on which at least one object was correctly localized (CorLoc), and the average fraction of objects correctly localized per image (detection rate).
As our method produces masks, we use the tightest bounding box around each mask as the proposal.
We evaluate on the trainval split of
PASCAL VOC 2012 and COCO-20k, as is standard in the literature (Vo et al., 2020; Simeoni et al., 2021).
COCO-20k is a subset of COCO 2014 filtered to contain no instances annotated as crowd.
Results (Table 3)
The results show that our method manages to focus on the salient parts of the data.
On the majority of images, at least one object is captured by a slot.
Also, our method reaches results comparable or better to what has been reported in the literature on this task.
Interestingly, zero-shot transferring a model trained on COCO to PASCAL yields better results than a model trained on
PASCAL itself; we explain this by the size of the training data, which is around 10 × larger for COCO.
4. U NSUPERVISED S EMANTIC S EGMENTATION
In this task, each pixel of the image has to be assigned a semantic class label.
As in the unsupervised setting, no direct correspondence to ground truth classes are given, a successful model needs to output segments that consistently cover the same semantic content on different images.
We report results on two different versions of this task: in object segmentation, only foreground classes have to segmented; in scene decomposition, the background has to be split into semantic categories as well.
For object segmentation, we evaluate on PASCAL VOC 2012 and COCO 2017, with instance masks merged to semantic class masks.
For scene decomposition, we evaluate on the COCO-Stuff dataset (Caesar et al., 2018), in two different variants: COCO-27 uses a reduced set of 27 coarse classes from the COCO class hierarchy, whereas COCO-171 uses the full set offoreground and 91 background classes available in the dataset.
To the best of our knowledge, we are the first to report results on this challenging setting.
We evaluate with mean intersection-over-union over classes (mIoU) and per-pixel accuracy (pAcc).
As our method does not directly output labels required for semantic segmentation, we obtain them by running k-means clustering on features associated with each slot after training the model, then assigning clusters to ground truth classes by maximizing IoU using Hungarian matching.
Similar ways of evaluating are commonly used in the literature (Gansbeke et al., 2021; Melas-Kyriazi et al., 2022); we refer to App.
C.3 for more details.
Results on Object Segmentation (Table 4a)
This task is not naturally suited for our method, which decomposes the scene into semantically meaningful parts without distinguishing fore- and background.
We still obtain competitive results with recently published work (Gansbeke et al., 2021;
Melas-Kyriazi et al., 2022).
It is also important to note that GroupViT, reporting the strongest results on this task, use additional supervision from text.
Moreover, DeepSpectral, MaskDistill and COMUS employ additional steps of training segmentation networks which improves results and allows them
[Table]
to run at the original image resolution.
In contrast, the masks we evaluate are only of size 14 × 14 ; we leave it to future work to improve the resolution of the produced masks.
Results on Scene Decomposition (Table 4b)
We are competitive with the state-of-the-art method
STEGO on the setting with 27 coarse classes.
On the novel setting with all 171 classes, our method improves upon STEGO by 3.2 IoU points.
This demonstrates that DINOSAUR is able to handle the full complexity of real-world scenes better.
4. A NALYSIS
In this section, we analyze different aspects of our approach: the importance of feature reconstruction, the impact of the type of pre-training, and the role of the decoder.
Additional experiments are included in App.
B, for example analyzing the influence the number of slots has on the inferred masks.
Importance of Feature Reconstruction (Table 5)
First, we analyze the choice of image vs. feature reconstruction under different encoder training modes: training from scratch, freezing, or finetuning DINO pre-trained weights.
To this end, we use a ViT-B/16 encoder on COCO with Slot Attention’s spatial broadcast decoder for image reconstruction and MLP decoder for feature reconstruction.
Training from scratch results in divergence of the training process with both training signals.
Predictably, feature reconstruction with finetuning results in training collapse, as explained in Sec. 3.
Image reconstruction under both the frozen and finetuning setting fails to yield meaningful results, as visible by the striped mask patterns (see Fig. 6).
Thus, even when starting from features that are highly semantic, image reconstruction does not give enough signal towards semantic grouping.
The only setting that “works” (in the sense of discovering objects) is feature reconstruction with a frozen ViT encoder.
Second, we analyze whether the success of our approach stems from using pre-trained encoders, or from feature reconstruction. To do so, we train ResNet34 encoders from scratch using feature reconstruction as a training signal, and find that the performance is overall similar to using a pretrained ViT encoder (see Table 12).
This suggests that the feature reconstruction signal is the key component that allows object-centricness to emerge on real-world data.
We expand in App.
B.2.
Choice of Pre-Training Method (Table 6)
We compare the object discovery performance on
COCO using different backbones and pre-training schemes.
ViT encoders use the Transformer decoder, ResNet50 encoders the MLP decoder.
Besides DINO, we evaluate three other self-supervised pre-training methods: MoCo-v3 (Chen et al., 2021), MSN (Assran et al., 2022), and MAE (He et al., 2022).
All choices perform better than training with image reconstruction. Remarkably, all selfsupervised schemes perform well for the task of object discovery (cf.
Fig. 19). This demonstrates that self-supervised training on ImageNet translates into a useful general bias towards discovering objects. In comparison, supervised training via ImageNet classification is a worse source of features.
Moreover, there is a gap between ViT and ResNet features: visual inspection shows that ResNettrained models seem to rely on a mix of semantic and geometric grouping of the image (cf.
Fig. 20).
[Table] [Figure]
[Table] [Table]
In Table 8, we extend the results, also studying differently sized ViTs. We conclude from this experiment that self-supervised training is important for real-world grouping to emerge, but that the specific method is less important, and that scaling the encoder yields better results.
Choice of Decoder (Table 7)
We compare the choice of MLP vs. Transformer decoder for object discovery.
Both options use a ViT-B/16 encoder.
Generally, we find that the MLP decoder is better on ARI whereas the Transformer decoder is better on mBO.
For MOVi-C, visual inspection (see Fig. 13) shows that the Transformer tends to produce tighter masks and a cleaner background separation, but uses excess slots to split objects.
For PASCAL and COCO, what’s striking is the Transformer decoders’ improvement of 9–10 class mBO.
This reveals that the Transformer decoder is biased towards grouping semantically related instances into the same slot, which we suggest stems from its global view on the image, but also from the generally increased expressiveness of the architecture (cf.
the lower reconstruction loss).
In contrast, the MLP decoder is able to separate instances better (see Fig.
5 and also Fig. 18), which is reflected in the higher ARI scores.
Researching how different decoder designs affect semantic vs. instance-level grouping is an interesting avenue for future work.
In App.
B.3 and App. B.4, we further study different decoder properties.
C ONCLUSION
We presented the first image-based fully unsupervised approach for object-centric learning that scales to real-world data.
Our experiments demonstrate significant improvements on both simulated and real-world data compared to previously suggested approaches and even achieve competitive performance with more involved pipeline methods from the computer vision literature.
This work only takes a first step towards the goal of representing the world in terms of objects.
As such, some problems remain open.
One issue concerns semantic vs. instance-level grouping.
As evident from the presented examples, our approach covers a mix of both, with semantically related objects sometimes being grouped into a single slot.
While we found the type of decoder to influence this behavior, more fine-grained control is needed.
A related issue is the detail of the decomposition, e.g. whether objects are split into parts or stay whole.
We found this to be dependent on the number of slots, with a fixed number often not being appropriate (see App.
B.1).
How models can dynamically choose a suitable level of detail while staying unsupervised but controllable will be an important challenge to fully master the ambiguities the real world inherently presents.
In this work, we exclusively focused on object discovery.
Future work could examine the properties of the learned slot representations, for instance robustness to distribution shifts, generalization and usefulness for downstream tasks.
Another interesting direction is how our approach can be combined with image generation to build flexible and compositional generative models of natural data.
R EPRODUCIBILITY S TATEMENT
Appendix C.1 contains detailed information about the DINOSAUR architecture and all hyperparamers used for all experiments.
Appendix C.2 contains details about how baselines were trained.
Appendix D contains information about task evaluation and datasets.
All datasets used in this work (MOVi, PASCAL VOC 2012, COCO) are public and can be obtained on their respective web pages.
Source code will be released upon acceptance latest.
A PPENDIX A E XTENDED R ELATED W ORK
Unsupervised Semantic Segmentation.
The challenging task of decomposing natural scenes into semantically meaningful regions without any human annotations can be tackled from different directions.
One line of work (Ji et al., 2019; Ouali et al., 2020; Cho et al., 2021; Gansbeke et al., 2021; 2022; Hamilton et al., 2022; Zadaianchuk et al., 2022) first learns a dense representation for each pixel in the image.
Such dense representations can then be clustered to form discrete categories and thus allow to decompose the scene to semantically coherent regions.
Methods that try to learn and cluster dense representations without additional assumptions (Ji et al., 2019; Ouali et al., 2020) were shown to be effective only on small-scale datasets with narrow visual domains.
Furthermore, incorporation of geometric consistency (Cho et al., 2021) or unsupervised saliency detection (Gansbeke et al., 2022;
Zadaianchuk et al., 2022) simplifies dense representation learning and the grouping task.
Additionally, using self-supervised DINO representations (Caron et al., 2021) by contrasting them (Hamilton et al., 2022) or to self-train a semantic segmentation model (Zadaianchuk et al., 2022; Gansbeke et al., 2022; Melas-Kyriazi et al., 2022) allows obtaining a more accurate image decomposition to semantic categories.
Object Localization is another research direction that also aims at structuring the scene by predicting object information such as bounding boxes and object category from images.
Recent works (Vo et al., 2020; 2021; Simeoni et al., 2021; Melas-Kyriazi et al., 2022; Wang et al., 2022) showed the potential benefit of using networks pre-trained in a supervised self-supervised way for both object localization (i.e, bounding box prediction) and object clustering.
First, rOSD (Vo et al., 2020; 2021) was using supervised classifier features to localize salient objects in the image.
Next, LOST (Simeoni et al., 2021) showed that dense DINO features could be used for localization of the most prominent object in the image.
They proposed some heuristic estimate of the object seed and showed that patches that correlated with the seed patch can form an object proposal mask.
Finally, DeepSpectral (MelasKyriazi et al., 2022) and TokenCut (Wang et al., 2022) showed that additional transformation of the original DINO features to a spectral embedding can improve the quality of the object proposals significantly.
While the proposed methods can be combined with self-training of object detectors like
R-CNN, they are limited by the quality of the set of salient objects used as pseudo-labels.
In contrast,
DINOSAUR splits each scene to a set of object proposals enabling multi-object scene decomposition without an additional recognition network.
Architectures making use of slot-like components have also enjoyed popularity aside from the research explicitly focused on object-centric learning.
For instance, the Perceiver model learns to distill a large set of input tokens from different modalities into a reduced set of embeddings, which can be seen as slots (Jaegle et al., 2022).
Prominent recent successes of large-scale multi-modal modeling such as Flamingo (Alayrac et al., 2022) or Gato (Reed et al., 2022) are based on the
Perceiver architecture.
In dynamics modeling, recurrent independent mechanisms (Goyal et al., 2021) use a competition between independent modules to read inputs into a slot-structured latent state. Recent work on contrastive learning such as SlotCon (Wen et al., 2022) or Odin (H´enaff et al., 2022) also can be seen as learning slot representations, although with a focus on semantic (class-based) slots rather than instances. In the case of SlotCon this is due to the reliance on prototypes which naturally lead to semantically focused representations whereas in the case of Odin the contrastive objective does not encourage differentiation of one or multiple instances. Further, Odin is positioned as a strategy to pretrain a backbone which can then be finetuned on downstream tasks and is not intended to be an unsupervised method that directly leads to instance level grouping.
Finally, ORL Xie et al. (2021) addresses the reliance of many contrastive methods on ImageNet-specific biases which allows intra-image contrasting of the same object with different augmentations applied. This is done by identifying correspondences of objects in different images using models trained via intra-image contrasting and subsequently training the model using inter-image contrasting by selecting objects of the same type from different images. [Figure]
B A DDITIONAL R ESULTS
In this section, we include several experiments that did not fit into the main part.
First, Tableand Table 9 complement the encoder and decoder analysis from Sec.
4.4 with results on object localization and unsupervised semantic segmentation.
We also refer to App.
E, where example model predictions for these analyses are included.
Second, in Sec.
B.1, we discuss how the number of slots affects the quantitative and qualitative results of our method.
Third, in Sec.
B.2, we compare the effect of training the encoder from scratch vs. using a pre-trained encoder.
Fourth, in Sec.
B.4, we analyze how the Transformer decoder behaves with different encoders.
Fifth, in Sec.
B.5, we compare the performance of masks from slot attention and the Transformer decoder. Sixth, in Sec.
B.6, we investigate the question of whether or not our method performs semantic segmentation on the MOVi datasets.
Finally, in Sec.
B.7, we add an unsupervised segmentation baseline for object discovery on the MOVi datasets.
B. N UMBER OF S LOTS
MOVi-E (Table 10)
On the synthetic MOVi-E dataset, we found that changing the number of slots can greatly impact performance.
We attribute this to the strong impact object splitting (distributing an object over two or more slots) can have on the ARI score.
While setting the number of slots to the maximum number of objects that can appear in the scene (23 on MOVi-E) allows the model to fully cover crowded scenes, most scenes have fewer objects, leading to impaired performance on those. We show examples on MOVi-E with 11 slots for all tested methods in Fig.
PASCAL (Table 11) & COCO (Fig. 8, Fig. 9)
On both PASCAL VOC 2012 and COCO, our method performs well over a range of slots, as long as a minimum number of slots is given.
Slot
Attention, in contrast, fails to produce meaningful results for all tested number of slots.
For our [Table] [Table] method on COCO, the analysis shows that there are different sweet spots for instance-level (9–11 slots and class-level segmentation (7 slots).
Visual inspection (see Fig. 7) suggests that the optimal number of slots (unsurprisingly) depends on the complexity of the specific image: using few slots tends to lead to grouping of related objects, whereas using many slots leads to oversegmentation of objects into parts.
Choosing the number of slots per-dataset rather than per-image thus only optimizes results for the average scene.
An interesting question for future work is how models could deal with the variance of visual complexity and number of instances that they are faced with in real-world situations.
[Figure]
B. T RAINING THE E NCODER FROM S CRATCH
In this experiment, we attempt to train an encoder from scratch instead of using a pre-trained, frozen encoder.
As discussed in Sec.
4.4, we were not able to train a ViT encoder from scratch using feature reconstruction as the objective.
ViTs are known to be difficult to optimize, requiring careful training recipes. Combining ViT training with the slot attention module might thus also need non-trivial efforts to train successfully. Here, instead of a ViT, we train a ResNet34 backbone (the same ResNet architecture we use for our Slot Attention baseline) for 300 000 steps. As the reconstruction target,
[Table] [Table] [Figure] we still use features from a pre-trained ViT. Notably, we were not able to optimize the ResNet from scratch using the Transformer decoder, as the model degraded to only use a single slot for the full image.
Instead, we used the MLP decoder. For each dataset and type of target features, we compare with using the target feature network also as a frozen, pre-trained encoder (i.e. our usual setup). Table 12 lists the results. Example predictions on COCO are included in Table 20 and Table 21.
We find that training from scratch is a viable alternative to using a pre-trained encoder, in some cases slightly improving on the pre-trained encoder. We think this is because training from scratch allows adaptation to the domain, whereas the pre-trained encoder may to some degree be misaligned. On COCO, the improvement may also partially stem from the ResNet’s higher spatial resolution ( 28 × 28) — although the decoder’s mask resolution is the same ( 14 × 14 ).
However, using a frozen encoder has computational advantages: no backward pass is needed, and only one forward pass through the encoder is needed (instead of one through the encoder being trained, and one through the encoder providing the target features). Overall, this experiment shows that the crucial element making our approach work is feature reconstruction. While the type of encoder and the use of pre-trained weights can yield further improvements, they are less important compared to the use of feature reconstruction.
[Figure]
[Table]
Extending our decoder analysis from Sec. 4.4, we investigate how the capacity of the decoder and the slot dimensionality affects the object discovery performance on COCO.
In particular, we compare the
MLP with the Transformer decoder: for the MLP decoder, we use a 4-layer MLP and test hidden layer dimensionalities 1024, 2048, and 4096. For the Transformer decoder, we test different layer numbers 2, 3, and 4. For each decoder setting, we test it under slot dimensionalities 64, 128, 256, and 512.
We show the results in Fig.
The general trend of Table 7 stays unchanged: the MLP decoder has higher ARI (indicating better instance grouping), whereas the Transformer decoder has higher mBO (indicating better mask quality). For the MLP decoder, we find that scaling the MLP decoder has slight benefits in terms of ARI, and that it is robust to the slot dimensionality (with the exception of slot dimensionality 512). For the Transformer decoder, scaling appears to have little impact. However, we find that the Transformer decoder is sensitive to the slot dimensionality: setting it too low (64, 128) can cause the model to collapse to just use one or two slots to model the whole image.
Setting it too high (512) causes training instabilities (i.e. NaNs or loss spikes). These instabilities under slot dimensionality 512 also occur with the MLP decoder. We conjecture this might be related to the use of 16-bit mixed precision and the attention mechanism of slot attention, but we did not investigate further.
B. T RANSFORMER D ECODER WITH D IFFERENT E NCODERS
In Table 13, we compare the Transformer decoder with different ViT encoders on COCO object discovery.
We find that scaling up the encoder leads to better results.
However, specifically combining the Transformer decoder with the ViT-B/8 encoder leads to poor results – the MLP decoder has no such issues (cf.
the results on the MOVi datasets in Table 1).
Visual inspection shows that the model collapses to use just one or two slots for the whole image.
We attribute this to the high capacity of the decoder , which leads to it being able to reconstruct the token map well while using the conditioning information from only a few slots.
Note that the capacity of the decoder is related to the [Figure] encoder through the token dimensionality, and the number of tokens (which has strong influence on a
Transformer’s capacity as it directly influences the amount of memory and computation available throughout the blocks).
Indeed, when reducing the decoder’s capacity by either reducing the feature dimensionality (ViT-S/8) or the number of tokens the decoder operates on (using bicubic interpolation on the feature map to reconstruct), the problem disappears.
In principle, the issue could also arise from optimization becoming unstable due to switching to a larger model, but the loss curves do not suggest such instabilities.
For ViT-B/8, we also list results for scaling the target token map by a factor of 0 .
5 (from 784 to 196 tokens) and by a factor of 0 .
75 (to 441 tokens).
Object-centricness does no longer emerge for
ViT-B/8, which we attribute to the high capacity of the decoder: when reducing the capacity (by reducing the feature dimensionality or the number of tokens), the problems disappear.
[Table]
M ASK T YPE
As noted in Sec.
[Table]
the Transformer decoder offers a choice between attention masks from the slot attention module [Table] and attention masks from the decoder.
In Table
[Table]
we compare the two types of masks and find that decoder attention masks are consistently better on all tasks [Table] and especially outperform the masks from slot attention on segmentation.
Note that we could in principle also use masks from slot attention with the MLP decoder.
We point to Singh et al.
(2022b)
[Table]
who analyze this choice and find that alpha masks produced by reconstruction are superior to attention masks
[Table]
[Table]
B. D OES DINOSAUR P ERFORM S EMANTIC G ROUPING ON MOV I ?
On the MOVi datasets, DINOSAUR seemingly separates objects into slots,
i [Table]
e. instance grouping.
As DINOSAUR exhibits some bias towards semantic grouping on real-world datasets, an interesting question is whether our approach does in fact also perform semantic grouping on MOVi.
If this would be the case, it would be hard to detect by evaluating on the original MOVi datasets, because objects of the same category rarely appear on the same image on those datasets.
To answer this question
[Table]
we designed two variants of the MOVi-C dataset.
In the first, only objects of the same semantic category appear in each image.
In the second, each image contains the exact same object several times.
All other characteristics of the dataset stay unchanged.
If our method would perform semantic grouping on MOVi, objects would be grouped together on those datasets.
In Table
[Table]
we list the results for training on MOVi-C and testing on the new “semantic” variants.
We find that the results do not change significantly compared to evaluating on the original MOVi-C dataset, indicating that our method does in fact perform instance grouping on the MOVi datasets.
[Table]
[Table]
An interesting question is to what degree unsupervised semantic segmentation methods (which we compare to on real-word data in
Sec. 4.3) can already solve the object discovery task on synthetic datasets designed to benchmark object-centric learning methods.
If this were the case, it would showcase that those datasets are flawed in the sense that instance-level grouping (one goal of object-centric learning) is not needed to tackle them.
To this end, we test the state-of-the-art scene segmentation method STEGO (Hamilton et al., 2022) on the MOVi datasets.
STEGO assigns each pixel to one of several clusters/classes, where the clustering is performed over the full dataset.
We run STEGO with 22 and 27 clusters, corresponding to the 17 semantic categories of the MOVi dataset (see Sec. D), plus 5 or 10 clusters which could be used for the backgrounds.
We use a DINO ViT-B/8 backbone for feature extraction, other hyperparameters are listed in Sec.
C.2.
To evaluate on the object discovery task, we use the segmentation masks produced by STEGO as if they were object masks.
That is, each cluster/class mask is used as a separate object mask.
We list quantitative results in Table 16 and show examples in Fig.
STEGO is able to segment objects and background to some degree.
However, objects are not properly split, with a single cluster often covering several objects.
Compared to our method, STEGO’s masks are also significantly more cluttered and less geometrically consistent.
Quantitatively, on MOVi-C, STEGO performs slightly better than our Slot Attention baseline on mBO, but significantly worse on the ARI metric (reflecting the poor object splitting behavior).
On MOVi-E, STEGO’s mBO is also worse than Slot Attention’s, indicating that STEGO struggles with complex scenes of many smaller objects.
Overall, this result suggests that unsupervised semantic segmentation methods can not satisfactorily solve the object discovery task, justifying the existence of object-centric learning methods.
[Figure]
[Figure]
[Figure]
GO C I MPLEMENTATION D ETAILS C. A RCHITECTURE AND H YPERPARAMETERS
In the following, we describe implementation details of the DINOSAUR architecture.
The hyperparameters for our experiments are given in Table 17.
ViT Encoder
We use the Vision Transformer implementation and provided by the timm library 2 .
Depending on the experiment, we use the following configurations: ViT-S/16 (token dimensionality 384, 6 heads, patch size 16), ViT-S/8 (token dimensionality 384, 6 heads, patch size 8), ViT-B/16 (token dimensionality 768, 12 heads, patch size 16), ViT-B/8 (token dimensionality 768, 12 heads, patch size 8).
The specific timm model names (for DINO weights) used are vit small patch16 224 dino (ViT-S/16), vit small patch8 224 dino (ViT-S/8), vit base patch16 224 dino (ViT-B/16), vit base patch8 224 dino (ViT-B/8).
All models use 12 Transformer blocks, linear patch embedding and additive positional encoding.
The output of the last block (not applying the final layer norm) is passed on to the Slot Attention module and used in the feature reconstruction loss, after removing the entry corresponding to the CLS token.
We always use images of size 224 × 224 as input to the ViT; this yields N = 14 2 = 196 patches for patch size 16 and N = 28 2 = 784 patches for patch size 8.
If the image size is not equal to 224 , we use bicubic interpolation for resizing.
For the pre-trained weights, we use the timm library for DINO and third-party releases for MoCo-v3 3 , MSN 4 , and MAE 5 .
Slot Attention
The Slot Attention module largely follows the original formulation from Locatello et al.
After applying a layer norm, the set of N input features is transformed to slot dimensionality D slots by a two-layer MLP with hidden size equal to the feature size D feat , followed by another layer norm.
An initial set of K slots is randomly sampled from a normal distribution parametrized by learnable mean µ ∈ R D slots and log standard deviation log σ ∈ R D slots .
The slots are then updated over several iterations, where each iteration involves a layer norm step on the current slots, an attention step where slots compete for input features, updating the slots using a GRU (Cho et al., 2014), and applying a residual two-layer MLP.
The query/key/value projections’ size is the same as the slot dimensionality D slots ; they do not use a bias.
The key-value dot product is scaled by
D − 0 .
5 slots , and an epsilon of 10 − 8 is added to the softmax weights for stability reasons.
The MLP has a hidden size equal to 4 · D slots .
On MOVi-C and MOVi-E, we use 11 and 24 slots respectively, which corresponds to the maximal number of objects that can appear in the scene plus one slot that can capture the background.
On
PASCAL VOC and COCO, we use 6 and 7 slots respectively, as we found those values to perform quantitatively well and give visually appealing results (cf.
App. B.1).
MLP Decoder
We use a four-layer MLP with ReLU activations, with output dimensionality
D feat + 1 , where the last dimension is used for the alpha value.
The MLP has hidden layer sizes of 1024 for the MOVi datasets, and 2048 for COCO.
A learnable positional encoding of size N × D slots is added to the slots after broadcasting them to the number of patches N .
Transformer Decoder
We use the standard decoder design from Vaswani et al.
(2017) with pre-normalization (Xiong et al., 2020), conditioning on the set of slots output by the slot attention module.
Similar to ImageGPT (Chen et al., 2020a), the decoder autoregressively reconstructs N patch features of dimensionality D feat starting from the top-left and proceeding row-by-row over the image (“raster order”).
To this end, its initial input is the set of target features shifted back by one, removing the last feature and inserting a learned “beginning-of-sentence” feature at the start.
Each Transformer block then consists of self-attention on the set of tokens (using causal masking to prevent attending to future tokens), cross-attention with the set of slots, and a residual two-layer
MLP with hidden size 4 · D feat .
Before the Transformer blocks, both the initial input and the slots are linearly transformed to D feat , followed by a layer norm.
The Transformer thus operates on tokens of dimensionality D feat .
The output of the last Transformer block is directly used as the reconstruction y .
We do not apply dropout as we did not find it to be beneficial.
Also, we do not add positional encoding as the target features forming the input already contain positional information (from the
ViT encoder).
As mentioned in the main text, we use the attention mask from the decoder as the slot mask used for evaluation.
In particular, the cross attention step involves a softmax over the slots for each of the N tokens, resulting in N attention weights per slot, which can be reshaped to form a √ N × √ N mask.
Intuitively, each mask entry measures how important the corresponding slot was for reconstructing the patch feature at that position.
For the evaluation, this mask is bilinearly resized to the image size.
We use the attention mask from the last decoder block, which we found to perform best against the alternatives of using masks from earlier blocks or averaging the masks across blocks.
Details for Specific Experiments
Here we list settings for experiments aside from our main experiments.
• Image vs. Feature Reconstruction.
In this experiment, we train COCO and combine a ViTB/16 encoder with different decoders: Slot Attention’s spatial broadcast decoder (as detailed in App.
C.2) for image reconstruction and the MLP decoder for feature reconstruction (as detailed above).
Image reconstruction uses 128 × 128 as the target resolution.
For finetuning, we use a lower learning rate of 4 · 10 − 5 for the encoder, compared to 4 · 10 − 4 for Slot Attention and the decoder.
For training from scratch with feature reconstruction, the target signal is produced by a separate pre-trained ViT, not the encoder being trained from scratch.
For training a ResNet34 encoder from scratch with feature reconstruction, we use the same encoder architecture as for slot attention (described in App. C.2).
All models use 7 slots.
Other training and architecture settings are the same as for the main experiments.
• Choice of Pre-Training Method.
In this experiment, we use different pre-trained networks for feature extraction and reconstruction.
The specific timm model names used are resnet50 (ResNet50, supervised), vit small patch16 224 (ViT-S/16, supervised), vit base patch16 224 (ViT-B/16, supervised), vit small patch16 224 dino (ViT-S/16, DINO), vit base patch16 224 dino (ViT-B/16, DINO). For MoCo-v3, MSN and MAE, we use the ViT-B/16 implementation of timm with third-party weight releases (referenced above).
All networks were pre-trained on the ImageNet dataset.
The ResNet50 encoder uses the features output from the second last block (feature levelin timm), resulting in a 1024 -dimensional feature map of size 14 × 14 .
Like Kipf et al.
, we add linear positional encoding to this feature map before Slot Attention.
The ViT encoders use the Transformer decoder whereas for the ResNet encoder, we use the MLP decoder as training with the Transformer decoder did not yield good results.
All models use 7 slots.
• Choice of Decoder.
This experiment uses 11 slots for MOVi-C, 6 slots for PASCAL VOC 2012, and 7 slots for COCO.
For the Transformer decoder on MOVi-C, we use 4 attention heads instead of 8, which produced better results.
[-ViT-B Patch Size   
-]
[Table]
[Figure]
Block Pattern
The block mask patterns are generated by dividing the image into a specified number of columns, then dividing the columns such that a target number of slot masks is reached (see Fig. 12).
For less than 9 masks, 2 columns are used, for 9–15 masks, 3 columns, and for more than 15 masks, 4 columns are used.
For the block pattern results given in the main text, MOVi-C, PASCAL VOC 2012 and COCO usemasks, and MOVi-E 24 masks.
Slot Attention
The hyperparameters used to run the experiments can be found in Table 18.
For the encoder, we use a ResNet34, using the modifications suggested by Kipf et al.
and Elsayed et al.
That is, we modify the basic ResNet34 architecture ( resnet34 in the timm library) to use a stride of 1 in the first convolution, and replace batch normalization with group normalization.
This results in a 512 -dimensional feature map of size 16 × 16 for image size 128 × 128 .
The slot attention module largely follows the description given in App.
C.1 for the DINOSAUR architecture.
However, like Kipf et al.
, we add a linear positional encoding to the feature map (coordinates in the four cardinal directions scaled to [ − 1 , 1] , linearly projected to the feature dimension and added to the features).
The number of slots was set to 11 and 24 for MOVi-C and MOVi-E, and 11 for PASCAL and COCO.
For PASCAL and COCO, this value was chosen according to what performed best in a scan over several values (cf.
Fig.
8); for none of the values, slot attention succeeded in successfully discovering objects on those datasets.
For the decoder, we use the same spatial broadcast decoder as (Kipf et al., 2022), i.e. 5 transposed convolutions with 5 × 5 kernels, channel size 64 and ReLU activations, upsampling from size 8 × 8 to 128 × 128 .
Input and target images are scaled to range [ − 1 , 1] .
SLATE We base our implementation on the official SLATE implementation 6 .
The hyperparameters are close to the ones used in SLATE (Singh et al., 2022a) and STEVE (Singh et al., 2022b) and are presented in Table 19.
STEGO To evaluate STEGO on the COCO-Stuff 171 dataset (Sec. 4.3), we modified the official STEGO implementation 7 to use batch k-means clustering with 172 clusters.
Other parameters were [Table] [Table] not changed from the STEGO configuration on the COCO-Stuff 27 dataset as the training data is the same for those two datasets.
For the experiment evaluating STEGO on the MOVi datasets (Sec. B.7), we run a DINO ViT-B with patch size 8, using 22 and 27 clusters, and train for 20 000 steps.
We manually tested different hyperparameters and settled on λ self = 0 .
6 , λ knn = 0 .
3 , λ rand = 0 .
6 , b self = 0 . 26 , b knn = 0 . 21 , b rand = 0 . 4 for MOVi-C, and λ self = 0 .
6 , λ knn = 0 .
3 , λ rand = 0 .
6 , b self = 0 .
22 , b knn = 0 .
17 , b rand = 0 .
4 for MOVi-E. We refer to Hamilton et al.
for the meaning of those parameters.
C. U NSUPERVISED S EMANTIC S EGMENTATION
For unsupervised semantic segmentation, we need to assign a discrete label to each discovered mask.
As common in the literature (Gansbeke et al., 2021; Melas-Kyriazi et al., 2022), we run k-means clustering on features associated with each mask for this purpose.
In particular, for each slot generated by the model, we compute a feature vector by taking a weighted average between the slot’s mask and features from ViT block 9.
We experimented with different features and found the outputs of ViT block 9 to yield the best results.
Each vector is then L2-normalized, and the set of vectors from all images is clustered by running k-means clustering, and each slot is associated with a cluster.
Finally, the clusters are assigned to ground truth classes by maximizing the total IoU between clusters and classes using the Hungarian method (Kuhn, 1955).
For k-means clustering, to avoid sensitivity to initialization, we run 20 repetitions and use the run with the lower sum of squared distances to the cluster centers.
As we noticed there is still some variance between different evaluation runs, we run the inference and clustering procedure three times and use the average as the final value for the training run.
Results for unsupervised semantic segmentation are dependent on the number of clusters used.
Prior methods (Hamilton et al., 2022; Gansbeke et al., 2022) typically use the number of classes labeled in the datasets as the number of clusters.
For unsupervised object segmentation, this presents an issue for our method, as it captures entities in both foreground and background, and thus clustering using the number of (foreground) object classes leads to over-merging of classes.
To ensure a fairer comparison, we instead estimated the number foreground plus background classes occurring on the datasets and used those as the number of clusters.
In particular, for PASCAL VOC 2012, we use 105 clusters (the number of classes occurring on at least 0.5% of images, using the PASCAL-Context (Mottaghi et al., 2014) labeling and class statistics), and for COCO, we use 172 clusters (the number of classes labeled on the COCO-Stuff dataset (Caesar et al., 2018)).
As this results in more clusters than ground truth classes, Hungarian matching leaves some clusters unassigned; we treat all unassigned clusters as being assigned to the “background” class.
For scene decomposition, we simply use 27 clusters for
COCO-Stuff 27 and 172 clusters for COCO-Stuff 172.
D D ATASET AND E VALUATION S ETTINGS
MOVi As the MOVi datasets are video datasets, we convert them into an image dataset by sampling 9 random frames per clip.
This yields 87 732 images for training and 2 250 images for evaluation.
Note that we use the validation split provided by the MOVi dataset for evaluation, and not the test split containing out-of-distribution objects and backgrounds.
This is in line with prior work (Kipf et al., 2022; Elsayed et al., 2022).
For the “semantic” MOVi-C variants used in Sec.
B.6, we generated 250 videos using the Kubric simulator (Greff et al., 2022), yielding 2 250 images.
For the “semantic objects” variant, after sampling the first object in each video, only objects of the same category were allowed to be sampled. The categories are: “Action Figures”, “Bag”, “Board Games”, “Bottles and Cans and Cups”, “Camera”, “Car Seat”, “Consumer Goods”, “Hat”, “Headphones”, “Keyboard”, “Legos”, “Media Cases”, “Mouse”, “Shoe”, “Stuffed Toys”, “Toys”. Objects that do not have one of those categories assigned were not considered.
For the “duplicate objects” variant, the exact same object was resampled multiple times.
Otherwise, the same settings as for MOVi-C were used.
PASCAL VOC
We train on the trainaug variant with 10 582 images.
For object discovery, we evaluate on the validation set of trainaug, with 1 449 images.
For unsupervised object segmentation, we use the 20 object classes plus one background class.
For object localization, we evaluate on the trainval split, with 11 540 images, as standard for this task (Simeoni et al., 2021; Melas-Kyriazi et al., 2022).
COCO
We train on the COCO 2017 dataset with 118 287 images, and evaluate on the validation set with 5 000 images.
For unsupervised object segmentation, we use the 80 “things” classes plus one background class.
For unsupervised scene decomposition, we use the COCO-Stuff 27 labeling (Ji et al., 2019; Cho et al., 2021), resulting from merging the 80 “things” classes into 12, and the“things” classes into 15 super-classes, and the COCO-Stuff 171 labeling, using all “things” and “stuff” classes.
For object localization, we evaluate on COCO-20k, with 19 817 images.
COCO-20k is a subset of COCO 2014 training and validation images, filtered to remove images with only crowd instances and removing crowd annotations.
Evaluation Settings
For scene decomposition on COCO-Stuff 27 and COCO-Stuff 171, we take a center crop of the image after resizing the minor axis to 320 pixels and evaluate the masks at 320 × 320 resolution to be consistent with prior work (Ji et al., 2019; Hamilton et al., 2022). For [Figure]
object discovery on PASCAL VOC 2012 and COCO, we use the same settings.
For object discovery on MOVi, we use the full image.
Again to be consistent with prior work, for object localization on
PASCAL VOC 2012 and COCO-20k as well as object segmentation on PASCAL VOC 2012 and
COCO, we evaluate the full image after resizing to 224 × 224 , ignoring aspect ratio.
Our model proved to be robust to the aspect ratio distortion to some degree; however, we believe that results can be further improved, e.g. by taking the distortion into account during training, or by using multi-crop evaluation.
Soft probability masks are converted to hard masks using the arg max .
E A DDITIONAL E XAMPLES
We include additional mask predictions of our model for MOVi-C (Fig. 13), for MOVi-E with(Fig. 15) and 11 slots (Fig. 16), for PASCAL VOC 2012 (Fig. 17) and COCO (Fig. 18, Fig. 19,
Fig. 20, Fig. 21).
We also show results for SLATE on the images of the main text in Fig.
[Figure] [Figure]
I m a g e DINOSAUR S l o t A tt e n ti on
S L A TE
[Figure] [Figure]
ViT-B/
Supervised
ViT-S/
DINO
ViT-B/
DINO
ViT-S/
DINO
ViT-B/
MoCo-v
ViT-B/
MSN ViT-B/ MAE RN
Supervised
RN50DINO
RN34 (S)
DINO RN
RN50MoCo-v
RN34 (S)
MoCo-v
RN
ViT-B/
DINO
RN34 (S)
DINO ViT
[Figure]
[Figure]
[Figure]
[Figure]
MSN ViT
ViT
[Figure]
MAE
[Figure]
(S)
MAE ViT