Paper: Cycle Consistency Driven Object Discovery (ICLR-2024)

Reviewer comment (Novelty):
In my view it is not sufficiently motivated why training unsupervised object-centric models on RGB images is the best approach for improving "object-based reasoning capabilities":  
- Segment Anything (Kirillov et al. 2023) suggests that generalizable object segmentation can be learned from limited supervised data.  
- Object-centric methods based on contrastive features such as DINO features are very capable, e.g. Dinosaur (Seitzer et al. 2022) or CutLER (Wang et al. 2023). It is argued that it "limits the applicability of the method to domains that the pretrained encoders are trained on". The works mentioned earlier however show that the resulting models work well on a range of datasets, including datasets that were not used to train DINO (e.g., MOVi).  
- Some works show that using additional data, such as optical flow or depth, allows training strong object centric methods (e.g., Karazija et al. 2022). The paper claims that "relying on [...] optical flow and motion is not feasible since many datasets and scenes do not come with this information". In my experience however, unlabelled video data is abundant for most practical settings.  
In summary, it is not clear to me why the restriction to unsupervised, image based methods trained from scratch is adequate for the goal of "developing object-based reasoning".

Author response:
**Motivation behind training unsupervised object-discovery methods on RGB images**

The reviewer has questioned whether the direction of training unsupervised object-discovery methods on rgb images is still a valid direction. They have presented 3 points to illustrate their concern. We would like to address each of the 3 points - 

1. **Segment anything shows that segmentation masks can be learned from limited supervised data** - First, we would like to point that our setting is completely different than SAM. We consider a fully unsupervised approach where in we do not have access to any ground truth object masks. On the other hand, SAM utilizes supervised masks in 3 phases of its training
    1.  In the first phase, the sam model is first trained with already available segmentation datasets which already have annotated masks.
    2.  In the second phase (Assisted Manual Phase), humans assisted by SAM annotate 120k images which are then used for training. Here most of the annotation is done by humans. 
    3.  In the third phase (Semi-Automatic phase), more training images (180k images) are collected wherein humans only annotate those parts of the image where is sam cannot confidently output a mask. 

    Therefore, in each of the 3 phases of SAM training - some form of annotation is used for training. Therefore it already utilizes a lot more annotated data than used in any of the tasks in this paper.
    While it is true that SAM has strong zero-shot object-discovery performance, we actually find that it struggles a bit in slightly obscure domains that it may not have seen during training. We evaluate and compare SAM against our BO-QSA + Cyclic model on the scannet dataset. We present results in the table below - 

    |Model | AP@05 | PQ | Precision | Recall | 
    | ---- | ----- | -- | --------- | ------ |
    | SAM | 15.09 | **34.32** | 35.54 | 41.29 |
    | BO-QSA + Cyclic | **29.20 ± 1.1** | 26.09 ± 1.4 | **37.03 ± 1.4** | **42.09 ± 1.8** |
    
    We find that except for PQ, BO-QSA+Cyclic outperforms SAM in all other metrics.
    Another drawback for SAM is that it depends on prompts. For an image without prompts, it will first generate an exhaustive grid of points on the image for prompts and then generate a segmentation mask for each point ordered by their predicted IoU. In general, this results in more than 100 masks per image which makes SAM very slow and impractical to apply to real-time scenarios such as in the downstream tasks studied in the paper. For comparison - SAM takes **2.5 sec** to process an image while BO-QSA takes **0.02 sec** to process an image. 

2. **Object-Centric Methods based on Contrastive Features such as DINO** - We agree that approaches like DINOSAUR perform well on datasets that dino has not been trained on such as in movi datasets. However, we would like to point that movi may still be considered fairly close to the imagenet dataset which dino was trained on since movi consists of 3d scans of real-world objects placed in real-world backgrounds [1]. The main difference is that while imagenet has images with one object, movi has images with multiple objects.
    On the other hand, the atari and causal world domains considered in the paper are very different than the domains that the dino encoder was trained on i.e. the imagenet dataset. Hence dinosaur may not perform very well on such cases. In fact, the atari DQN replay dataset [2] that we use in the paper for the decision transformer experiments consists of gray scale atari frames therefore DINOSAUR cannot be used for this setting since DINO requires RGB images as input. 
    Therefore, in this paper we consider an end-to-end training approach where the encoder is trained from scratch. This allows us to evaluate the model on downstream tasks that may contain varying types of observation spaces. 
    Furthermore, we also apply the proposed cycle consistency objectives to the DINOSAUR model and find that it improves performance of the dinosaur model on movi-c and movi-e datasets -
    
    | Model | MOVi-C | MOVi-C | MOVi-E | MOVi-E |
    | ----- |---------------|---------------|----------------|------------|
    | | **FG-ARI** | **mBO** | **FG-ARI** | **mBO** | 
    | DINOSAUR (ViT-B/8)    | 68.9 ± 0.4    | 38.0 ± 0.2 | 65.1 ± 1.2     | 33.5 ± 0.1 |
    | DINOSAUR + Cyclic    | **72.4 ± 2.1**| **40.2 ± 0.5**| **69.7 ± 1.6** | **37.2 ± 0.4** |
    
    Therefore, while the main setting of our paper is that where the model is trained from scratch, we find that the proposed cycle consistency objectives also bring improvements to the setting where pretrained encoders such as dino are used.

Reviewer comment (Evaluation):
Only FG-ARI is used as a metric for quantifying segmentation performance. It has been pointed out several times in the literature that this metric is problematic since it does not take into account whether object boundaries are accurate and favours undersegmentation (e.g., Engelcke et al. 2020, Karazija et al. 2021, Monnier et al. 2021). More established segmentation metrics, such as mIoU or AP, should be used that do not share these problems.

Author response:
**On evaluation metrics**

The reviewer points that we only use FG-ARI for evaluating segmentation which can be flawed. We agree that FG-ARI can be flawed. However, we would like to point that we have already used the AP and mIoU metric mentioned by the reviewer.

In table 5 in the main paper we use AP@05, PQ, Precision and Recall for evaluating segmentation performance on coco and scannet. 

In Appendix Table 11, we use mIoU and Dice to evaluate the segmentation performance on Cars, Dogs, Birds, and Flowers datasets. 

[1] Kubric: A scalable dataset generator Greff et al 2022

[2] https://research.google/resources/datasets/dqn-replay/

Reviewer comment (Experiments):
For example, does the Slot-feature-slot consistency objective reduce the total number of required slots creating a bottleneck? Does it have an effect on the size of the feature dimension of the slots?

Author response:
**does the Slot-feature-slot consistency objective reduce the total number of required slots**

The reviewer raises the question whether the proposed cycle-consistency objectives reduce the total number of required slots by creating a bottleneck. 

In our qualitative evaluation, we find that this is not the case. In general, we find that the slot utilization of models trained with the cycle consistency objectives is same as that for the models trained without it.

As we can see in Figure 2 and Appendix Figure 7, both the baseline and the cyclic models utilize the same number of slots while the cyclic model result in better disentanglement of objects into slots.

**Effect on Feature Dimension of the slots**

We thank the reviewer for pointing this out. We haven't studied the effect of the proposed cycle consistency objectives on the dimensions of the slots or the features. We will conduct this analysis and add it to the paper.

Reviewer comment (Theory):
In Eq. 8 the softmax is applied using $\tilde F$ but only along the diagonal? Can the authors elaborate what happens if the full matrix is used for the loss? Isn't that a stricter case?

Author response:
**On using the full feature similarity matrix for the feature-slot-feature loss**

The reviewer has asked about using the full feature similarity matrix for the feature-slot-feature loss.

First, we would like to point that the softmax in the feature similarity matrix \tilde{F} is applied across the rows of the matrix not along the diagonal. The loss is calculated only for the diagonal elements. 

We have conducted an experiment wherein the loss is calculated for the entire matrix instead of only the diagonal of the matrix. We performed this experiment for the ObjectsRoom, Shapesstacks, and ClevrTex datasets. We have presented the results for this experiment in Appendix Table 8. We find that applying the loss only for the diagonal elements works consistently better than applying it for the entire matrix.

Reviewer comment (Evaluation):
The authors point out one of the limitations of existing methods that a notable gap exists for the learned object-centric representations to be used in the downstream tasks. However, it does not make sense to claim the proposed method overcomes this by achieving better performance on downstream tasks. The logic here is somewhat doubtful.

Author response:
**On the claim that the proposed method addresses the gap in downstream tasks of existing object-centric methods**

The reviewer says that our claimed that the proposed approach addreses the limitations of existing object-centric models in downsteam tasks is doubtful.

We apologise for the confusion. We do not want to claim that the proposed approach solves all limitations of object-centric methods on downstream tasks. Rather, we want to claim the proposed approach takes a step towards addressing the limitations of existing methods on downstream tasks.

We will update the paper to reflect this.

Reviewer comment (Evaluation):
As shown in Fig. 2, in the bottom right, multiple semantics exist in slot 6. Why the model cannot depart them?

Author response:
**On multiple semantics in slot 6 for Figure 2**

The reviewer has questioned as to why multiple semantics exist in slot 6 of figure 6.

In general, in slot-attention based methods it is observed that one slot ends up representing the background while other slots are representing the foreground object. This is also followed in Figure 2. We can see that slot 6 in Figure 2 (for the cyclic one) represents the base of the green arm and the bowl which constitutes the background. While the rest of the slots represent parts of the foreground. For example, slot 1 represnts the actual arm which moves the objects. Slot 4 and 5 represent the golden and blue blocks respectively. 

We would like to once again thank the reviewer for their valuable feedback and time. We hope that the rebuttal has addressed all their questions. If there are further questions still remaining or if there are any more experiments that the reviewer would like us to perform we would be happy to do so.