Paper: Grounded Object-Centric Learning (ICLR-2024)

Reviewer comment (Experiments):
I would appreciate more ablation studies in the experiments section. The current version primarily presents the state-of-the-art (SOTA) performance for two case studies, but additional ablation studies would provide further insights into the specific contributions and the impact of different components or techniques employed in the proposed method.

Author response:
We agree that the ablation can provide more insights about the method; we have included ablations on sampling, types of spectral decomposition, and sample efficiency on multiple tasks and datasets. Additionally, we have included the ablations on codebook size and its implications on results in the appendix.  
In case we are missing any ablations that would add to the clarity in the context of CoSA, please let us know.

Reviewer comment (Experiments):
The author mentions that the method incorporates the object-level property vector, but there is a lack of evidence regarding how it functions. For instance, it is unclear whether the method can effectively discriminate between multiple instances with similar appearances.

Author response:
The evidence of selecting object-level property vectors can be observed in both object discovery and reasoning tasks. In the case of object discovery, FG-ARI measures this property, whereas HMC measures the same for reasoning tasks measures this property, whereas for reasoning tasks, HMC measures the same.  

Similarly, in the case of Object discovery, we have added new results to qualitatively demonstrate the binding across multiple instances; please refer to Figure 19 in the appendix section.

Reviewer comment (Evaluation):
The visualization of the COCO results does not appear to be accurate, and it seems that the method may not be as applicable to real-world scenarios.

Author response:
We do observe that the results on the COCO dataset are not as good as supervised segmentation models, but it is important to note that the proposed model is fully unsupervised, and based on the initial number of slots hyperparameter, the model does segregate between different types of objects in the image. We leave further exploration in improving the model performance to match closely to the segmentation ground truths to future work.

Reviewer comment (Experiments):
The experiments on more complex scenes are required. For example, multiple instances of the same object category, it would be interesting to show the property binding ability in this case.

Author response:
We share the reviewers view on examining the binding ability of the framework on multiple object instances of the same object. Given that CLEVR has several of these examples in the dataset, we included a section in Appendix H.5, where we demonstrate the reusability of dictionary elements based on repeated object instances. As illustrated in Figure 19, similar looking object slot representations are sampled from the same distribution.

Reviewer comment (Experiments):
Does the model build on pre-trained backbones? Or a randomly initialized encoder $\Phi$ is also sufficient to provide cues for spectral decomposition and discritization?

Author response:
For CLEVR, ObjectsRoom, Bitmoji, FFHQ, and Tetrominoes datasets we start with a randomly initialized encoder. 
For COCO, given the very high diversity of image content, we start with a pre-trained image encoder similar to DINOSAUR and apply CoSA on the extracted features.

Reviewer comment (Experiments):
I suggest authors to run the trained conditional slot attention with grounded slot dictionary on some video data, e.g., DAVIS-2017, to validate whether the slot dictionary can track objects across time and more vividly show the binding ability to object types.

Author response:
We thank the reviewer for the great suggestion. We agree that extending the proposed method to a large-scale video dataset like DAVIS-2017 would be interesting and valuable. However, at this stage, we believe that this would be best served by a dedicated investigation in future work. 

Nonetheless, to reassure the review of our method’s ability to correctly identify and bind multiple object instances, we have added a discussion section and some illustrations using  CLEVR in the appendix (section H.5).

Reviewer comment (Theory):
The conventional slot attention based methods use reconstruction loss as the self-supervised objective to guide object decomposition. What is the relationship between the objectives used in this work and the recounstruction loss?

Author response:
The objective function in our work can be viewed as the sum of reconstruction loss (likelihood loss) + KL regularization. The additional KL term is used to train the strategies of selecting the elements from GSD for a given image.  

Our work introduces a probabilistic perspective of slot attention by optimizing a principled variational lower bound of the log likelihood of the data. The image likelihood we maximize is related to the reconstruction loss in previous works in the sense that minimzing a MSE loss is analogous to maximizing a pixel-wise factored Gaussian likelihood.

Reviewer comment (Experiments):
The paper is missing some ablations that I think would be important in evaluating the significance of the method: Conversely, how much is attributed to the GSD? What if we used a fixed $K$ with GSD?

Author response:
Our apologies for the confusion. Although it is technically possible to perform dynamic estimation of the number of slots K at training time using our method, it requires masking of surplus slots, which somewhat complicates the training pipeline without providing guaranteed benefits. Our method was used at inference time only; we have made this clear in the revised version of the paper. 

Therefore, comparing performance improvements from training with the dynamic estimation of K versus the GSD is not applicable, since the GSD was always trained with fixed K. If we remove the GSD, we end up with the standard Slot-Attention model, which we compare with in our ablation study (e.g. see the SA model in Table 1, Table 2 etc).

Reviewer comment (Evaluation):
What is the distribution of the number of objects for the different datasets? This would be important to interpreting the MAE values.

Author response:
Thanks for highlighting this point, we have updated the dataset information in appendix section C, with the distribution of the number of objects for the different datasets.

Reviewer comment (Theory):
While I can appreciate the probabilistic interpretation of the model, I feel it does not add to the clarity of the paper and may be more appropriate for the appendix. Specifically, if I understand correctly, if $q(\tilde{z}|x)$ is deterministic and $p(\tilde{z})$ is uniform as in VQ-VAE, then the KL term is just a constant and not actually used to optimize the model?

Author response:
We thank the reviewer for the astute observation. It is true that under a uniform prior and deterministic sampling, the KL term drops to a constant. However, this is not the case for the Gumbel sampling scenario, as while the prior is still uniform, the sampling is stochastic. For this reason, the probabilistic exposition in our paper is designed to be generally consistent with any (discrete) representation learning setup.  

We have clarified this in the paper and have included additional background on quantization clarifying this in the appendix.

Reviewer comment (Presentation):
The discussion of the different sampling strategies (Euclidean, Cosine, Gumbel) does not seem necessary for the main text since (from my understanding) the experiments in the main text are only done with the Cosine version. I do see additional experiments on other sampling strategies in the appendix, but if that is the case, this discussion can be removed from the main text.

Author response:
We thank the reviewer for this suggestion; we have briefly introduced different types of sampling for the sake of completion, providing different options available for constructing the dictionary. We have included details of quantization and the sampling methods in the appendix.

Reviewer comment (Experiments):
The codebook size seems like a potentially important parameter. What size do you choose for the different experiments? How sensitive are the results to codebook size?

Author response:
In all our experiments, we initialize the codebook with a sufficiently large number of embeddings for the complexity of the datasets used. We’ve included additional ablations and further discussion on codebook size in the appendix section H.4. 

Here are the ablation results on the CLEVR dataset. 

| M| MSE | FG-ARI | CBD | CBP |  
| :----: |:----: |:----: |:----: |:----: |  
| | | Cosine | | |  
| M = 16 | 8.11  | 0.73 | 0.99 | 14.96 |  
| M = 64 | 3.14 | 0.96 | 0.99 | 44.39 |  
| M = 128 | 3.46  | 0.94 | 0.98 | 51.65 |  
| M = 256 | 6.68  | 0.85 | 0.98 | 82.20 |  
| | | Euclidian | | |  
| M = 16 | 4.81 | 0.93 | 0.99 | 8.33 |  
| M = 64 | 4.04 | 0.94 | 0.99 | 32.40 |  
| M = 128 | 3.64 | 0.94 | 0.95 | 20.69 |  
| M = 256 | 4.34 | 0.89 | 0.96 | 19.79 |  
| | | Gumble | | |  
| M = 16 | 6.64 | 0.88 | 1.0 | 11.06 |  
| M = 64 | 3.82 | 0.93 | 1.0 | 29.62 |  
| M = 128 | 3.89 | 0.93 | 0.99 | 27.16 |  
| M = 256 | 5.46 | 0.92 | 0.99 | 30.28 |

Reviewer comment (Reproducibility):
I want to confirm that the Abstraction Module is not differentiable and just uses the output of the encoder, which is trained through the Slot Attention path. Then, there are similarly no gradients flowing through the GSD and it is only updated with EMA (Appendix F). Is this understanding correct? If so, I think this could be stated more explicitly for clarity in the main text.

Author response:
Yes, the reviewer is indeed correct; the abstraction module is nonparametric, but the gradients flow through it. 

In the case of GSD, the key part of the dictionary $\mathfrak{S}^1$ is updated via EMA, while the distribution parameters in $\mathfrak{S}^2$ are updated with gradients from the SA module.