Paper: Object-Centric Learning with Slot Mixture Module (ICLR-2024)

Reviewer comment (Novelty):
The key contribution of this paper is the application of GMM mechanism for better slot representation learning, but the improvements are more about the performance in supervised tasks: image reconstruction and set prediction, instead of the more desirable unsupervised object segmentation. This makes the general contribution to be less appealing.

Author response:
**Q1. Unsupervised tasks.** We would like to clarify that the image reconstruction task with SLATE, as featured in our study, is indeed an unsupervised task. In this task, the model is given an image and tasked with reconstructing the same image, similar to unsupervised object discovery. This approach encompasses the use of an autoregressive transformer conditioned on slot representations, aimed at reconstructing the input image in latent space.

This methodology is particularly significant as it demonstrates scalability to more complex scenes while maintaining object-level disentanglement. We see the ability to reconstruct an image in latent space, preserving object identities and relationships as one of the key aspects of OCL. Thus, our work, contributes to the field by enhancing the capability of models to handle complex, unsupervised learning scenarios effectively.

We believe that our focus on image reconstruction and set prediction tasks highlights the versatility and applicability of our approach in both supervised and unsupervised learning contexts, thereby underscoring its relevance and importance in the broader landscape of OCL.

Reviewer comment (Novelty):
In page 2, the claimed Contributions (2) and (3) are not very meaningful, or at least both can be combined, because both sentences describe the experimental results.

Author response:
**Q2. Combining Contributions (2) and (3).** We agree that Сontributions (2) and (3) should be combined as they both describe the experimental results. We have made this change in the revision of the paper.

Reviewer comment (Presentation):
In Equations (1)(2), it's suggested to use math symbols instead of English words. It’s also suggested to use bold symbols to represent vectors.

Author response:
**Q3. Use of math symbols instead of English words.** We have made this change in the revision of the paper. Thank you for pointing this out, the introduction of the suggested notation has improved the readability of the paper.

Reviewer comment (Writing):
In page 4, it is unclear how the function f_theta(x, u, diag) works. Obviously, the input has three elements instead of two as described in the text "R^{NxD} x R^{KxD} -> R^{NxK}". More details should be provided because this is the primary technique contribution of this paper.

Author response:
**Q4. Function f_theta(x, u, diag).** Thank you for bringing this ambiguity to our attention. We have clarified the description of the function f_theta(x, u, diag).

Reviewer comment (Evaluation):
In Section 4.3, for the experiments of object discovery, the dataset (ClevrTex) is a bit too simple and the evaluation metric ARI is actually not suitable as well because the scores can easily achieve perfect numbers. It is advised to evaluate on more complex (real-world) datasets and use additional metrics such as AP scores, as also pointed out in the paper "Promising or elusive? unsupervised object segmentation from real-world single images, NeurIPS 2022".

Author response:
**Q5. More complex datasets and additional metrics.** We thank you for your valuable advice and agree that experimenting with more complex datasets and using additional metrics will further improve our paper. Unfortunately, the duration of the rebuttal period does not allow for additional experiments due to the lengthy training of the model. Nevertheless, we can additionally present the results for three seeds for the Shapestacks FG-ARI = 0.87 +- 0.02 dataset and the results of cumputting the AP(0.5) metric for three seeds for the ClevrTEX dataset: AP(0.5) = 0.47 +- 0.15.

Reviewer comment (Novelty):
In Section 5 “Related Works”, in the field of object-centric learning, recently, there are a number of works using pretrained feature representations to discovery objects, such as DINOSAUR (Bridging the Gap to Real-World Object-Centric Learning, ICLR 2023), Odin (Object discovery and representation networks, ECCV 2022), and CutLER (Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023). They should be discussed appropriately.

Author response:
**Q6. Section “Related Works".** We have expanded the related works section in the revision of our paper.

Reviewer comment (Evaluation):
_Redundant Slots_: Do the authors ever observe multiple slots sharing the encoding of the same object in a scene, or does "competition" between the spatial attention over the CNN feature maps work strongly enough to prevent this, even in SMMs?

Author response:
**Q1. Redundant Slots.** Hypothetically speaking, the SMM should exhibit a reduced tendency for redundant slot assignment compared to the SA model. This is because, in SMM, once a slot has a few vectors assigned, its prior weight decreases, making it less likely to capture information from other objects. In contrast, SA can be conceptualized as having equal prior weights for each slot, which could potentially increase the non-useful flow of information between slots.

However, in our practical observations with the SMM, the situation concerning redundant slots was similar to that in SA. Generally, redundant slots did not capture significant information from the same object. However, there were instances, particularly in the ShapeStacks dataset, where parts of the ground were split between two different slots, a phenomenon we also noted with SA.

Reviewer comment (Theory):
At the end of page one the authors write that "We believe... set prediction... requires distinguishing objects from each other" — this reads as if they are saying that set prediction is a better measure of OCL competency than object discovery. The reviewer is not sure they follow this argument if so, as it seems that entanglement of representations should be less of a hindrance to object discovery than the production of e.g. object-wise masks.

Author response:
**Q2. Competence of set prediction and object detection tasks for OCL.** Our intention was not to suggest that set prediction is a superior measure of OCL competency compared to object discovery. Instead, we aimed to highlight that set prediction is also a valuable and effective indicator for assessing OCL methods. Our point is that set prediction, alongside object discovery, provides a more holistic view of an OCL method's capabilities. It is not about one being better than the other; rather, each task addresses different aspects of OCL proficiency.

Reviewer comment (Reproducibility):
At the start of section 3 the authors state that "GRU [...]. takes current and previous slot representations as input and hidden states" but in Algorithm 1 it seems that the GRU is fed only the means.

Author response:
**Q3. GRU input.** Thank you for pointing out this inaccuracy in our description of the GRU's input in Section 3. We acknowledge that there is indeed a discrepancy between the text and Algorithm 1. You are correct in pointing out that the GRU is fed only with the mean, not the entire slot representation, which includes both concatenated mean and covariance.
This oversight in our description has been corrected. We appreciate your attention to detail and your help in improving the clarity and accuracy of our paper.

Reviewer comment (Presentation):
Figure 1 (left) might differ from Algorithm 1 in a few ways (at least, it was sufficiently unclear that perhaps labelling edges would be worthwhile). Most notably, the L2 difference for computing the covariance matrix is taken after the GRU-mean update in the algorithm, but before in the figure. Additionally, the reviewer notes that the MLP / LayerNorm are not represented in the figure.

Author response:
**Q4. Discrepancy between Figure 1 and Algorithm 1.** Thank you for your insightful feedback regarding Figure 1. We acknowledge the discrepancy you've pointed out, specifically regarding the sequence of the L2 distance computation for the covariance matrix. Algorithm 1 represents the correct order of operations, where this computation occurs post-GRU mean update.
In the interest of clarity, we chose to simplify Figure 1 by omitting common steps like MLP / LayerNorm to focus on the unique aspects of our model. However, we recognize that this may have caused some confusion. Based on your feedback, we are considering adding these details in a revised figure to align it more closely with the algorithm, ensuring both clarity and completeness.

We appreciate your contribution to enhancing the precision of our manuscript.

Reviewer comment (Novelty):
Since k-means is a particular case of the GMM framework (with the GMM allowing for a learnable variance in clusters), a discussion regarding the advantages of going for the more general model should be included. What are some real world scenarios where a gaussian-based method perform better?

Author response:
**Q1. Advantages of going from k-means to GMM.** As you mentioned, the GMM’s ability to consider variance as part of the final representation is a significant advantage over K-Means. This is particularly important in scenarios where clusters may have the same mean but different variances. Such situations are common in many real-world applications. For example, in image processing, different object categories might have similar color distributions (means) but vary significantly in their spatial spread (variance), which GMM can capture effectively. Moreover, the use of prior weights in GMM provides an additional layer of flexibility and insight. This aspect is especially useful in identifying redundant slots, as we mentioned in our paper.

Reviewer comment (Evaluation):
From the results in Appendix, Table 9 the SMM model seems to be more sensitive to the number of iterations compared to the traditional SA. Is this a consequence of the EM algorithm not converging or are there other optimisation issues that causes this phenomena?

Author response:
**Q2. Sensitivity to the number of iterations.** It's indeed typical for GMM to require more iterations for convergence compared to simpler methods like k-means, which may partially explain the observed phenomenon with SMM. However, in our experiments, we didn't find a direct correlation between the inner likelihood and the final performance of the model. This suggests that the increased sensitivity of SMM to iterations may not be solely due to EM algorithm convergence issues, and other factors in the model's optimization process could contribute to this behavior. Further investigation is required to fully understand this phenomenon.

Reviewer comment (Experiments):
As mentioning in Section 3, the SMM differs from SA in 3 aspects: the dot product is replaced by the Gaussian density function; both covariance and mean values are updated in the iterative process and the slot representation is a combination of mean and covariance. It would be insightful to see an ablation study showing the contribution of each one of the changes.

Author response:
**Q3. Ablation study of the contribution of each change.** Thank you for suggesting an ablation study. While our current study focused on the collective impact of these changes, we agree that isolating each change could provide deeper insights into their respective contributions. An in-depth ablation study was beyond our initial scope, but is a valuable idea for future research. We aim to explore this in subsequent work to better understand the individual and combined effects of these model improvements.

Reviewer comment (Theory):
There is a typo in Equation 7. The covariance matrix $\Sigma^{\*}$ should be computed based on the updated mean $\mu^{\*}$.

Author response:
**Q1. Typo in Equation 7.** Thank you for pointing out this typo. Yes, it should be \mu_{k}^{*} instead of \mu_{k}. We have corrected this error in the text. Unfortunately, LaTeX limitations prevent us from highlighting the formula.

Reviewer comment (Presentation):
The qualitative results could be improved, in my opinion. The images depicted in Figure 2 are quite blurry. This makes it quite difficult to assess whether SMM brings actually any substantial improvements over SA or not. I'd recommend the authors to provide higher quality images to strengthen their manuscript.

Author response:
**Q2-Q3. Quality of image reconstruction.** Regarding the blurriness and relatively low resolution of the images (during training we used images with a resolution of 96x96), we acknowledge that it might have impacted the ability to visually assess the improvements our approach brings over the SA. In our paper, we discussed how the limitations of the encoder/decoder models used in our experiments are a significant factor affecting image reconstruction quality. While our quantitative metrics indicated noticeable improvements in the quality of latent space reconstructions by SMM, these enhancements might not be as evident visually in the source space due to these limitations.

We set up training at higher resolutions (128x128 and 256x256), but training at this resolution takes much more time. If the model manages to converge by the end of the rebuttal period, we will include those results in the paper.

Reviewer comment (Evaluation):
In Figure 2, the images produced from SMM seem quite distorted. These distortions seem more pronounced for the ClevrTex dataset. I am under the impression that given how expressive SMM (when compared to SA) is the reconstructions from the learned slots would be more naturally-looking. It is not clear to me whether the distortions result from the considered Image GPT model, or whether it's because the learned slot representations are not that informative.

Author response:
**Q2-Q3. Quality of image reconstruction.** Regarding the blurriness and relatively low resolution of the images (during training we used images with a resolution of 96x96), we acknowledge that it might have impacted the ability to visually assess the improvements our approach brings over the SA. In our paper, we discussed how the limitations of the encoder/decoder models used in our experiments are a significant factor affecting image reconstruction quality. While our quantitative metrics indicated noticeable improvements in the quality of latent space reconstructions by SMM, these enhancements might not be as evident visually in the source space due to these limitations.

We set up training at higher resolutions (128x128 and 256x256), but training at this resolution takes much more time. If the model manages to converge by the end of the rebuttal period, we will include those results in the paper.

Reviewer comment (Evaluation):
Comparing *quantitatively* the attention maps learned by SMM against those learned by SA would have been quite helpful. I am under the impression that for more complex image scenes, the attention maps learned by SA would be more accurate than those learned by SMM due to potentially the high-variance involved.

Author response:
**Q4. Comparison of the attention maps quality.** Attention maps in models such as SMM and SA change with each iteration, and typically, only the maps from the final iteration are considered or depicted. However, these final iteration maps might not comprehensively represent the information contained in the slots throughout the iterative process. Therefore, a comparison based solely on these maps may not be fully informative or indicative  of the overall performance of the models.

**Per-slot attention masks for SMM and SA.** We have included examples of per-slot attention masks for a qualitative comparison of SMM and SA in the Appendix C in Figure 7.

Reviewer comment (Evaluation):
My main concerns mainly pertain to the quality of the reconstructed images. Any improvements in that aspect would strengthen the manuscript.

Author response:
**Q2-Q3. Quality of image reconstruction.** Regarding the blurriness and relatively low resolution of the images (during training we used images with a resolution of 96x96), we acknowledge that it might have impacted the ability to visually assess the improvements our approach brings over the SA. In our paper, we discussed how the limitations of the encoder/decoder models used in our experiments are a significant factor affecting image reconstruction quality. While our quantitative metrics indicated noticeable improvements in the quality of latent space reconstructions by SMM, these enhancements might not be as evident visually in the source space due to these limitations.

We set up training at higher resolutions (128x128 and 256x256), but training at this resolution takes much more time. If the model manages to converge by the end of the rebuttal period, we will include those results in the paper.

**Higher resolutions training results.** Unfortunately, the models trained at higher resolutions (128x128 and 256x256) did not converge by the end of the rebuttal period. We believe that no reasonable conclusions can be drawn from the intermediate results; therefore, we have chosen not to include these results. However, we find it important to include the final results and intend to do so in the final version.