Paper: Explicitly Disentangled Representations in Object-Centric Learning (ICLR-2024)

Reviewer comment (Experiments):
It would be good to see ablation tests on exactly how much of an impact the quality of the edge filter has (for example, how well does it still disentangle if you don't use any edge filter at all?), or an experiment that uses slightly more complex textures (ClevrTex for example). For me, this is the most critical point since it pertains to how well we can expect it to work with realistic data; if it can be addressed I am willing to change my score to an accept.

Author response:
- As for the first question, we agree that a study regarding the impact of the Sobel filter on the disentanglement of texture and shape is valuable. Therefore,  we included it in Appendix F.2. In that section, we also similarly analyze the importance of the variance regularization on the disentanglement. 
Since we also concur with the importance of experimenting with a dataset with more complex textures, we trained on CLEVRTex and presented the results in the updated version of the paper.

Reviewer comment (Experiments):
In the generative experiments, what happens when you sample shapes instead of textures?

Author response:
- To answer the second question, we included in Appendix F.4 Figure 34 shape generation results. It is important to know that this capability is strongly limited by the nature of the datasets we trained on, as all of them present very little variability in terms of object shapes.

Reviewer comment (Novelty):
It is not clear to me if basing the approach on the ISA approach instead of base SA is necessary at all, or simply a modeling choice.

Author response:
- Regarding the last question, basing DISA on ISA instead of SA is not strictly necessary. When using SA instead of ISA, position and scale information get included in texture and/or shape components, without preventing a quantitative study of the texture and shape disentanglement. However, there are two main reasons for choosing to employ ISA. (1) It enables a more efficient training process and (2) allows disentangling position and scale. In particular, the latter helps conduct the compositional and generative experiments. Without this property, during the texture transfer experiments, we may end up transferring between objects even position and scale information (which is undesired). Similarly, we may alter the position and scale of objects while generating new shapes.

Reviewer comment (Experiments):
I think one of the strengths of the prior work ISA was the incorporation of more realistic settings such as Objects Room, CleverTex, and WaymoOpen. However, the results in this work seem limited to simpler synthetic datasets with only very smooth almost mono-color textures. I think it would be necessary to demonstrate improved results on at least some of those harder datasets to convincingly show that the solution proposed here is a step in the direction of real-world applicability.

Author response:
Thanks for your response to our comment.

Regarding your concern about the absence of experiments on WaymoOpen and ObjectsRoom: we honestly believe that, with limited resources, including all the datasets you mentioned is unfeasible in a 12-days-long rebuttal. Experimenting on CLEVRTex has already been a great effort considering 3 resource-consuming repetitions for each of the studied models (SA, ISA, and DISA) with 11 slots each. Moreover, CLEVRTex is currently one of the most complex benchmarks in object-centric learning, and we achieved results aligned with ISA (with CNN backbone and trained for 500K steps in the original paper) after only 150K training steps.

Concerning your sentence "The FG and BG-Ari results on ClevrTex is a nice result but the MSE suggests to me that, at the very least, we cannot conclude we have better disentanglement than in ISA.", we find ourselves in disagreement. Looking at disentanglement results, both quantitative and qualitative, it is clear that DISA achieves some degree of texture and shape disentanglement. For example, Figure 27 shows that we are able to swap the textures of a white cube and a grey cylinder. Even if the MSE is not perfect, and thus the textures are not as detailed as in the input image, we would not be able to obtain an identical grey cube and a white cylinder without some extent of disentanglement. At the same time, as shown in Figure 13, we maintain ISA's position and scale disentanglement property. ISA, on the contrary, does not work towards the disentanglement of texture and shape, thus, as the base SA, does not disentangle those factors (we actually do not even know where those factors could be encoded in the latent space). We can therefore quite confidently assume that we have an overall better disentanglement than ISA. Finally, the marginally worse MSE of DISA compared to ISA on CLEVRTex is most probably not very meaningful. In fact, for simplicity, we kept the same division of texture and shape features (32 and 32) on all datasets. However, as the CLEVRTex textures are far more complex, they could require a larger number of components to match or surpass ISA in reconstruction quality (as in the other datasets). This experiment could have been carried out, but as stated in the paper "Note that this work seeks to achieve the desired disentanglement within the latent space of DISA rather than focusing on obtaining state-of-the-art results in unsupervised object discovery and reconstruction quality." our goal is not to obtain SOTA and hence we considered it to be of marginal importance.

Finally, as for the last point, we cannot of course say with absolute certainty that DISA would keep outperforming the baselines on increasingly more complex datasets. However, the additional experiment on CLEVRTex seems to suggest that we may observe such a trend. In fact, the FG-ARI improvement of DISA over SA and ISA is significantly larger on CLEVRTex than CLEVR6 (23% and 11% vs 5% and 3%).

We hope to have addressed all your concerns. Let us know if you have any other questions.

Reviewer comment (Evaluation):
Are there known limitations to using the sobel filter? How does the sobel filter setup perform on high frequency texture patterns?

Author response:
Thank you for your review and for raising valuable concerns.

- Concerning your first question, there are some limitations to the Sobel filter, such as the production of thick edges. Another limitation is that, as it is designed to approximate horizontal and vertical edges, it can be inaccurate with diagonal ones (e.g., very high gradients). A more sophisticated filter could, of course, be beneficial to DISA.
In Appendix A, you can find an explanation of the Sobel filter, including a visual example (Figure 6) of how the Sobel filter works on all the considered datasets (CLEVRTex included, to show it on high-frequency texture patterns).

Reviewer comment (Reproducibility):
Can the authors also clarify the exact formulation for the variance regularization loss? Is the variance computed across dimensions like layernorm? or more like batch norm and across batch elements?

Author response:
Thank you for your review and for raising valuable concerns.

- Regarding your question about the variance regularization, we compute the variance across all the objects in the batch images. Precisely, for each slot component, we calculate its variance across all the slots in the batch. This is mentioned below Equation 7 with the sentence "When the loss is computed over a batch of images, $N_s$ becomes $N_s$ times the number of images in the batch".

Reviewer comment (Experiments):
the paper motivates the intro with better representation, generalization and downstream transfer by having an explicit disentangled representation, however doesn't compare with ISA or SA with that metric.

Author response:
- Regarding the first question and weakness, we indeed think that generalization and downstream transfer are the principal motivations for investigating this direction. However, we do not study the benefits of using the representations learned by our model on downstream tasks as we think it would require a separate paper. In fact, to answer such a research question, we cannot rely on a single experiment such as the property prediction used in our work. We think it would be necessary to conduct an extensive study on more diverse tasks than simply property prediction, including for instance visual question answering and RL-related problems. In our opinion, this study should not be simply included in the appendix of a paper but be the core of a distinct one.

Reviewer comment (Writing):
The paper relies on sobel filter to achieve this disentaglement but doesn't do a good job at explaining how exactly does the Sobel filter work and is able to remove the texture. Is sobel filter limited to CLEVR images for removing filter or can scale to COCO like images?

Author response:
- To address the second question, we included a dedicated section in the appendix (Appendix A). Here, we present the math and intuitive idea behind the Sobel filter, other than examples of its application to all the considered datasets. We also included the more real-world dataset CLEVRTex, as we additionally trained on it and presented the results in the updated version of the paper.

Reviewer comment (Novelty):
The paper says: "However, to the best of our knowledge, no research has been carried out on the explicit disentanglement of the texture and shape dimensions in object-centric learning,", i don't think this is true as there is work such as: D3DP (https://arxiv.org/abs/2011.03367), that do disentangle shape and texture explicitly in an object-centric manner. They use adaptive instance norm instead of using Sobel filter.

Author response:
- Concerning the last question and weakness pointed out, we indeed missed D3DP, which tackles the explicit disentanglement of shape and texture in an object-centric manner. However, they rely on ground-truth bounding boxes to decompose a scene into objects. Our method, instead, is completely unsupervised. Therefore, we changed that sentence to "However, to the best of our knowledge, no research has been carried out on the explicit disentanglement of the texture and shape dimensions in unsupervised object-centric learning, especially with non-probabilistic models.". Moreover, we included an additional section (Appendix C) in which, in the first paragraph, we compare DISA with D3DP. Finally, as for the comparison with AdaIn, we agree that it would be interesting to conduct an experiment using it when decoding textures. However, we unfortunately decided not to pursue it in favor of using our reserved computational resources for experimenting on CLEVRTex (i.e. extremely resource-consuming) and providing an ablatation study on the importance of variance regularization and Sobel filter for the disentanglement.

Reviewer comment (Experiments):
How does the paper compare against ISA or SA in terms of Figure 2 or other downstream tasks such as the one considered in this paper https://arxiv.org/pdf/2305.11281.pdf?

Author response:
- Regarding the first question and weakness, we indeed think that generalization and downstream transfer are the principal motivations for investigating this direction. However, we do not study the benefits of using the representations learned by our model on downstream tasks as we think it would require a separate paper. In fact, to answer such a research question, we cannot rely on a single experiment such as the property prediction used in our work. We think it would be necessary to conduct an extensive study on more diverse tasks than simply property prediction, including for instance visual question answering and RL-related problems. In our opinion, this study should not be simply included in the appendix of a paper but be the core of a distinct one.

Reviewer comment (Theory):
What is the intuitive idea + the math behind sobel filter? Would such Sobel Filter generalize to real world texture like the one seen in COCO?

Author response:
- To address the second question, we included a dedicated section in the appendix (Appendix A). Here, we present the math and intuitive idea behind the Sobel filter, other than examples of its application to all the considered datasets. We also included the more real-world dataset CLEVRTex, as we additionally trained on it and presented the results in the updated version of the paper.

Reviewer comment (Experiments):
I think this authors should discuss D3DP in their paper. Also compare against AdIN, as a way of disentangling shape and texture instead of using Sobel Filters.

Author response:
- Concerning the last question and weakness pointed out, we indeed missed D3DP, which tackles the explicit disentanglement of shape and texture in an object-centric manner. However, they rely on ground-truth bounding boxes to decompose a scene into objects. Our method, instead, is completely unsupervised. Therefore, we changed that sentence to "However, to the best of our knowledge, no research has been carried out on the explicit disentanglement of the texture and shape dimensions in unsupervised object-centric learning, especially with non-probabilistic models.". Moreover, we included an additional section (Appendix C) in which, in the first paragraph, we compare DISA with D3DP. Finally, as for the comparison with AdaIn, we agree that it would be interesting to conduct an experiment using it when decoding textures. However, we unfortunately decided not to pursue it in favor of using our reserved computational resources for experimenting on CLEVRTex (i.e. extremely resource-consuming) and providing an ablatation study on the importance of variance regularization and Sobel filter for the disentanglement.

Reviewer comment (Novelty):
The architecture is explicitly designed for shape and texture and cannot be extended to other latent dimensions, thus limiting the significance of the results. This represents a significant effort which is ad-hoc and hence should be saved only to crucial latent dimension; not sure texture is such.

Author response:
Concerning the first weakness point, we think that explicitly designing the architecture for shape and texture disentanglement does not prevent it from being extended to other latent dimensions. For example, by employing the Invariant Slot Attention mechanism instead of the base Slot Attention mechanism (as we did in our work), it is possible to extend the latent dimensions to scale and position components. It is also feasible to do the same for rotation features by including the rotation factors in the ISA mechanism. Moreover, we think that texture is indeed one of the main properties describing an object, along with others such as position, size, shape, and orientation. Furthermore, assuming to have a strategy to extract additional object information, it would be possible to concatenate it to the texture and shape representations before the decoding phase to exploit it.

Reviewer comment (Evaluation):
Texture decoding uses shape information, which reduces the applicability of the method outside the context of the current projectt.

Author response:
For the second weakness point, we are not sure we understand the meaning of it. In fact, all SA-based models use both shape and texture information to decode the object textures: usually, they feed the complete slot vectors (encoding all the object information, texture and shape included) to the decoder. Could we please ask for a clarification on it? Thank you very much in advance.

Reviewer comment (Experiments):
Shape decoding uses texture removing filter, which may by itself be responsible for some of the benefits of the approach.

Author response:
Regarding the last weakness pointed out, the Sobel filter is indeed responsible for part of the benefits of our approach. To analyze this, we decided to provide an ablation (Appendix F.2) on its impact in disentangling shape and texture. However, we do not consider this point to be a weakness, but a modeling choice. It would also be possible to replace the Sobel filter with a different filter, for instance, a more sophisticated one that could lead to improved performance.

Reviewer comment (Experiments):
Why was rotation invariance removed from the experiments?

Author response:
To answer the first question, we did not use the rotation factors in DISA as, on Tetrominos and CLEVR, ISA performed better without it. We extended this decision to the other datasets as well.

Reviewer comment (Experiments):
Is the new shape and texture dimensions disentangled from position and scale information? To what extent were position and scale varied in the experiments?

Author response:
Finally, we address your last question by including a qualitative analysis of the position and scale disentanglement (Appendix F.1 Figure 13). The results suggest that this property is indeed successfully achieved.
Regarding the extent to which position and scale were varied in the experiments: we trained on the complete training sets, where the objects can be found in approximately every location, while the scales are dataset-dependent. On Tetrominoes, a single fixed size is present, while Multi-dSprites has 6 values linearly spaced in [0.5, 1], where 1 represents a given default object size. CLEVR has 2 sizes (small and large) while CLEVRTex has 3 (small, medium, and large). However, for these two datasets, the proximity of an object to the camera affects its size in the image.