Paper: Object centric architectures enable efficient causal representation learning (ICLR-2024)

Reviewer comment (Novelty):
The authors do however miss the opportunity to connect their research with previous work on disentanglement learning in a more systematic way. Indeed their approach of encouraging sparse transitions has already been explored in disentangled models (albeit not object-centric ones) in Klindt et al., (2021) and Montero et al., (2022). and thus these are both relevant work that should be discussed (though I don't think a comparison with these methods is required).

Author response:
We thank reviewer HmRq for bringing these works to our attention, however, we would like to emphasize that we are not claiming contribution to the sparse transitions in disentanglement, we are simply building on top of Ahuja et al. [1] as a modeling choice. Other choices are possible like the ones mentioned here by reviewer HmRq. We acknowledge that we should make this modeling choice more clear and mention the possibility of using the aforementioned works as well.

Reviewer comment (Evaluation):
While I don't have an issue with the evaluation per se, I do think that the scores of the models should be separated by properties as well as averaged over all of them (as is currently presented according to my understanding).

Author response:
We thank reviewer HmRq for bringing up this important point. While we acknowledge the usefulness of the additional insight provided by reporting per-property disentanglement scores, we would like to clarify that the presented scores aren't exactly average of per-property disentanglement scores. Intuitively, if say, we aim to disentangle $(p_x,p_y,\text{colour},\phi)$, and the model has only been able to separate just one property from the rest, then the score would be about $0.25$ or $\frac{1}{d}$, where $d$ is the number of target properties. If two properties are separated from the rest, then the score would be about $0.5$, and so on. So a score $0.95$ indicates that almost all properties are identified to a high degree. We hope this clarifies how the disentanglement scores work, and additionally, we will provide per-property disentanglement scores as well before the end of discussion period. Please expect another reply on this matter.

Reviewer comment (Evaluation):
My only question is related to how good the prediction of each individual object property is. My understanding is that the prediction scores are averaged over all properties, but previous research has show that some properties are easier to predict than others (eg. rotation is harder than position). Thus, I would like to see the scores for these different properties in isolation.

Author response:
Again, we thank the reviewer for bringing up this important observation. Above, we explained in more detail how these scores can be interpreted. However, we acknowledge that we have observed similar patterns to those that reviewer HmRq mentions. In particular, position properties are always far easier separated than the rest of the properties. We have also particularly witnessed more challenges with discrete properties, i.e., colour and shape, while rotation and size have also been fairly easy to disentangle. In summary, approximately we find the following ordering in increasing difficulty of separating the property:

$p_x,p_y < \text{size}, \text{rotation} < \text{colour}, \text{shape}$

However, we do believe there can be additional insight by reporting per property disentanglement scores, and we will provide those before the end of the discussion period.

We thank the reviewer again for raising these interesting points.


We thank the reviewer for their kind comment. We conducted additional experiments on 3D shapes in the presence of 2,3,4 objects with the following target properties $(p_x,p_y,\text{colour},\text{rotation})$ to assess the individual property disentanglement scores, and the findings show that the originally presented scores indeed reflect the individual property disentanglement as well, and basically we don't see any discrepancy with the originally presented numbers. I.e., the scores for this case study are very close ($\pm0.01$) to those of table 7 in the appendix. However, different properties do not get disentangled at the same time, with discrete ones (colour in this case) still being the more challenging properties, however, once disentangled, all properties are very well and equally disentangled.

However, if the reviewer has further suggestions, we could set up a more systematic evalution of individual property disentanglement and assess the behavior on more settings (2D for instance) and other combinations of target properties.

Thanks again for your insightful comments.

Reviewer comment (Novelty):
Insight: While technical sound, the proposed method itself appears not as interesting. If I understand correctly, the method is essentially nothing but adding a slot matching step to the existing weak-supervised disentangled representation learning.

Author response:
In response to the concern raised, we would like to emphasize that our work addresses the previously unexplored territory of "provable" disentanglement of object-centric representations. Our approach connects identifiability results from prior work to real-world scenarios with multisets, filling a crucial gap in the literature. Furthermore, we believe the simplicity of our model adds to its strength. By demonstrating the effectiveness and consistency of a straightforward model across various settings, we aim to establish it as a reliable and strong approach for generalizing previous results to realistic environments.

Reviewer comment (Novelty):
Positioning: The claim of being the first to `achieve disentangled representations in environments with multiple interchangeable objects` is overstated. Similar objectives have been pursued in recent studies, e.g., Block-Slot Representation [1]. Slot matching has also been considered in recent work [2]. A more thorough positioning in the literature about multi-object scenes might be needed.

Author response:
We appreciate the reviewer's observation regarding our positioning and would like to address this concern. We acknowledge the need for a more explicit emphasis on the positioning of our work. While [1] explores a similar problem **empirically**, we want to highlight that our contribution lies in providing a **provable** disentanglement of object-centric representations, followed by strong experimental results against strong baselines. As outlined in our paper (please refer to the footnote on page 3), [1] does not address the identifiability of their approach, and our work fills this crucial gap by offering formally accompanied by empirical results. We thank the reviewer for drawing attention to this aspect and will make necessary revisions to emphasize this point more effectively.

Regarding slot matching in [2], we recognize the presence of some form of slot matching in their work, but it is essential to clarify that our primary contribution is not the matching algorithm itself. Instead, our focus is on identifying and addressing challenges that arise from objects violating standard assumptions for latent variable identification. We demonstrate how leveraging object-centric architectures can effectively handle these challenges, showcasing the sample efficiency of our method, as highlighted in the appendix. In summary, even though our matching approach differs from [2], it is not also a central contribution of our work. Moreover, [2] provides a benchmark for causal representation learning, and the matching used in [2] is for intervention modeling.

Reviewer comment (Experiments):
I feel the experimental setup was quite limiting for the proposed claim

Author response:
**Weakness 1**: We would appreciate it if reviewer 9Arp could elaborate on this, and kindly let us understand what they have in mind, and what kinds of experiments would be interesting to be added to our work?
We would like to highlight two aspects of this work in response to this concern. First, the experiments conducted in this work are significant steps toward realistic applications of causal representation learning (CRL), and they are significantly more challenging than the experiments of the current literature in causal representation learning that has mostly focused on very toy settings with one or two circles [1,2,3,4]. Secondly, an important aspect of evaluating such approaches in CRL is access to ground-truth which is usually achievable through synthetic datasets, yet we have tried to bridge the gap as much as possible by using synthetic samples that are realistic (CLEVR-style). The synthetic nature of our dataset provides extra handle on stress testing our method, and allows us to quickly identify the failure modes, necessary practical assumptions, etc. However, we strive to improve our work and are keen to learn more about the experiments that reviewer 9Arp believes would strengthen this work.

Reviewer comment (Experiments):
How do we handle multiple set of latents? What happens when there are shared and not-shared latents across this set how does the disentanglement look like?

Author response:
Regarding the following,
> Experiments are conducted with a single set of latent variables
> 
we would like to clarify and highlight that our experiments are *not* carried out with *only* a single set of latent variables. While we deal with shared set of latents across objects, we show the effectiveness of our approach with varying set of latents, i.e., disentangling only $(p_x,p_y)$, or $(p_x,p_y,\text{colour},\text{shape})$, or $(p_x,p_y,\text{colour},\phi)$ (Please see the appendix for all the combinations we have tried.)

**Weakness 2**: In the presence of non-shared latents, the method still applies in its current form, however, unsurprisingly, we lose the benefit of requiring $k$ fewer perturbations. Good examples illustrating this point can be found in video games. There are properties such as characters moving around that are shared across all of them, and such properties are far easier to be identified due such knowledge being reused over and over. However special characters in the game might have superpowers unique to them, and consequently, learning about such special abilities is only possible through observing that one special character, without any knowledge transfer from the features of other characters who do not share this property.

Reviewer comment (Experiments):
Authors mentions limitations on number of objects and background although I would like to see the effect on some real life dataset (maybe can be constructed with multiple colored mnist images in the scene?)

Author response:
**Weakness 3**: We thank reviewer 9Arp for the suggestion. We would like to highlight that despite the differences between the proposed dataset and our experiments at the surface level, these experiments are actually not qualitatively different **in terms of disentanglement**. The only difference is **perception** (i.e., slot attention + the preceding vision model), not the disentanglement approach. We know from other works using slot attention ([5,6,7,8]), that the perception works just fine in far more realistic settings than our shapes experiments and the proposed MNIST dataset. The proposed disentanglement method is contingent on good slot representations, and given the success of slot attention, we think there is limited qualitative difference **in terms of disentanglement** between our experiments and the reviewer's proposed dataset.
[We are reading the question as meaning MNIST == realistic dataset, and asking us to create a dataset with multiple coloured MNIST digits with varying sizes, rotations etc.]