Paper: Provable Compositional Generalization for Object-Centric Learning (ICLR-2024)

Reviewer comment (Experiments):
How does the effect of the consistency loss scale with the number of slots?

Author response:
Thank you for asking this interesting question! In App. C.3, we now include an experiment that ablates the impact of the number of slots on the consistency loss. As the number of slots increases from 2 to 3 and 4, optimization of the consistency loss becomes more challenging, and its value grows. This is not unexpected as the number of possible latent slot combinations grows combinatorially with the number of slots, making sampling the space more challenging. Furthermore, as the consistency loss increases, we see worse performance on OOD metrics, as our theory suggests. We now include a discussion of potential solutions for scaling this loss to more complex settings in App. B.5.

Reviewer comment (Experiments):
How does the method hold up on non-synthetic data, especially if you slightly relax some constraints? For instance, what if you have expressive slot-wise decoding, but allow for a low-expressivity non-linear combination at the end for rendering?

Author response:
We agree with the reviewer that understanding our results on non-synthetic data is important; however, this is challenging for two main reasons. Firstly, for real-world data, one does not generally have access to the ground-truth latents making our evaluation schemes inapplicable. Secondly, even if access to ground-truth latent information is available, our experiments require being able to sample latents densely from a slot-supported subset. Specifically, our experiments rely on sampling from a diagonal strip in the latent space with a small width. If such a region were sub-sampled from an existing dataset, this would leave a tiny number of data points that are insufficient to train a model. To this end, our experiments require access to a dataset's ground-truth renderer so latents can be sampled densely. This is not available in most cases, however. We now discuss this in a paragraph in App. B.6.

Reviewer comment (Theory):
The assumptions of compositionality and irreducibility are quite restrictive. Most real-world datasets likely violate these.

Author response:
Regarding compositionality, we agree with the reviewer about the restrictiveness of this assumption. Compositional decoders are special cases of additive decoders (see App. A.7) and thus share the same limitations: They cannot perfectly model objects with complex interactions such as occlusion or reflection. Irreducibility, on the other hand, roughly states that parts of the same object share information. We argue that this assumption is a more fundamental property of objects and will thus be valid more generally for multi-object scenes.

Reviewer comment (Theory):
The additive decoder limits modeling of complex object interactions and relations.

Author response:
As noted in our “Extensions of Theory” paragraph in the Discussion section, we agree with the reviewer that additivity is indeed insufficient for modeling more complex object interactions. We believe, however, that this assumption can be relaxed to allow for more complex interactions and view this as an important direction for future work.

Reviewer comment (Experiments):
The consistency regularization implementation requires sampling implausible object combinations. More principled schemes could improve results in complex environments.

Author response:
As noted in our “Extensions of Experiments” paragraph in the Discussion section, we agree regarding the limitations of our consistency loss implementation and approaches for improving it. One such approach could be to learn a prior over latent slots and sample slot combinations more rigorously according to their likelihood under the prior. This could avoid sampling implausible combinations; however, such a scheme is challenging as it relies on the likelihood of being valid for OOD combinations of slots. Another possibility would be to include heuristics to directly filter combinations based on prior knowledge of the data-generation process, e.g., to filter objects with similar coordinates that would intersect. We believe that further explorations in such directions are a promising and important direction for future work to make this method practical for more complex datasets. We now include this discussion in App. B.5.

Reviewer comment (Experiments):
Experiments only validate the theory on simple synthetic datasets. Testing on more diverse and realistic data would better demonstrate applicability, though the evaluation would also be more challenging.

Author response:
We agree with the reviewer that understanding our theoretical results on more diverse and real-world data is important for better understanding its applicability. This is indeed challenging for real-world datasets, however, as one does not generally have access to the ground-truth latents making our evaluation schemes inapplicable. Moreover, using the synthetic datasets typically used in object-centric learning is also not straightforward for our experiments due to the need to sample latents densely from a slot-supported subset. Specifically, our experiments rely on sampling from a diagonal strip in the latent space with a small width. If such a region were sub-sampled from an existing dataset, this would leave only a tiny number of data points that are insufficient to train a model. To this end, our experiments rely on access to the ground-truth renderer for a dataset such that latents can be sampled densely; however, this is not available in most cases. We now discuss this paragraph in App. B.6.

Reviewer comment (Experiments):
The proposed methods, especially when ensuring encoder-decoder consistency and handling latent slots, might pose scalability issues for very large datasets or more complex models. A discussion on the scalability, computational costs, and potential solutions would make the paper more robust.

Author response:
Thank you for mentioning this. Methods based on latent slots have in fact been shown to scale to more complex settings (e.g., [1, 2]); however, we agree regarding the scalability of the consistency loss, which we aimed to highlight in our “limitations of experiments” paragraph. We have included a figure in App. C.3 highlighting the challenges in scaling this loss as the number of objects increases. Regarding computational costs, the loss requires additional passes through the encoder and decoder as well as computation of the encoder’s gradients wrt. the loss. We found this to increase training time by a maximum of 28% across runs. We have incorporated a discussion of this point in App. B.5.