Paper: Towards Interpretable Controllability in Object-Centric Learning (ICLR-2024)

Reviewer comment (Writing):
Overall the paper is hard to follow, because common terms like equivariances are called different names (e.g. sustainability) and not clearly defined.

Author response:
> Terminology of sustainability (instead of equivariance)

We agree that clarifying "sustainability" as a pursuit of "equivariance" in slot manipulation will enhance the term's understandability. 
However, it is important to highlight that our paper presents a novel approach to slot controllability.
The introduction of the term "sustainability" is aimed at conceptualizing the challenges, including the durability test, in slot manipulation which has not been explored yet, while underscoring the significance of slot reusability for the robust utilizations.

Our intention in bringing "sustainability" is to suggest a direction for OCL studies, rather than to claim novelty for the term itself. 
We believe our novel contribution lies in the initial pursuit of interpretable controllability over slots and enhancing their robustness, with the ultimate goal of achieving equivariance in slot manipulation.

Reviewer comment (Experiments):
The experiments are only on variants of CLEVR, with no evaluations on real-world data.

Author response:
Nevertheless, it is true, as the reviewer jXVc mentioned, that “it is hard to use this on real-world datasets” while “other works (e.g. instruct pix2pix) show zero-shot results on real-world datasets.”
This is a prevalent problem in recent OCL research, not just ours, considering that several zero-shot and foundation models like SAM demonstrate exceptional performance across various real-world datasets in downstream tasks.
We believe that the difference between OCL and the state-of-the-art methods for the real-world datasets can be attributed to the task-agnostic nature of OCL.
OCL concentrates on examining how to define the objects, how to represent them, and how to understand the object interaction within the visual scene, devoid of any predefined target tasks.
In other words, OCL research investigates meaningful patterns defining what we identify as objects, and aims to capture the objects into vector representations only with RGB images, distinct from other artificial tasks influenced by human biases.
Therefore, in many OCL studies, the primary objective function for training is an RGB reconstruction loss in a self-supervised manner; here, the reconstruction is not even the target task of the OCL research.

Reviewer comment (Experiments):
Even on CLEVR, the results are not very convincing, and the method requires defining specific image augmentations for each equivariant action — so it is hard to use this on real-world datasets.

Author response:
> Real-world datasets.

Regarding the reviewer’s point about datasets, extending our research to real-world datasets requires two essential components: 1) a more powerful object discovery model and 2) an improved decoding model.

1. Achieving a more potent object discovery model involves various approaches, primarily using supervision signals. 
While directly training the current model on the target dataset might yield promising results, it deviates from the task-agnostic learning of visual sensory input we aimed for, as discussed in the previous response (1/3).
Secondly, the reviewer's mention of foundational models like SAM for segmentation mask extraction and spatial binding presents a method used to achieve the state-of-the-art. 
2. To enhance our decoding model's capabilities, we may consider incorporating techniques from Generative Adversarial Networks (GANs) or Diffusion. 
Notably, very recent research such as LSD [1] and SlotDiffusion [2], both known as spotlight papers at NeurIPS 2023, are delving into incorporating diffusion into slots.
Differing from concurrent studies on Slot + Diffusion, which concentrate on refining decoding performance in OCL model, our research emphasizes rethinking individual slot decoding within OCL specifically concerning the image-to-object loss separation.
Combining our proposed aspects with Slot + Diffusion research could yield valuable results in the future.

We recognize that these approaches offer promising directions for expanding OCL models including ours to real-world datasets, however, we consider these avenues to be beyond the scope of our current paper.

[1] Jiang et al., “Object-centric slot diffusion.” (arXiv 2023)

[2] Wu et al., “SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models.” (arXiv 2023)

Reviewer comment (Writing):
The authors introduce (as a contribution) new language for agreed-upon terms like equivariance, and the new language doesn’t add anything in my opinion. It is neither intuitive nor well-defined, and only serves to make the work harder to understand.

Author response:
> Terminology of sustainability (instead of equivariance)

We agree that clarifying "sustainability" as a pursuit of "equivariance" in slot manipulation will enhance the term's understandability. 
However, it is important to highlight that our paper presents a novel approach to slot controllability.
The introduction of the term "sustainability" is aimed at conceptualizing the challenges, including the durability test, in slot manipulation which has not been explored yet, while underscoring the significance of slot reusability for the robust utilizations.

Our intention in bringing "sustainability" is to suggest a direction for OCL studies, rather than to claim novelty for the term itself. 
We believe our novel contribution lies in the initial pursuit of interpretable controllability over slots and enhancing their robustness, with the ultimate goal of achieving equivariance in slot manipulation.

Reviewer comment (Experiments):
Experiments are on variants of CLEVR, which is a very simple dataset that was generated in ways that privilege this algorithm. No evaluation on real-world datasets. Other work (e.g. instruct pix2pix https://arxiv.org/pdf/2211.09800.pdf) DOES show zero-shot results on real-world datasets.

Author response:
> Real-world datasets.

Regarding the reviewer’s point about datasets, extending our research to real-world datasets requires two essential components: 1) a more powerful object discovery model and 2) an improved decoding model.

1. Achieving a more potent object discovery model involves various approaches, primarily using supervision signals. 
While directly training the current model on the target dataset might yield promising results, it deviates from the task-agnostic learning of visual sensory input we aimed for, as discussed in the previous response (1/3).
Secondly, the reviewer's mention of foundational models like SAM for segmentation mask extraction and spatial binding presents a method used to achieve the state-of-the-art. 
2. To enhance our decoding model's capabilities, we may consider incorporating techniques from Generative Adversarial Networks (GANs) or Diffusion. 
Notably, very recent research such as LSD [1] and SlotDiffusion [2], both known as spotlight papers at NeurIPS 2023, are delving into incorporating diffusion into slots.
Differing from concurrent studies on Slot + Diffusion, which concentrate on refining decoding performance in OCL model, our research emphasizes rethinking individual slot decoding within OCL specifically concerning the image-to-object loss separation.
Combining our proposed aspects with Slot + Diffusion research could yield valuable results in the future.

We recognize that these approaches offer promising directions for expanding OCL models including ours to real-world datasets, however, we consider these avenues to be beyond the scope of our current paper.

[1] Jiang et al., “Object-centric slot diffusion.” (arXiv 2023)

[2] Wu et al., “SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models.” (arXiv 2023)

Reviewer comment (Evaluation):
The results even on CLEVR are not convincing — leading to little to no improvement for object detection.

Author response:
> “The results even on CLEVR are not convincing.”

As previously mentioned, and outlined in Section A of the appendix, we acknowledge the reviewer's observation regarding our weakness—our evaluation being limited to synthetic datasets.
However, we would like to emphasize that recent works in OCL [1,2,3] frequently conduct evaluations on the CLEVR dataset. 
Furthermore, they classify CLEVRTEX and PTR as challenging datasets due to their visual complexity and intricate object patterns.
We conducted evaluations on these challenging datasets and believe that the results strongly demonstrate the effectiveness of our method, and this sentiment aligns with the feedback from Reviewer byFg, who mentioned, "The qualitative results are very impressive, even though the datasets are quite simple.”
For these reasons, we respectfully disagree with the comment — “the results even on CLEVR are not convincing” — considering the context of the OCL domain.

Reviewer comment (Writing):
Many terms feel philosophical, when they could be stated more concretely. E.g. “Then, the model performs spatial binding on img_{ref} to produce slots{_ref}”. Meaning you run the image through the model to get the slot latents?

Author response:
> “Meaning you run the image through the model to get the slot latents?”

We appreciate your feedback regarding the use of certain terms in our paper. 
The reviewer is correct in the interpretation of 'spatial binding' as the process of processing the image through the model to obtain the slot latents.
While we have used the term “spatial binding” in line with its technical usage in previous works [1-3], we understand the need for clarity and simplicity in our language to enhance the paper's readability. 
Moving forward, we will strive to use more straightforward and easily comprehensible terminology.
We thank the reviewer for pointing out this aspect and will make the necessary revisions to address this concern.

Reviewer comment (Novelty):
For interpretable latents (e.g. VAE or GANs), the authors note that these require “manual efforts to identify the features associated with specific properties.” In this work, too, you have to hand-design the augmentation and regenerate appropriate training data. This is also a manual effort, and arguably harder than a post-hoc approach.

Author response:
> This is also a manual effort, and arguably harder than a post-hoc approach?

While it might appear that our methods involve manual effort, such as designing the data augmentation, we contend that this effort is negligible when compared to the post-hoc procedures required in previous methodologies.

In the previous methods like IODINE, empirical studies during inference are crucial to establish relationships between specific values in a latent vector for an object and various properties like position, size, and color.
To find the relations, one should traverse all values in a latent vector and also decode the vector into an image.
Discovering these relationships involves traversing all values in a latent vector and decoding it into an image. 
Furthermore, ensuring consistent functionality of the relation function across different latent vectors or objects poses uncertainty, making the post-hoc process in previous methods cumbersome.

In contrast, designing data augmentation demands minimal effort compared to the aforementioned post-hoc procedures. 
The process primarily involves incorporating random sampling procedures—such as translating, scaling, and tinting—during training.
During inference, devoid of empirical studies on each slot-property relation, our method allows direct control over slots and their corresponding decoded images using instruction vectors, just as during training.
In this context, we assert that our method achieves a more straightforward and interpretable controllability than previous methodologies, with minimal additional manual efforts.

Reviewer comment (Writing):
Regarding the "Durability Test": This is defined in the paper as “The multi-step test involves a series of instructions to modify an object and another series to restore it to its initial state.” While this is a fine definition, the reviewer asks why not also write out the equations: e.g. $(g_1 * … * g_k)^{-1} (g_1 * … * g_k) x = x$?

Author response:
> Definition of "Durability Test”

Firstly, we are grateful to the reviewer for recommending a more concrete expression of our terms. 
Incorporating equations, as suggested, can enhance the understandability of our experimental concept.
However, as indicated by the reviewer, we maintain that our current definition of the task is also clear.
And in fact, to fully describe the durability test, for instance the multi-step test, we need to articulate the process of forward and backward execution N times such as 

$(g_{1}g_{2}...g_{k})^{-1}(g_{1}g_{2}...g_{k})...
(g_{1}g_{2}...g_{k})^{-1}(g_{1}g_{2}...g_{k})(x) = x$.

Compared to this equation, we think that the current textual description may convey the information in a more concise manner.
Still, we value the reviewer’s comment, encouraging us to explore alternative ways to articulate the definition of the durability test.

Reviewer comment (Theory):
The requirement that image augmentations are invertible is a strong one (e.g. viewpoint change is not invertible from the image alone). Why not make the approach more general and focus on measuring equivariance?

Author response:
Secondly, we have conducted an experiment to measure the equivariance in the both relationships $g_{1}(image) = h_{1}(slots)$ and $(g_{1}^{-1}g_{1})(image) = (h_{1}^{-1}h_{1})(slots)$, with the results presented in Table A in the appendix.
In line with the reviewer's insightful suggestion, we acknowledge the value of conducting a durability test for the whole slots from an image while measuring the degree of the equivariance. 
We are working on implementing the suggested evaluation, however, constructing the benchmark and evaluating our models is taking longer than anticipated due to the limited availability of benchmark datasets at present.
Therefore, it may not be feasible to incorporate these experiments in the upcoming revision during our rebuttal period. 
However, we are committed to including them in the subsequent future version of our paper.

Again, we appreciate the insightful question and valuable suggestion by the reviewer.

Reviewer comment (Evaluation):
Suggestion — You can measure the equivariance of the whole model by augmenting the images $(g_1 * … * g_k) (image) = (h_1 * … * h_k) (slots)$ where $g_i$ is an image aug and $h_i$ is the corresponding instruction ref2aug.

Author response:
Secondly, we have conducted an experiment to measure the equivariance in the both relationships $g_{1}(image) = h_{1}(slots)$ and $(g_{1}^{-1}g_{1})(image) = (h_{1}^{-1}h_{1})(slots)$, with the results presented in Table A in the appendix.
In line with the reviewer's insightful suggestion, we acknowledge the value of conducting a durability test for the whole slots from an image while measuring the degree of the equivariance. 
We are working on implementing the suggested evaluation, however, constructing the benchmark and evaluating our models is taking longer than anticipated due to the limited availability of benchmark datasets at present.
Therefore, it may not be feasible to incorporate these experiments in the upcoming revision during our rebuttal period. 
However, we are committed to including them in the subsequent future version of our paper.

Again, we appreciate the insightful question and valuable suggestion by the reviewer.

Reviewer comment (Experiments):
In section 4.1.1, the authors claim that one of the reasons their method works is because of the spatial broadcast decoder independently decoding for each slot. This is supported in the appendix by an experiment on SLATE which uses a decoder where the slots are not completely independent. This seems potentially limiting as several recent works in scaling object-centric learning (OCL) to realistic scenes [2, 3, 4] rely on decoders where each slot may not be decoded independently. This may limit the applicability of this method to more realistic scenes that are supported by those OCL methods.

Author response:
As the reviewer correctly noted, recent research has been integrating large decoders, allowing for interactions between slots to improve decoding quality. (In this response, we will refer to this approach as a "mixture decoder.")

Taking a slightly different viewpoint from the aforementioned research, our study reevaluated the concept of independence decoding, as demonstrated by the spatial broadcast decoder proposed in the original Slot Attention. We believe that independent decoding holds value comparable to the excellent decoding quality achieved by mixture decoders. It's crucial to note that in this research, we empirically and theoretically demonstrated that independent decoding (not only the spatial broadcast decoder but also the transformer-based SRT decoder) divides image-level loss into object-level loss. However, this does not imply that mixture decoding cannot achieve the same; it's just that success is not guaranteed, as illustrated in the SLATE case in the appendix.

Future research could explore how object-centric learning varies based on the decoder's configuration. Additionally, structuring a decoder with both mixture decoding and slot-wise decoding as two branches holds promise for achieving high-level decoding quality while learning controllability. We look forward to investigating these aspects in upcoming studies.

Reviewer comment (Experiments):
The second claim in section 4.1.1, that the use of ARK is important, does not seem to be supported by any experiments. How well does this method work with vanilla slot attention? Is ARK required for this method to work?

Author response:
In fact, ARK is not necessary for our method. However, as shown in the SLASH paper, the original Slot Attention (SA) suffers from a bleeding issue, wherein the attention or segmentation masks for objects leak into the background. As we mentioned in the proof in the appendix, the performance of object discovery plays a significant role in object-level controllability. If our proposed training scenario arises where bleeding issues do not occur in the original SA, it can be achieved without the need for ARK.

The use of ARK is not intended to enhance object discovery performance in a single training session; rather, it is employed to ensure consistent results across multiple experiments. Thanks to the reviewer’s feedback, we acknowledge the necessity of providing more detailed descriptions of these aspects, and also experimental results. We are conducting the experiments with the original Slot attention and will add the results with the descriptions in the revision during the rebuttal period.

Reviewer comment (Experiments):
I could not find which datasets are used for Table 1 and section 4.3 (Table 3 and Figure 6). The fact that segmentation quality is maintained and representation quality potentially improved is an important result. I would be curious about these results broken down by datasets.

Author response:
Thank the reviewer for finding the lack of description regarding the datasets used in our study. The results are from CLEVR6. 

As you have already figured out, the focus of our study is not to enhance the performance of the object discovery, and the experiment results support our claim that our method does not negatively impact the performance of the existing OCL model. 

Nevertheless, we also acknowledge the importance of including results from other datasets, such as CLEVRTEX and PTR, which would show the robustness and applicability of our method. We plan to carry out experiments on these datasets and intend to include these additional results in the appendix if the results are ready during the ongoing discussion. Even though we cannot include the experimental results in the revision, we will incorporate them in the following version of our paper.

Reviewer comment (Reproducibility):
In the appendix, the authors mention that the scaling augmentation takes into account the predicted attention maps between the encodings and the slots to handle the translation of objects during scaling. I am a bit confused about this. Does this mean that the augmentation changes as the model gets trained better? Or does this use some other pre-trained Slot Attention encoder?

Author response:
We appreciate the reviewer for posing this question since this question, we believe, signifies the reviewer's dedication to comprehending our work in detail, and we value the effort invested in understanding our paper thoroughly.

As the model undergoes training, we obtain more precise calibration results, corresponding to the more accurate predicted attention maps. However, the thing is that we do not use either the augmentation curriculum as the model gets trained better or the pre-trained encoder. Initially, we also considered equipping the model with object discovery and translation abilities before employing scaling augmentation during training. However, we found that a straightforward training curriculum, incorporating all types of augmentation (scaling, translating, and tinting) from the first epoch, proved effective without compromising performance in object discovery and manipulation. To maintain simplicity and clarity in presenting our work, we opted not to include any curriculum learning components.

Reviewer comment (Presentation):
In terms of the presentation, I was initially unsure of the significance of sustainability until I saw the experimental results in section 4.2. For clarity, I would suggest showing a motivating example earlier in the text to explain the necessity of the AIM and SCLoss components.

Author response:
We fully agree with the reviewer's suggestion and are considering the incorporation of motivation for sustainability into Figure 1 in the main paper. We appreciate the reviewer's feedback and will update the figure in the upcoming revision to reflect this addition.

Reviewer comment (Evaluation):
Are the qualitative examples cherry-picked? Are there common failure scenarios the reader should be aware of?

Author response:
In addition to the failure examples provided in the extreme durability test section of the appendix in the submitted paper, we agree with the reviewer’s feedback on the general failure cases. We will append more failure examples in an upcoming revision and notify the reviewer of the update. We hope to give a deeper understanding of and an insight into our method.

Reviewer comment (Experiments):
I am confused about the use of the SRT decoder in some of the experiments since that method does not have any notion of slots. Was this supposed to be the OSRT decoder [1]?

Author response:
Thanks again for the reviewer's delicate question. We use SRT decoder rather than OSRT decoder. The reason we chose the SRT decoder is that it is simple and allows for independent decoding for each slot, while the OSRT decoder, due to its incorporation of "slot mixing," does not align with our requirement for independent decoding.

The original SRT decoder was designed to operate at the image level, and the OSRT paper introduced a modification to decode slots simultaneously as shown in the Figure 2 in the OSRT paper. However, we slightly modified it to decode each slot independently following the spatial broadcast decoder.

This selection was made to demonstrate that our proposed method is not limited to CNN-based spatial broadcast decoders used in Slot Attention but can robustly operate within transformer-based decoders as well, given the appropriate conditions for independence.

We will add more details in the upcoming revision.