Paper: Learning to Compose: Improving Object Centric Learning by Injecting Compositionality (ICLR-2024)

Reviewer comment (Experiments):
Most experiments are conducted on the synthetic objects, more experiments on realistic complex scenes, e.g., COCO, are desired.

Author response:
We appreciate the valuable comments. First, we would like to kindly note that learning object-centric representation in complex real-world images is widely perceived as challenging, and existing works rely on frozen pre-trained encoders and decoders to alleviate the problem. Since freezing the image encoder with a shallow slot attention module limits the opportunity to investigate the true impact of the compositional objective, we follow the standard settings that employ the synthetic data for evaluation.

Nonetheless, we agree with the reviewer that the synthetic scenes are insufficient to demonstrate the scalability of our compositional objective to complex scenes. Following the reviewer’s suggestion, we conducted preliminary experiments on the BDD100K dataset, which contains real-world driving scenes with many objects (cars, signs, buildings, trees, pedestrians, etc.) as well as complex interactions among objects and with the environment (occlusion, shadow, lighting, reflection, etc.). Following the prior work [1], we employed the pre-trained DINOv2 [2] as an image encoder and Stable Diffusion [3] as the generative prior.  We then apply warm-up training for the one-shot decoder for 140K steps with an auto-encoding objective, and apply our compositional objective for the next 100K steps. 

The results are summarized in Section B.7 in the Appendix (Figures 10 and 11). Based on visualization of the slot attention map (Figure 10), we observe that our method captures composable object instances not only from the foreground such as cars, buses, and trucks, but also from static environments such as road signs, trees, buildings, sidewalks, etc. In contrast, learning with only diffusion loss often separates an object into multiple slots, or contains multiple objects into a single slot. When composing two images by mixing their slots (Figure 11), we observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. It appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes even global illumination change caused by removing the sun. Our method also generates more accurate compositions compared to the baseline, which often fails to adhere to the addition or removal of the objects. 

To summarize, our preliminary experiments showed promising results that the proposed compositional objective can scale to complex scenes and model complex correlations among objects. We believe that incorporating the compositional objective in self-supervised learning can greatly enhance object-centric learning in complex scenes through end-to-end training with the deep encoder, which we plan to investigate in future work.

[1] Jiang et al., Object-centric slot diffusion, NeurIPS 2023.  

[2] Oquab et al., DINOv2: Learning robust visual features without supervision, arXiv preprint 2023.  

[3] Rombach et al., High resolution image synthesis with latent diffusion models, CVPR 2022.

Reviewer comment (Experiments):
More compared methods should be included, e.g., DINOSAURE [1].

Author response:
We appreciate the comment. As suggested by the reviewer, we included the DINOSAUR as an additional baseline and reported the comparison results in the table below. Although it performs reasonably well in CLEVRTex and PTR datasets, it fails dramatically in MutiShapeNet and Super-CLEVR datasets; we observe that it is because DINOSAUR encodes the entire foreground objects into a single slot in these datasets. Such failures are attributed to the frozen image encoder, which fails to generalize to domains unseen during the pre-training. It shows that learning the image encoder end-to-end with object-centric representation is important for building a robust and general object-centric learning framework. 

We would also like to note that the main contribution of the DINOSAUR is employing the pre-trained features in object-centric learning, which is orthogonal to our contribution and hence can be combined together. In Section B.7 in the Appendix, we provided results on a real-world dataset, where we incorporated the compositional loss on top of the pre-trained DINO.  


| **CLVERTex** | **FG-ARI (↑)** | **mIoU (↑)** | **mBO (↑)** |
|--------------|:--------------:|:------------:|:-----------:|
|   DINOSAUR   |      70.67     |     48.20     |    50.03    |
|    SLATE+    |      71.29     |     52.04    |    52.17    |
|      LSD     |      76.44     |     72.32    |    72.44    |
|     Ours     |    **93.06**   |   **74.82**  |  **75.36**  |


| **MultiShapeNet** | **FG-ARI (↑)** | **mIoU (↑)** | **mBO (↑)** |
|-------------------|:--------------:|:------------:|:-----------:|
|      DINOSAUR     |      1.99      |     12.53    |     20.30    |
|       SLATE+      |      70.44     |     15.55    |    15.64    |
|        LSD        |      67.72     |     15.39    |    15.46    |
|        Ours       |    **89.80**    |   **59.21**  |   **59.40**  |

| **PTR**  | **FG-ARI (↑)** | **mIoU (↑)** | **mBO (↑)** |
|----------|:--------------:|:------------:|:-----------:|
| DINOSAUR |      58.23     |     23.20     |    26.48    |
|  SLATE+  |    **91.25**   |     14.10     |    14.22    |
|    LSD   |      61.10      |     10.18    |    10.33    |
|   Ours   |      90.65     |   **40.89**  |  **41.45**  |


| **SuperCLEVR** | **FG-ARI (↑)** | **mIoU (↑)** | **mBO (↑)** |
|----------------|:--------------:|:------------:|:-----------:|
|    DINOSAUR    |      2.92      |      5.70     |    8.93    |
|     SLATE+     |      43.73     |     29.12    |    29.49    |
|       LSD      |      54.79     |     14.12    |    14.43    |
|      Ours      |    **63.08**   |   **47.17**  |  **48.03**  |

Reviewer comment (Experiments):
There are some recent works focusing on the binding ability of the slots to specific object types or properties, e.g., [2]. Is it possible to employ the binding ability to enhance the interpretablity of slot mixing and further enahnce the composition ability?

Author response:
We appreciate the insightful comment. Indeed, our work investigates a general problem of learning compositional representation and employs simple mixing strategies for generality.  We believe that incorporating the binding ability of the slots can enhance the mixing strategy by reducing the search space for the valid slot combination, and improving the interpretability of the composite representation which is useful especially in manipulation tasks.

Reviewer comment (Experiments):
Why background slots do not conflict: Could the authors explain whether / why, if not, background slots conflict when sampling slots with the random mixing strategy? In this case 1) is there any reason to expect canonical slot ordering, such that two background slots are unlikely to be sampled or 2) any reason why if two background slots are sampled, this does not lead to invalid scenes?

Author response:
**A2.** We appreciate the comment. In the random mixing strategy, each slot is randomly and independently initialized in each image, and combined into a composite representation by randomly sampling slots from two images. Due to this randomness, there is a chance that mixed slots (*i.e.*, composite representation) lead to ***invalid*** combinations, such as the ones composed only with the objects without background. To avoid this, we share the slot initialization between two images, and mix the slots ***exclusively*** according to their slot indices (*e.g.*, we can sample i-th slot either from the first or second images, but NOT from both). Formally, let $I_1$ and $I_2$ be a random partition of slot indices  *i.e.*, $I_1\\cup I_2=\\{1,...,N\\}, I_1\\cap I_2=\\emptyset$. Then we construct the composite slot by $\\mathbf{S}^c=\\mathbf{S}^1_{I_1}\\cup \\mathbf{S}^2_{I_2}$, where $\\mathbf{S}^1$ and $\\mathbf{S}^2$ are slots extracted from the first and the second images, respectively. Since the slots are initialized identically between two images and the slots extracted from the identical initialization never appear twice in the composite representation, the model can learn to associate certain slot initialization to a specific part of a scene, such that the composite representation is ensured to be always valid. Indeed, we observed that the model learns to associate the background to a certain slot, as shown in Figure 6 in the appendix. We appreciate the comment and will revise the paper to clarify this point.

Reviewer comment (Evaluation):
On the relative contributions of Regularization+Shared Slot Initializations (Row 2) and the compositional prior (Row 1) in Table 2: It is interesting that Row 2 is roughly as effective as Row 1. Was there a significant difference in the qualitative behaviour of the models in these two settings; i.e. did Row 1 yield "better" slots, but lead to worse metrics for some other reason - or can we conclude that the bias towards compositional slot representations in both of these settings was comparably strong? (which would be somewhat surprising!). Perhaps the decoder receives a stronger learning signal in Row 2, and in Row 1 the generative prior acting by itself is constrained because the decoders don't learn as well, and so provide a weaker compositional signal?

Author response:
**A3.** We would like to first note that both the regularization and compositional loss (generative prior) contribute to enhancing the compositionality of the representation since they are computed from the composition of two images and provide learning signals orthogonal to the auto-encoding path. We observe that the two losses also promote complementary behaviors to the model; The composition loss provides a direct learning signal that the composite image should be realistic, promoting the compositionality of the slot representation. However, we observed that it sometimes captures loose object boundaries especially when the background is monotonic, since they are highly correlated hence compositional. On the other hand, the regularization loss encourages the part of the image captured by a slot to be consistent before and after composition, without considering the realism of the composite image. Combining these two, our method learns to capture compositional components of a scene to a slot while maintaining the consistent association between the slot and a scene, leading to the highest performance.

Reviewer comment (Experiments):
**Synthetic Data & Scalability**: The experiments are performed over synthetic data only. I recommend exploring scalability to real-world data too. This is especially important since it is unclear to me whether such an approach would scale well to more diverse real-world data, where there are correlations between object occurrence as well as appearance. I don’t know for sure but it might be the case that for realistic data the loss could damage the model’s learning compared to standard auto-encoding, by encouraging it to unlearn important object correlations. For instance, even for the images at figure 1, the lighting of the mixed objects does not fit the context, potentially making the model less aware of the impact of lighting conditions on the object’s appearance. For this reason I believe it is really critical to study this objective in the context of realistic data too.

Author response:
**A1.** We appreciate the valuable comments. First, we would like to kindly note that learning object-centric representation in complex real-world images is widely perceived as challenging, and existing works rely on frozen pre-trained encoders and decoders to alleviate the problem. Since freezing the image encoder with a shallow slot attention module limits the opportunity to investigate the true impact of the compositional objective, we follow the standard settings that employ the synthetic data for evaluation.

Nonetheless, we agree with the reviewer that the synthetic scenes are insufficient to demonstrate the scalability of our compositional objective to complex scenes. Following the reviewer’s suggestion, we conducted preliminary experiments on the BDD100K dataset, which contains real-world driving scenes with many objects (cars, signs, buildings, trees, pedestrians, etc.) as well as complex interactions among objects and with the environment (occlusion, shadow, lighting, reflection, etc.). Following the prior work [1], we employed the pre-trained DINOv2 [2] as an image encoder and Stable Diffusion [3] as the generative prior.  We then apply warm-up training for the one-shot decoder for 140K steps with an auto-encoding objective, and apply our compositional objective for the next 100K steps. 

The results are summarized in Section B.7 in the Appendix (Figures 10 and 11). Based on visualization of the slot attention map (Figure 10), we observe that our method captures composable object instances not only from the foreground such as cars, buses, and trucks, but also from static environments such as road signs, trees, buildings, sidewalks, etc. In contrast, learning with only diffusion loss often separates an object into multiple slots, or contains multiple objects into a single slot. When composing two images by mixing their slots (Figure 11), we observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. It appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes even global illumination change caused by removing the sun. Our method also generates more accurate compositions compared to the baseline, which often fails to adhere to the addition or removal of the objects. 

To summarize, our preliminary experiments showed promising results that the proposed compositional objective can scale to complex scenes and model complex correlations among objects. We believe that incorporating the compositional objective in self-supervised learning can greatly enhance object-centric learning in complex scenes through end-to-end training with the deep encoder, which we plan to investigate in future work.

[1] Jiang et al., Object-centric slot diffusion, NeurIPS 2023.  

[2] Oquab et al., DINOv2: Learning robust visual features without supervision, arXiv preprint 2023.  

[3] Rombach et al., High resolution image synthesis with latent diffusion models, CVPR 2022.

Reviewer comment (Experiments):
**Compositional Synthesis**: The results about compositional synthesis aren't the most compelling. I would like to see object mixing in cases that involve more interesting interactions among the objects (like reflection, more substantial occlusions, and matching of the appearance between the object and its environment). It does perform better than the baselines so that’s still an improvement.

Author response:
**A1.** We appreciate the valuable comments. First, we would like to kindly note that learning object-centric representation in complex real-world images is widely perceived as challenging, and existing works rely on frozen pre-trained encoders and decoders to alleviate the problem. Since freezing the image encoder with a shallow slot attention module limits the opportunity to investigate the true impact of the compositional objective, we follow the standard settings that employ the synthetic data for evaluation.

Nonetheless, we agree with the reviewer that the synthetic scenes are insufficient to demonstrate the scalability of our compositional objective to complex scenes. Following the reviewer’s suggestion, we conducted preliminary experiments on the BDD100K dataset, which contains real-world driving scenes with many objects (cars, signs, buildings, trees, pedestrians, etc.) as well as complex interactions among objects and with the environment (occlusion, shadow, lighting, reflection, etc.). Following the prior work [1], we employed the pre-trained DINOv2 [2] as an image encoder and Stable Diffusion [3] as the generative prior.  We then apply warm-up training for the one-shot decoder for 140K steps with an auto-encoding objective, and apply our compositional objective for the next 100K steps. 

The results are summarized in Section B.7 in the Appendix (Figures 10 and 11). Based on visualization of the slot attention map (Figure 10), we observe that our method captures composable object instances not only from the foreground such as cars, buses, and trucks, but also from static environments such as road signs, trees, buildings, sidewalks, etc. In contrast, learning with only diffusion loss often separates an object into multiple slots, or contains multiple objects into a single slot. When composing two images by mixing their slots (Figure 11), we observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. It appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes even global illumination change caused by removing the sun. Our method also generates more accurate compositions compared to the baseline, which often fails to adhere to the addition or removal of the objects. 

To summarize, our preliminary experiments showed promising results that the proposed compositional objective can scale to complex scenes and model complex correlations among objects. We believe that incorporating the compositional objective in self-supervised learning can greatly enhance object-centric learning in complex scenes through end-to-end training with the deep encoder, which we plan to investigate in future work.

[1] Jiang et al., Object-centric slot diffusion, NeurIPS 2023.  

[2] Oquab et al., DINOv2: Learning robust visual features without supervision, arXiv preprint 2023.  

[3] Rombach et al., High resolution image synthesis with latent diffusion models, CVPR 2022.

Reviewer comment (Novelty):
**Novelty & Related Works**: The specific proposed idea is novel, but object mixing as a form of regularization isn’t. For instance, it would be good to cite and discuss the “Object Discovery with a Copy-Pasting GAN” paper as a related prior work that also uses mixes over image to encourage object discovery.

Author response:
**A2.** We appreciate the valuable comment on the related work. We agree with the reviewers that Copy-Pasting GAN (CP-GAN) shares similar motivation to ours in a spirit of discovering object masks by maximizing the realism of a copy-and-pasted image. Here we outline two major differences as follows. Firstly, our work focuses on learning compositional representation, which can be further employed in downstream tasks or compositional generation tasks, while the CP-GAN focuses on discovering object masks. Secondly, CP-GAN simulates the composition of objects by a simple copy-and-paste on RGB space, while our method combines two scenes using the object (slot) representation and employ powerful diffusion decoder to render the composite scene. We will include these discussions in the paper.

Reviewer comment (Experiments):
**Baselines**: It would be good to also compare to a vanilla slot attention as a baseline.

Author response:
**A3.** We appreciate the comment.  As suggested by the reviewer, we additionally conducted experiments with vanilla slot attention. The results are summarized in the tables below. Overall, the performance of the vanilla slot attention is considerably lower than the other baselines and ours except in the PTR dataset where it achieves slightly better mIoU and mBO than ours. We observe that it is because the slot attention tends to capture tight object boundaries while missing some object parts (*e.g.*, legs of tables), which is favored in mIoU and mBO but results in much lower FG-ARI scores. In all other datasets (CLEVRTex, MultiShapeNet, and Super-CLEVR), however, the vanilla slot attention fails to capture meaningful object masks, often suffering from position bias or leveraging simple color cues due to weak pixel mixture decoder as commonly reported in the literature. We appreciate the comment and will add the results to the main paper. 

|        CLEVRTex      | FG-ARI (↑) | mIoU (↑) | mBO (↑) |
|------------------------|:----------:|:--------:|:-------:|
| Vanilla Slot Attention |    34.75   |   13.63  |  14.69  |
|         SLATE+         |    71.29   |   52.04  |  52.17  |
|           LSD          |    76.44   |   72.32  |  72.44  |
|          Ours          |    **93.06**   |  **74.82**  |  **75.36**  |


|   MultiShapeNet   | FG-ARI (↑) |  mIoU (↑) |  mBO (↑) |
|------------------------|:----------:|:---------:|:--------:|
| Vanilla Slot Attention |    69.26   |   16.49   |   16.6   |
|         SLATE+         |    70.44   |   15.55   |   15.64  |
|           LSD          |    67.72   |   15.39   |   15.46  |
|          Ours          |  **89.8**  | **59.21** | **59.4** |


|           PTR           | FG-ARI (↑) |  mIoU (↑) |  mBO (↑)  |
|------------------------|:----------:|:---------:|:---------:|
| Vanilla Slot Attention |    56.8    | **50.64** | **51.02** |
|         SLATE+         |  **91.25** |    14.1   |   14.22   |
|           LSD          |    61.1    |   10.18   |   10.33   |
|          Ours          |    90.65   |   40.89   |   41.45   |


|    Super-CLEVR    | FG-ARI (↑) |  mIoU (↑) |  mBO (↑)  |
|------------------------|:----------:|:---------:|:---------:|
| Vanilla Slot Attention |    26.92   |   13.63   |   14.69   |
|         SLATE+         |    43.73   |   29.12   |   29.49   |
|           LSD          |    54.79   |   14.12   |   14.43   |
|          Ours          |  **63.08** | **47.17** | **48.03** |

Reviewer comment (Theory):
**Slot Initialization**: I didn’t fully understand the intuition why a shared slot initialization should help avoiding bad mixes of objects. It seems to me that learning good combinations of objects could be achieved instead by not sharing the initialization scheme between them and letting them interact across the two images to coordinate valid object compositions. If possible please explain the initialization point further.

Author response:
**A4.** We appreciate the comment. In the random mixing strategy, each slot is randomly and independently initialized in each image, and combined into a composite representation by randomly sampling slots from two images. Due to this randomness, there is a chance that mixed slots (*i.e.*, composite representation) lead to ***invalid*** combinations, such as the ones composed only with the objects without background. To avoid this, we share the slot initialization between two images, and mix the slots ***exclusively*** according to their slot indices (*e.g.*, we can sample i-th slot either from the first or second images, but NOT from both). Formally, let $I_1$ and $I_2$ be a random partition of slot indices *i.e.*, $I_1\cup I_2=\\{ 1,...,N\\}, I_1\cap I_2=\\emptyset$. Then we construct the composite slot by $\\mathbf{S}^c=\\mathbf{S}^1_{I_1}\\cup \\mathbf{S}^2_{I_2}$, where $\\mathbf{S}^1$ and $\\mathbf{S}^2$ are slots extracted from the first and the second images, respectively. Since the slots are initialized identically between two images and the slots extracted from the identical initialization never appear twice in the composite representation, the model can learn to associate certain slot initialization to a specific part of a scene, such that the composite representation is ensured to be always valid. Indeed, we observed that the model learns to associate the background to a certain slot, as shown in Figure 6 in the appendix. We appreciate the comment and will revise the paper to clarify this point.

Reviewer comment (Writing):
**Diffusion**: It would be helpful to make it clearer how the denoising diffusion and auto-encoding fit together in the proposed framework.

Author response:
**A5.** Our diffusion decoder is trained with auto-encoding loss (Eq.(4) in the paper) while also used to compute the composition loss (Eq.(6)). In the auto-encoding path, the diffusion decoder learns the marginal distribution of data, while in the composition path, it is used to evaluate the likelihood of the composite image. We apply the stop gradient to the diffusion decoder in the composition path to ensure that it learns with only the true data distribution through the auto-encoding path. We will clarify the paper to make it more clear.

Reviewer comment (Evaluation):
**Super-CLEVR**: The ARI performance drop on super-clevr is particularly large, even though multiShapenet and PTR also include objects with subparts, and the latter even includes more complicated backgrounds. Do you have an intuition of why is that the case?

Author response:
**A6.** The ground-truth object masks of the Super-CLEVR dataset are much more fine-grained than the other datasets. Since the resolution of the attention map in our method and baselines is much coarser than the mask, we observe that it generally degrades the FG-ARI score for all baselines (*e.g.*, a small error in a low-resolution attention map can lead to a bigger penalty), especially when the objects are occluded by the others (*e.g.*, occluded by the spoke of a bike).