Paper: Learning Object-Centric Representation via Reverse Hierarchy Guidance (ICLR-2024)

Reviewer comment (Experiments):
In the main text, the proposed method is only evaluated on relatively simple, synthetic datasets. As shown in the supplement, the method can be combined with state-of-the-art object-centric models that scale to natural images, but the performance improvements do not seem to be significant.

Author response:
In the supplemental material, the performance of DINOSAUR is directly taken from the report in [1]. However, DINOSAUR doesn't release public models. Therefore, we are not improving on their basis. Instead, we obtain a baseline model by simply changing the input and reconstruction objective of the network to DINO features, which achieve a bit lower performance than reported in [1]. This reduces the performance gains from our method. 

We have recently taken a closer look at the training tricks of previous work [1], for example, they reduce the dimensions of pre-trained features to lower dimensions at the input side and recover to the original dimensions at the output side. We find that these modifications can improve the clustering performance of the network with less computation. We apply these tricks to our model and retrain the model on real-world datasets. In the table below we provide a comparison of model performance before and after retraining. 

|Model|MOVi-C|COCO|
|:-:|:-:|:-:|
|DINOSAUR|67.20|40.50|
||||
|original reproduction|65.68|37.90|
|+Train-RHG|67.48|40.57|
|+Infer-RHG|70.88|40.80|
||||
|new reproduction|67.08|38.74|
|+Train-RHG|70.50|41.02|
|+Infer-RHG|73.00|41.14|

After more thorough training of the network, we can come to two conclusions: (i) our performance benefits from better network settings, reaching +5.80% and +0.64% ARI scores compared with DINOSAUR; (ii) Compared with our reproduced baseline model, both Train-RHG and Infer-RHG bring remarkable performance gains.

Reviewer comment (Theory):
The mathematical description of the method in Section 3 is imprecise at several places. How exactly are the mappings $\mathcal{K}$ and $\mathcal{Q}$ defined? I.e., what are the dimensionalities of the quantities involved in equation 3? In equation 4, the term in the summation does not depend on the summation index $K$.

Author response:
We'll add a more detailed description of the symbols in section 3. The mapping $\mathcal{K}$ and $\mathcal{Q}$ doesn't change the dimensionality of features, still retaining $C=64$ as described in section 4.1. In addition, the summation index $K$ in Equation 4 appears due to a typo error, and it should be removed. In the updated paper, we have corrected these problems.

Reviewer comment (Evaluation):
Does the Object IoU metric include evaluating IoU for the background segment?

Author response:
The computation of Object IOU doesn't consider background pixels because we focus on the ability of the model to distinguish objects.

Reviewer comment (Evaluation):
How were the object sizes chosen that were used to differentiate small, medium and large objects? Which fraction of the objects is small, medium and large, respectively? Beyond the aggregated evaluation, it could be helpful to present a scatter plot of object size vs IoU for all objects in the evaluation set.

Author response:
We divide objects by their size distribution. Specifically, we count the number of pixels that all objects occupy in the ground truth, and we classify objects into small, medium and large on a ratio of about 1 to 5 to 1. 

A scatter plot of object size vs IoU is a better way to express how the model's ability to discover objects varies with the size of objects. According to your suggestion, we have redrawn a plot and put it in the latest update, you can check it in the updated paper.

Reviewer comment (Experiments):
Which performance is achieved when the features are clustered using a conventional approach such as normalized cuts instead of Slot Attention?

Author response:
Considering the similarity of slot attention and k-means algorithm, we use k-means to cluster the low-level features output by the CNN encoder. We take the CNN encoder from two networks, namely RHGNet and BUNet in the paper, and cluster them respectively. After clustering the features of RHGNet, we obtain an ARI-FG score of 52.29%, while the clustering score of BUNet is 41.42%. The results verify that RHGNet learns better features.

Reviewer comment (Evaluation):
At inference, the model needs to be run for multiple times to get a performance gain. This increases computational time.

Author response:
The network can either quickly extract objects in the scene through a pure bottom-up inference, or it can run the network multiple times and search for a more precise perception through a top-down pathway. When N is set to 1, the network only runs the bottom-up process, which does not require more computational complexity, but still outperforms other SOTA models. When the top-down path is started during inference (Infer-RHG in the paper), the network requires more computation, and the performance will further increase. In the paper, every time we compare performance with other models, we list two performances that correspond to situations where N is set to 1 and 10 respectively.

The performance improvement of Infer-RHG comes from the selection of initial slot values. Random initial values in slot-attention will produce different attention distributions, eventually leading to performance differences, as shown in Figure 3 in the paper. This is similar to human visual perception, where sometimes we don't pay attention to objects with low saliency. Our model obtains multiple attention distributions through different initial slots so that these objects have a greater chance of being found. Using Top-down conflict as a signal, we pick out the perception that is most likely to find these objects, thus discovering small objects.

Reviewer comment (Evaluation):
I think it is worth acknowledging some limitation observed in the visualization. One of the examples in ObjectsRoom of Figure 4 actually shows the proposed model is the only one that hallucinates the yellow color of the floor seen through the hollow part of the triangle shape. From the supplementary material, it seems often "fill in" the hollow part of the triangle in the segmentation mask, which should actually be background. From visual inspection, it seems that for the more challenging datasets such as CLEVRTex, missing small and partial objects is still frequent. And the model appears to often ignore the detailed texture in reconstruction but only captures the average color. Of course all models have limitations, I think it is worth pointing these out in discussion, and it will be interesting to postulate why texture often gets ignored.

Author response:
The top-down pathway demands the network to retain more general features because slots need to match all the features in the area they occupy, which causes slots to tend to reconstruct an "average" object that represents the overall features of the area while discarding more detailed features such as textures. This sometimes causes the model to converge to less precise reconstructions, but this doesn't always appear. We examined three models on ObjectsRoom trained with different random seeds. The model used for display had the problem described in Weakness, while the other two performed correctly, and all three models had an ARI score of 87% to 88%. We have updated the paper and replaced these figures with a correctly reconstructed version.

In addition, note that our displayed images are not carefully selected, but are directly chosen by taking out the first several images from test sets, thus the displayed result represents the average performance of our model. Among the 18 images shown in the supplementary material, a total of 107 objects appear, and our model misses 4 of them, which has made improvement compared with the current models shown in Figure 4 in the paper.

Reviewer comment (Evaluation):
Although conceptually it makes sense that top-down modulation should help in object perception, because obviously the brain uses it, the computational mechanism of why it helps in the experiments presented in the paper still remains puzzling to me. I am happy to see the analysis in Figure 7 of the intra-object vs. inter-object feature variance and Figure 6 illustrates the failure of bottom-up only network. But my question is: what even caused these improvement. My guess is that "in-principle", when the top-down attention from the slot includes some aggregated information over pixels of the objects, it is possible for the low-level features to be biased towards such aggregated information. But "in-principle" does not mean this is guaranteed. Moreover, what drives the network being able to learn to detect small objects better? It sounds that the paper indicates that the conflict loss term is the guidance, but the conflict term is simply the conflict between two masks that are both internally generated and to be learned, with no additional inductive bias or teaching signals by introducing the top-down pathway.

Author response:
Conflict loss just drives $A$ to get closer to $M$. We add Information Integrity Loss $L_\mathrm{info}$ to avoid the situation that the network learns to decrease conflict loss by removing an object from low-level features. We use a shallow decoder to reconstruct the image from output features of the CNN encoder, and $L_\mathrm{info}$ is the reconstruction loss. This loss guarantees the existence of all objects in low-level features.

$A$ is initially random and is meaningless at this time, thus we set the weight of conflict loss to 0 at the beginning. Using only reconstruction loss, slot attention can discover objects in a simple scene in the form of object mask $M$. This preliminary perception may miss objects, but it is at least "broadly correct". We think of $M$ as a pseudo label and use it to supervise the CNN encoder: it learns a segmentation task through conflict loss. Despite given annotations with some errors, the encoder can still effectively learn object concepts, which in turn makes slot attention clustering easier, thus obtaining better object discovery results.

The authors of [1] experimented with the number of slot attention iterations during training, finding that more iterations may decrease performance. We list experimental data on the number of slot attention iterations from their paper in the table below:

|Iterations|ARI-FG on CLEVR6|
|:-:|:-:|
|1|85.2|
|2|98.0|
|3|98.8|
|4|98.7|
|5|92.0|

This suggests that the depth of the network is not necessarily beneficial for learning Object-Centric representations. Our top-down path does not introduce additional depth to the network, but rather introduces a shortcut that directly connects the perception of the upper level (slot) with the lower level (CNN features), thus benefiting the learning of low-level features.

[1] Object-Centric Learning with Slot Attention

Reviewer comment (Experiments):
One guess is that the major contributor for the better ability of detecting small objects is just the extra depth in the attention map pathway introduced by the top-down attention. If so, would introducing more iterations in the original slot attention model achieve similar improvement? I wonder if the authors can elucidate the actual mechanism.

Author response:
The authors of [1] experimented with the number of slot attention iterations during training, finding that more iterations may decrease performance. We list experimental data on the number of slot attention iterations from their paper in the table below:

|Iterations|ARI-FG on CLEVR6|
|:-:|:-:|
|1|85.2|
|2|98.0|
|3|98.8|
|4|98.7|
|5|92.0|

This suggests that the depth of the network is not necessarily beneficial for learning Object-Centric representations. Our top-down path does not introduce additional depth to the network, but rather introduces a shortcut that directly connects the perception of the upper level (slot) with the lower level (CNN features), thus benefiting the learning of low-level features.

[1] Object-Centric Learning with Slot Attention

Reviewer comment (Reproducibility):
I would like to get a confirmation that no component of the network is pretrained on other tasks. If some of them are pre-trained, please explain.

Author response:
All the models on synthetic datasets are trained from scratch without using pretraining weight. Models on real-world datasets use the output feature of DINO to provide reconstruction objectives, and the pre-trained models have been indicated.

Reviewer comment (Experiments):
At inference time, the model is run N times and the one with the smallest conflict C is chosen. I wonder whether the performance improvement mainly comes from this selection process. What happens to the performance if you set N to 1? I wonder if the authors think that this variation has some similarity to the brain? Perhaps the brain does not always detect all objects with one (or two) glances. The ones being missed by the brain depends on what a person's initial top down attention is on.

Author response:
The network can either quickly extract objects in the scene through a pure bottom-up inference, or it can run the network multiple times and search for a more precise perception through a top-down pathway. When N is set to 1, the network only runs the bottom-up process, which does not require more computational complexity, but still outperforms other SOTA models. When the top-down path is started during inference (Infer-RHG in the paper), the network requires more computation, and the performance will further increase. In the paper, every time we compare performance with other models, we list two performances that correspond to situations where N is set to 1 and 10 respectively.

The performance improvement of Infer-RHG comes from the selection of initial slot values. Random initial values in slot-attention will produce different attention distributions, eventually leading to performance differences, as shown in Figure 3 in the paper. This is similar to human visual perception, where sometimes we don't pay attention to objects with low saliency. Our model obtains multiple attention distributions through different initial slots so that these objects have a greater chance of being found. Using Top-down conflict as a signal, we pick out the perception that is most likely to find these objects, thus discovering small objects.