Paper: DisFormer: Disentangled Object Representations for Learning Visual Dynamics Via Transformers (ICLR-2024)

Reviewer comment (Experiments):
The experiments are not very persuasive because the datasets are simple toy datasets. The original SlotFormer has done experiments on dataset CLEVRER, which is harder than the 2D shapes/circles used in this paper. Are there reasons why CLEVRER, or CLEVRER with more complex textures, are not used for experiments?

Author response:
We used OBJ3D which has similar visual complexity of CLEVRER. We are still working to compile our numbers on CLEVRER and hope to shape them during discussion phase.

Reviewer comment (Experiments):
No ablations are provided. The model contains multiple components, but there are no ablation experiments to study the effect of each component. For example, the effect of recovering object representation versus image representation, the effect of learning the slots to be a linear combination of concepts, the number of slots, number of concepts, etc. should be studied.

Author response:
We are working on specific ablations and hope to shape them during discussion phase.

Reviewer comment (Experiments):
Not enough experiments are shown to prove the representations are “disentangled”. This disentanglement is the major advantage of the method. However, only several examples are shown in Fig. 2 to show the disentanglement. Quantitative results, or visualization of the learned slots (e.g. using t-SNE) would be preferred.

Author response:
We have provided disentanglement evaluation in appendix.

Reviewer comment (Reproducibility):
Missing experiment details. The training details including the hyperparameters are not provided. Some critical parameters (e.g. number of slots/concepts, loss weights) should be discussed.

Author response:
Added in the appendix.

Reviewer comment (Experiments):
A comparative evaluation with another disentangled object-centric representation learning method, SysBinder, is lacking, despite its mention in the text.

Author response:
We would like to point out to reviewer that Sysbinder discovers object representation for images and we would like to discover object representations for video. We still performed used frame by frame sysbinder and found satisfactory results on 2D BS dataset.

Reviewer comment (Evaluation):
The quantitative results presented could be bolstered with more illustrative examples, showcasing scenarios where the proposed model excels in comparison to its counterparts.

Author response:
We are working on compiling example images and hope to post them in the discussion phase.

Reviewer comment (Presentation):
In the introduction, could you incorporate a high-level architectural diagram or illustration of your model? This addition would facilitate a clearer and more immediate understanding for readers.

Author response:
We are working on high level diagram and hope to post them in the discussion phase.

Reviewer comment (Experiments):
The Mask R-CNN in your methodology is trained using a labeled dataset. This seems to introduce a discrepancy since the other models under comparison do not utilize labeled data. Could you perhaps validate your model's performance using Slot-Attention or another slot-based model as a substitute for Mask R-CNN?

Author response:
We have added the details in the appendix section

Reviewer comment (Presentation):
In Figure 1, “Note that each block has its own set of concept vectors; thus, C = || rj=1C j where r represents the number of blocks.”. Why? Shouldn’t the input $mathcal{C}$ be shared for every block?

Author response:
Each block has it’s own set of concept vectors. For instance the block(s) representing color would have concept vectors as {red, green, blue} etc. whereas those representing size would have a different set of concept vectors. two different attributes cannot be a linear combination of same concept vectors. Thus concept C is not shared across all the blocks.

Reviewer comment (Writing):
In section 3.3, “Let the transformer encoder output be δˆ s ti,b.” For clarity, could you specify that this is the output corresponding to $s_t^{i,b}$?

Author response:
Yes it is output corresponding to $s_{t}^{i,b}$

Reviewer comment (Writing):
Section 3.4 appears to be complex. To ensure my understanding is correct: is the process essentially projecting the concatenated block vectors through matrix $U$, calculating the Cosine Similarity, and then aligning the most similar representations as the same object? Further clarification and polishing of this section would be beneficial.

Author response:
Yes the reviewers understanding is correct.

Reviewer comment (Experiments):
How is the object position estimated within your model, as detailed in the experiment section? You've mentioned that position error is not reported for SlotFormer due to its lack of explicit object mask handling; does your model operate in a similar manner?

Author response:
We have added position error computation section in appendix

Reviewer comment (Experiments):
The examples in Section 4.4 intended to illustrate disentanglement seem to be lacking. Instead of swapping both color and shape attributes, could separate demonstrations of each be more effective in showcasing the model’s capabilities?

Author response:
In our choice of dataset the color and shape are correlated and are not represented by separate blocks.

Reviewer comment (Experiments):
Regarding Section 3.4, why is the permutation module utilized only during training and discarded during testing? Could its application during testing, potentially for aligning the input order of the Transformer module, lead to enhanced performance?

Author response:
That's very interesting suggestion. We are currently working on it and hope to add results in discussion phase.

Reviewer comment (Theory):
For the Dynamic loss calculation in Section 3.6, why is the comparison made between the weights of the concept vectors rather than the block vectors? Additionally, could you provide a definition for $\hat{w}_t^{i,b,j}$?

Author response:
We found that comparing weights of concept vectors result in faster convergence. The $\hat{w}_{t}^{i,b,j}$ is the predicted weight of $b^{th}$ block and its jth concept vector. It is output of transition model.

Reviewer comment (Experiments):
This paper proposes a range of novel model components, a detailed ablation analysis is however missing. The only comparison is to a model that replaces the pretrained object extractor with an MLP. So it is not clear to which degree the different components contribute to the improved performance of the model.

Author response:
Most of our model components are dependent on each other, so the entire system may not work if we remove any of those. At the same time, if the reviewer has specific suggestions on which ablations to try with respect to moving out certain components, we will be happy to report those. There was a typo in experiment section 4.2 where instead of object extractor we replaced block extractor with MLP.

Reviewer comment (Evaluation):
The paper does not discuss any particular success or failure cases of the proposed model. Are there specific situations which are predicted better by the proposed model? How do these relate to the model components introduced in the paper?

Author response:
This is a very important suggestion. We are working on compiling a list of failure cases of our model. We hope to upload this to these before the end of the discussion phase.

Reviewer comment (Evaluation):
The model learns a constant concept space $C$. Do the concept vectors correspond to interpretable attributes? Is there a separation of object attributes into those that contribute to dynamics and others that do not, as asked in the abstract?

Author response:
In our ablation experiment, we rigidly assigned concept weights for blocks to values of 0 and 1. Unfortunately, the decoding process with these blocks failed to produce any meaningful images. This experiment has conclusively shown that, under these conditions, individual concept vectors do not effectively represent interpretable attributes. About the second part of the question we are working defining these sepration. We hope to share the results before the end of the discussion phase.

Reviewer comment (Evaluation):
The disentanglement of object attributes is a core motivation behind this work. The disentanglement is however not quantitatively evaluated.

Author response:
We have added the appendix section of disentanglement evaluation.

Reviewer comment (Experiments):
I thought the explanation of the task and the actual problems came too late and were not depicted well enough for what the paper was trying to accomplish. 3 of the 4 datasets used I would call "toy" datasets of simple bouncing balls. The 3D dataset seems more visually driven, but even that uses CLEVR which is known to be visually simple. In the whole paper there is only a single picture depicting the actual task, and the tasks are only described in the experiment section. For a paper with "Visual Dynamics" in the title, I would have expected less toy problems, and more explanation to what the actual problems were. I think this could have been a stronger paper if it had foregone the visual component and worked directly with low level data.

Author response:
Thanks for pointing this out. We have added a task description in the introduction. While we agree that we have experimented only with visually simple datasets, we would like to to argue that disentanglement is a hard problem, and some of the SOTA techniques (Symbinder [1]) have only worked with images, which we have extended to video in our work. Further, SlotFormer [2] (which is a very recent work) for video prediction, also works on only simple rigid-body dynamics. We are working on compiling example images and hope to post them in the discussion phase.

Reviewer comment (Writing):
I found several explanations in the paper to be confusing/lacking detail. One key concept in the paper was that of a "Block". Here is the explanation from the paper: "Recently, (Locatello et al., 2020) in the paper on slot-attention proposed architecture for unsupervised discovery of objects via iterative refinement of what is referred to as a slot. Each slot binds to an object via attention over the entire image in their work. We extend their idea to the case of objects, where each slot now represents a block, which is iteratively refined by taking attention over a latent object representation". My rephrasing of this is "rather than take attention over the entire image to get a representation, we first extract an object mask, and then take attention over that". I'm not sure that is correct, and even if it is, I don't understand why they need attention if they already have a mask.

Author response:
We thank the reviewer for the suggestion and we agree with reviewers rephrasing. We would like to point out that we are not taking attention to the entire image and rather take attention to the image masked out by an object mask. We are aiming for disentangled representation of an object and thus we add that prior by inferring blocks which is iteratively refined by taking attention over latent object representation.

Reviewer comment (Experiments):
Another instance of this was the Permutation module. I did not understand the motivation behind it considering it is only used at training time. The authors say they learn a permutation to match up objects from one frame to the next, but that they supervise this permutation with ground truth knowledge. Then at test time, this component is removed. If you are already using ground truth information at train time, and at test time you don't use the module, why not just permute the features directly instead of learning a permutation matrix?

Author response:
We want to highlight to the reviewer that the permutation gadget undergoes training in a self-supervised manner and we don’t have ground permutations. Its sole purpose is to align predicted object blocks with ground object blocks for dynamic loss computation. Importantly, during testing, there is no necessity for loss computation, and therefore, the permutation module is excluded from the testing process. We have added more details in the appendix.

Reviewer comment (Experiments):
I thought the experiments in this paper were lacking in showing what the authors claimed. Predicting simple rigid body circle motion and even rigid body 3D synthetic CLEVR motion is not really convincing since it is such a problem removed from the complexities of the real world.

Author response:
We are unsure of the specific claim that the reviewer is referring to. To the best of our knowledge, we have not made any claim regarding our ability to do well with deformable objects. If the reviewer can point out a specific instance, we would be happy to correct it. While we agree that we have experimented only with rigid body surfaces, we would like to to argue that disentanglement is a hard problem, and some of the SOTA techniques (Symbinder refer) have only worked with images, which we have extended to video in our work. Further, SlotFormer [] (which is a very recent work) for video prediction, also works on only simple rigid-body dynamics. Hence, we strongly believe our setting is commensurate with some of the recent works in this area.

Reviewer comment (Experiments):
To convince me that there is a decoupling happening, it is crucial to have an experiment that directly probes this decoupling. The ablation study in 4.4 seems to do that in some way, but I don't understand the experimental setup from the explanations, and again, it is only on a toy setup so I can't say whether it would generalize to more complex scenes. It sounds like somehow the authors took the embeddings from one color setup and swapped them with another color setup to look at the output. I guess it seems trivially obvious that the output should change color, but does that prove that the dynamics module only encodes dynamics or just that masking a part of the image results in color features for that part of the image.

Author response:
In response to the reviewer's suggestion, we have incorporated additional evaluation and experiments illustrating the decoupling of blocks in the appendix section F. Importantly, we emphasize that these blocks are learned in a fully unsupervised manner. One such instance is the block that has learned to represent color, as demonstrated in section 4.4. However, the final part of the review is currently unclear to us, and we would appreciate further clarification.

Reviewer comment (Experiments):
The DisFormer seems to be another reasonable ablation on paper but not carried out as well as I'd liked. Details are sparse, but it sounds like the entirety of MaskRCNN was replaced with an MLP, which doesn't seem like a reasonable substitution "by replacing the object extractor (refer Section 3) by an MLP to create dense object representations".

Author response:
That was a typo and it should have been "by replacing the block extractor”. We have fixed it in the paper.

Reviewer comment (Theory):
I would like the authors to explain in more detail why the permutation model was necessary if it was only used during training.

Author response:
We have addressed the need for a permutation model by including a specific example in the newly added content within the appendix section C. Furthermore, we have enhanced and clarified the details in section 4.4 based on your feedback.

Reviewer comment (Experiments):
I would also like the authors to explain the DisFormer and experiment 4.4 better.

Author response:
We have addressed the need for a permutation model by including a specific example in the newly added content within the appendix section C. Furthermore, we have enhanced and clarified the details in section 4.4 based on your feedback.