Paper: Looping LOCI: Developing Object Permanence from Videos (ICLR-2024)

Reviewer comment (Novelty):
My major concern about the paper lies in the limited contribution and generalizability of the proposed method. The design of the model is a bit complex and ad-hoc. The core contribution is the introduction of an inner loop that enables the model to imagine object dynamics without sensory inputs. To be more specific, the proposed model is encouraged to ignore the sensory input when the object is being occluded. This does not make sense when the camera is moving, as the camera motion (which needs to be inferred from video) should also be considered for inferring the view-centered object motion. Failing to account for that makes the model rather limited. I am interested in authors’ opinions about how to extend the current model to support this. It is interesting to see the current model’s performance on the LA-CATER Moving dataset proposed in [1] which contains camera movements.

Author response:
Dear reviewer - thank you for your time. 

we should have re-mentioned this in the direct reply to you, however, in our general reply we state: 

* We emphasize that LOCI-Looped has the potential to scale to more real world objects, more complex backgrounds and moving cameras, which is confirmed by a recently published related work on the MOVi-* and another state-of-the-art benchmark suite.
   * Note that this work mostly beats the state of the art in MOVi-e and the other benchmark suite (comparing to NeurIPS 2022 SAVi++ [1] paper and relating to ICLR 2023 DINOSAUR paper [2]).

Thus, while not in this paper, but it has been demonstrated that this method generalizes. 

This paper is about learning object permanence and learning to adaptively fuse inner and outer information - a challenge that has been around and has been tackled for decades cf. [3]. We solve this challenge for the first time without the provision of any mask/object information whatsoever.

We would highly appreciate if the reviewer finds the time to consider these critical aspects.

[1] Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos, December 2022. URL http://arxiv.org/abs/2206.07764. arXiv:2206.07764

[2] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the Gap to Real-World Object-Centric Learning, March 2023. URL http: //arxiv.org/abs/2209.14860. arXiv:2209.14860.

[3] Yuko Munakata, James Mcclelland, Mark Johnson, and Robert Siegler. Rethinking infant knowl- edge: Toward an adaptive process account of successes and failures in object permanence tasks. Psychological Review, 104(4):686–713, 1997. doi: 10.1037/0033-295x.104.4.686.

Reviewer comment (Experiments):
The evaluation of the paper is rather limited. The main experiments regarding object tracking only consider ADEPT, a synthetic object with a simple background and at most 3 moving objects. It is unknown how the method will perform on more complex and realistic datasets. I understand this is a common concern for a lot of compositional scene representation learning works. But given the limited technical contribution of the paper and the complexity of the proposed method, I believe a more comprehensive evaluation will make the paper stronger. For example, why authors do not evaluate on the CLEVRER and Aquarium as in the original Loci paper?

Author response:
Dear reviewer - thank you for your time. 

we should have re-mentioned this in the direct reply to you, however, in our general reply we state: 

* We emphasize that LOCI-Looped has the potential to scale to more real world objects, more complex backgrounds and moving cameras, which is confirmed by a recently published related work on the MOVi-* and another state-of-the-art benchmark suite.
   * Note that this work mostly beats the state of the art in MOVi-e and the other benchmark suite (comparing to NeurIPS 2022 SAVi++ [1] paper and relating to ICLR 2023 DINOSAUR paper [2]).

Thus, while not in this paper, but it has been demonstrated that this method generalizes. 

This paper is about learning object permanence and learning to adaptively fuse inner and outer information - a challenge that has been around and has been tackled for decades cf. [3]. We solve this challenge for the first time without the provision of any mask/object information whatsoever.

We would highly appreciate if the reviewer finds the time to consider these critical aspects.

[1] Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos, December 2022. URL http://arxiv.org/abs/2206.07764. arXiv:2206.07764

[2] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the Gap to Real-World Object-Centric Learning, March 2023. URL http: //arxiv.org/abs/2209.14860. arXiv:2209.14860.

[3] Yuko Munakata, James Mcclelland, Mark Johnson, and Robert Siegler. Rethinking infant knowl- edge: Toward an adaptive process account of successes and failures in object permanence tasks. Psychological Review, 104(4):686–713, 1997. doi: 10.1037/0033-295x.104.4.686.

Reviewer comment (Evaluation):
In Figure 4, t=42, why does the green object cease to exist in the imagination of loci-looped?

Author response:
Thank you for acknowledging that Loci-Looped exhibits strong object tracking performance. Please see the general response as response to most of your comments. 

Indeed, in Figure 4 the model narrows down the hidden object as it approaches the edge of the occluder. This interesting behavior is fully emergent as the model never encountered vanishing objects in the training dataset. The model does not completely let the object vanish, which becomes apparent by the surprise signal.

In general, this inner loop should be applicable to other compositional scene representation learning methods if they use a latent prediction module. For example this is the case in SAVi, where a percept gate could be incorporated after the Correction module. In contrast, this would not work for Steve as this model does not predict the next object states. However, we hypothesize that the success of the inner loop in Loci-Looped also depends on its inductive bias to maintain stable object representations which is enforced by the GateLORD architecture as well as by the regularization terms.

Reviewer comment (Experiments):
Can this inner loop mechanism be leveraged in other compositional scene representation learning methods? Demonstrating the effectiveness of this mechanism on other models will also make the paper stronger.

Author response:
Thank you for acknowledging that Loci-Looped exhibits strong object tracking performance. Please see the general response as response to most of your comments. 

Indeed, in Figure 4 the model narrows down the hidden object as it approaches the edge of the occluder. This interesting behavior is fully emergent as the model never encountered vanishing objects in the training dataset. The model does not completely let the object vanish, which becomes apparent by the surprise signal.

In general, this inner loop should be applicable to other compositional scene representation learning methods if they use a latent prediction module. For example this is the case in SAVi, where a percept gate could be incorporated after the Correction module. In contrast, this would not work for Steve as this model does not predict the next object states. However, we hypothesize that the success of the inner loop in Loci-Looped also depends on its inductive bias to maintain stable object representations which is enforced by the GateLORD architecture as well as by the regularization terms.

Reviewer comment (Presentation):
**W1 - A critical aspect of the result - visualization of slot attention decomposition is missing**. The result section does not contain any illustrations of slot decomposition and roll-out results across time. Please see **Figure 6** in [1] and **Figure 5** in [2]. It is critical to visualize slot decompositions, especially given how strongly authors are attempting to make the "object permanence" claim.

Author response:
Thank you for the helpful comment, pointing out that the slot decomposition is missing in the illustrations. We now provide illustrations of the slot decomposition in the appendix. We hope this makes clear, that the objects persist in the slot representation when occluded. Moreover, we now provide videos in the supplementary material.

Reviewer comment (Experiments):
How important is each of the regularizers proposed in **Table 1**? Many changes are made going from Loci-v1 to Loci-looped.

Author response:
Concerning the importance of the different regularization terms: The Gestalt Change, the Position Change and Object Permanence Regularization are ablated in Traub et al. (2023), suggesting that the Gestalt Change and the Position Change regularization is important for learning. We are currently running ablation studies on the Input-Frame Reconstruction Loss and the Gate Opening Regularization and will add them to the Appendix.

Reviewer comment (Experiments):
**W5 - What are the limitations of this work? There does not seem to be any discussion.** For example, how would the system perform if non-rigid materials were present in the scene? Suppose a cup of water is being poured into a basket that is occluded by a board, a similar setup to your current experiments. Could the system still infer that the water is permanent across time?

Author response:
Indeed non-rigid materials present a challenge to the current model (and very likely to most others). We are happy to include this aspect as a limitation in the discussion section among other limitations.

Reviewer comment (Experiments):
I believe that the ability of correctly segmenting objects in the newly proposed framework largely comes from the information bottleneck. Although it works well with the chosen datasets which have almost pure colors in each object, I doubt it will work well in environments with more complex texture as in nature or in datasets such as MOVI-c and beyond (https://github.com/google-research/kubric/tree/main/challenges/movi). Although I guess segmentation is not treated as a major contribution, I worry about how much the proposed principle can be scaled up and generalize.

Author response:
We also recognize the necessity of testing the model on more complex occlusion scenarios, such as non-linear object movements or containment situations. This limitation has been now duly addressed in our discussion, emphasizing the need for datasets that pose greater challenges in terms of segmentation, localization, and tracking to further evaluate and enhance the model's abilities.

Reviewer comment (Experiments):
If we are restricted to a simple environment, one can imagine that another model that simply clusters pixels based on colors and estimate the center of mass of the clustered pixels could likely segment and localize objects correctly in the CRATER dataset, without using a neural network. Then an RNN that learns to predict the center of mass based on the previous trajectory extracted with the above approach and weights its loss function based on the number of pixels in the correct color corresponding to that object can also shut down gradient when an object is occluded, and then receive teaching signal once the object reappears. This way, the RNN may also be able to learn to predict a linear moving trajectory. Now perhaps what I describe here essentially is similar to the idea of perceptual gate and perhaps the advantage of the current model is that the gate is learned rather than being hard coded as being decided by pixel counts, what I am trying to say is that the environment may not pose enough challenge for the task that the model aims to solve (all of segmentation, localization and tracking). If an environment allows defining the gate by a pre-defined rule, then learning it seems trivial. I am not against using such dataset for proof of principle. But I think this limitation should not be ignored.

Author response:
We appreciate the suggestion for a rule-based baseline. We have added a Loci-Visibility baseline, whose percept gate is controlled solely based on the perceived occlusion state i.e. Loci switches to the inner-loop when objects become occluded. We find that it performs superior to Loci-Unlooped highlighting the importance of the inner loop. However, it performs inferior to Loci-Looped, demonstrating the importance of learning an adaptive gate control function that flexbily balances the inner and the outer looop, rather than approximating a simple rule.

Reviewer comment (Evaluation):
The slot error is claimed to serve as a surprise signal. But some details of its pattern appears strange to me. In Figure 3, after frame 30, the slot error is of similar magnitude for both the reappearing and vanishing object. If as the paper tries to claim that the model learns to imagine objects behind the plate during occlusion, then I would expect that it should predict the appearance and location of the object more or less correctly when the object should reappear. In other words, the slot error should be smaller in the reappearing case than in the vanishing case (which violates object permanency and should not be predictable at all). An indifference here seems to indicate either the prediction of occluded object is quite wrong upon reappearing or that the model somehow learns to predict vanishing object somewhat correctly?

Author response:
Thank you for pointing out that Figure 3 was a bit confusing in that respect. Due to averaging effects the greater surprise in the vanishing trials was not explicitly illustrated in the plot. We have added an additional box plot (Figure 3a) showing that the maximum slot errror is significantly larger when hidden objects fail to reappear after occlusion and after the occluder rotates to the ground, compared to trials in which objects do reappear. We also now add a video to the supplementary material, which shows that our system expects the reappearance and then "parks" the object behing the occluder until the occluder falls over. Please note that this latter behavior is fully emergent, as our system was not trained on vanishing trials. 

Thank you for pointing that the correlation needs more explanation. The concept here revolves around the similar trajectories of objects reappearing and vanishing. The visibility spike of reappearing objects (observed after frame 30 in Figure 3) signifies an expected moment when both vanishing and reappearing objects should become visible again. Therefore, the visibility of reappearing objects serves as an indicator for the visibility of vanishing objects. Our observation reveals that both the slot error and the visibility mask for vanished objects rise within this specific time interval. This suggests that the model anticipates the vanishing objects to become visible again during this interval, effectively predicting their reappearance. We've revised the paragraph to convey this argument more clearly and moved it to the appendix.