Paper: VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE (ICLR-2024)

Reviewer comment (Presentation):
The current title is too broad, which makes the readers hard to understand the specific contributions of this paper.

Author response:
> The current title is too broad, which makes the readers hard to understand the specific contributions of this paper.

We have updated the title to make it more specific: 

"VONet:  Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE",

to highlight the two key innovations of the paper, as appreciated by most reviewers.

Reviewer comment (Novelty):
The motivation of each contribution of the proposed method is not clearly clarified in Introduction, especially the difference or new insights w.r.t previous methods.

Author response:
> The motivation of each contribution of the proposed method is not clearly clarified in Introduction, especially the difference or new insights w.r.t previous methods.

The second paragraph in the Introduction section talks about the motivation and contributions of VONet. We agree that the current paragraph might be a little short. We've expanded the Introduction section to elaborate the motivation and contributions of this work. Please also see our consolidated response for more clarifications.

Reviewer comment (Presentation):
It is hard to refer to the expression of the symbols in Figure 3.

Author response:
> It is hard to refer to the expression of the symbols in Figure 3.

We've updated Figure 3 by adding a text label besides each symbol, to provide a better context. Also we'd like to clarify that the '*' in each symbol's subscript represents the collection of $K$ slots in parallel (only two slots are plotted in the figure). The updated figure is below:

https://i.imgur.com/UijSdnc.png

Reviewer comment (Experiments):
As already mentioned in the paper when the predefined slot numbers is larger than the actual number of objects, an unwanted side-effect is over-segmentation.

Author response:
Instead of combining multiple slots into one, a possible alternative is to let the attention process learns to disable a certain slot by outputting a very small probability of that slot over *all pixels*. Imagine that in addition to generating a slot probability distribution at every pixel location, the attention network also generates a global gate signal to control the on/off of that slot.  This gate value in $[-\infty,0]$, same over all pixels on an image, will be added to the attention logits of that slot at every pixel. If the gate is very negative, then that slot is basically turned off. 

Besides this architecture, we also need some objective to drive the model to learn such gate values in the way we want. In general, when the number of objects is small in a video, we don't want to waste slots to encode and oversegment an object. If we treat slots as valuable resources, then an objective that penalizes turning on slots will be helpful (e.g, penalizing gate values close to 0). There is inherently a conflict between reconstructing the input image (better if more slots are used) and turning on fewer slots. How to cleverly strike a balance between the two is an interesting challenge.

Reviewer comment (Experiments):
Can the authors provide any insights on how to potentially combine multiple slots to prevent such overfitting?

Author response:
Instead of combining multiple slots into one, a possible alternative is to let the attention process learns to disable a certain slot by outputting a very small probability of that slot over *all pixels*. Imagine that in addition to generating a slot probability distribution at every pixel location, the attention network also generates a global gate signal to control the on/off of that slot.  This gate value in $[-\infty,0]$, same over all pixels on an image, will be added to the attention logits of that slot at every pixel. If the gate is very negative, then that slot is basically turned off. 

Besides this architecture, we also need some objective to drive the model to learn such gate values in the way we want. In general, when the number of objects is small in a video, we don't want to waste slots to encode and oversegment an object. If we treat slots as valuable resources, then an objective that penalizes turning on slots will be helpful (e.g, penalizing gate values close to 0). There is inherently a conflict between reconstructing the input image (better if more slots are used) and turning on fewer slots. How to cleverly strike a balance between the two is an interesting challenge.

Reviewer comment (Experiments):
In order to show the advantages of parallel processing, it would be beneficial that a comprehensive latency/accuracy comparison with MONet could be provided in the single-frame scenario.

Author response:
> In order to show the advantages of parallel processing, it would be beneficial that a comprehensive latency/accuracy comparison with MONet could be provided in the single-frame scenario.

**Latency comparison**. With the same U-Net architecture, we report the average time of generating attention masks for a varying number of slots on an image of size 128x128. With a mini-batch size of 32, we timed the attention inference process for a total of 200 calls on a 3090 GPU, and recorded the average time. The results are summarized in the figure below. VONet demonstrates a strong advantage of being able to maintain a nearly constant inference time regardless of the increasing number of slots. In stark contrast, MONet’s time grows linearly with respect to the slot number, which is expected as MONet generates one attention mask after another. The superior efficiency of VONet on individual frames forms the basis for extending it to video frame sequences.

https://i.imgur.com/5Zt4KEJ.png

**Accuracy comparison**. We compare VONet with MONet on individual video frames from MOVI-{C,D,E}. We modified VONet to train with a sequence length of 1 and excluded training from past slot states. FG-ARI and mIoU are calculated on individual frames, referred to as Image FG-ARI and Image mIoU.

| **Method** | **Image FG-ARI (MOVI-C)** | **Image FG-ARI (MOVI-D)** | **Image FG-ARI (MOVI-E)** | **Image mIoU (MOVI-C)** | **Image mIoU (MOVI-D)** | **Image mIoU (MOVI-E)** |
|------------|---------------------------|---------------------------|---------------------------|-------------------------|-------------------------|-------------------------|
| MONet      | $24.8{\scriptstyle \pm 0.1}$ | $22.0{\scriptstyle \pm 1.7}$ | $18.8{\scriptstyle \pm 0.9}$ | $34.8{\scriptstyle \pm 0.9}$ | $17.4{\scriptstyle \pm 2.0}$ | $13.6{\scriptstyle \pm 1.6}$ |
| VONet  | $47.0{\scriptstyle \pm 2.8}$ | $51.0{\scriptstyle \pm 1.2}$ | $50.3{\scriptstyle \pm 3.6}$ | $36.2{\scriptstyle \pm 3.0}$ | $28.5{\scriptstyle \pm 0.6}$ | $18.1{\scriptstyle \pm 1.3}$ |


The results suggest that VONet is not only more efficient than MONet, but also more competent to discover objects from complex images. We've put this experiment in the Appendix due to page limit.

Reviewer comment (Experiments):
VoNet is a work based on MoNet and is highly similar to ViMON, a comparison of results with MoNet and ViMON should be reported in the experimental phase to demonstrate the advantages of VoNet.

Author response:
Although both inspired by MONet, VONet and ViMON are different in many ways, as stated in Section 2 (last paragraph): 

"*...  several major differences exist between the two, including: i) VONet parallelizes the original MONet architecture for efficient inference while ViMON still generates slots recurrently. ii) VONet seeds the attention network with context vectors while ViMON does this using previous attention masks. iii) VONet formulates an object-wise sequential VAE to promote temporal consistency, whereas ViMON ignores this by using a typical intra-step VAE..*." 

Regarding i), a parallel attention process can greatly speed up the inference, especially when the number of slots increases. Please see our additional experiment results in response to Reviewer 9HJm on the latency differences between MONet and VONet on single frames. On a single GPU, VONet's attention inference time remains constant w.r.t. the number of slots, while MONet's inference time grows linearly. The superior efficiency of VONet in the single-frame scenarios forms the basis for its practical application to video sequences. As for ii), note that when generating attention, our context vector can in theory encode information about the history of a slot all the way back to T=0, while VIMON conditions only on the attention masks of T=t-1 . This property enables VONet to handle occlusions (see our response to the next question). Lastly for iii), our object-wise sequential VAE is a crucial component to ensure temporal consistency as verified by our ablation study (Figure 6), and such a similar formulation is lacked in VIMON. 

As suggested by the reviewer, we have added ViMON as another baseline to our experiments (Section 6.2). We used the official released code of ViMON (https://github.com/ecker-lab/object-centric-representation-benchmark) and made several adjustments to its default hyperparameters to adapt its training to the MOVI datasets. We set the VAE `latent_dim=64`, `n_steps=6` (training video seq length), and updated `n_slots` to either 11 (MOVI-{A,B,C)) or 16 (MOVI-{D,E}) as in other comparison methods. The results are below:

| **Method** | **FG-ARI (MOVI-A)** | **FG-ARI (MOVI-B)** | **FG-ARI (MOVI-C)** | **FG-ARI (MOVI-D)** | **FG-ARI (MOVI-E)** | **mIoU (MOVI-A)** | **mIoU (MOVI-B)** | **mIoU (MOVI-C)** | **mIoU (MOVI-D)** | **mIoU (MOVI-E)** |
 |------------|----------------------|----------------------|----------------------|----------------------|----------------------|---------------------|---------------------|---------------------|---------------------|---------------------| 
 | ViMON | $62.8{\scriptstyle \pm 2.3}$ | $26.2{\scriptstyle \pm 4.4}$ | $18.0{\scriptstyle \pm 2.1}$ | $22.5{\scriptstyle \pm 5.7}$ | $17.7{\scriptstyle \pm 1.5}$ | $50.0{\scriptstyle \pm 1.5}$ | $34.4{\scriptstyle \pm 5.0}$ | $27.1{\scriptstyle \pm 0.9}$ | $19.6{\scriptstyle \pm 1.5}$ | $17.8{\scriptstyle \pm 1.2}$ | 
 | VONet | $\mathbf{91.0{\scriptstyle \pm 1.5}}$ | $\mathbf{60.6{\scriptstyle \pm 0.8}}$ | $\mathbf{45.3{\scriptstyle \pm 0.4}}$ | $\mathbf{50.7{\scriptstyle \pm 1.1}}$ | $\mathbf{56.3{\scriptstyle \pm 0.5}}$ | $\mathbf{60.5{\scriptstyle \pm 2.0}}$ | $\mathbf{59.7{\scriptstyle \pm 2.9}}$ | $\mathbf{42.8{\scriptstyle \pm 0.5}}$ | $\mathbf{37.7{\scriptstyle \pm 0.3}}$ | $\mathbf{36.4{\scriptstyle \pm 0.7}}$ |

We see that ViMON faces challenges in more complex videos (MOVI-{C,D,E}) as the original paper mostly tested it on simple synthetic datasets with only a few objects in a video.

Finally, it is not straightforward to directly compare VONet with MONet as the latter was originally proposed for single-image  scenarios. As suggested by Reviewer 9HJm, we've performed an experiment of comparing VONet with MONet regarding their latency and accuracy on individual video frames of MOVI-{C,D,E}. The conclusion is that  VONet is not only much more efficient in attention mask generation, but also more competent to decompose complex images into objects. Please see our respsonse to Reviewer 9HJm for details.

Reviewer comment (Experiments):
Can the method in this paper correctly handle objects appearing in the middle of the video or reappearing after being occluded and maintain temporal consistency? The MOVI dataset does not seem to be able to model this situation, consider adding experiments in natural scenes.

Author response:
MOVI datasets do model these scenarios. We have rendered several demos below to show that VONet is able to generate temporally consistent object masks under occlusion. 

In each demo, 'fg_seg' is the foreground objects output by VONet and 'gt_seg' is the groundtruth. 

https://i.imgur.com/pAV0bS9.gif
(The gray cylinder was occluded by the big red cube and appeared as the cube moved away. The small purple ball passed by the small skyblue cube in the back, and was occluded by it and then reappeared.)

https://i.imgur.com/BDxBDAN.gif
(The gold cylinder was once completely occluded by the red cube in the middle of the video.)

https://i.imgur.com/QCfEQDt.gif
(The shoe was completely blocked by other objects in the beginning.) 

https://i.imgur.com/mSZbOKW.gif
(The book entered the frame from the top and moved towards the bottom.)

In all these examples, the generated object masks are consistent w.r.t. time (although not in a perfect sense sometimes).

The reason why VONet is able to handle these cases is that, our slot latent, modeled by a GRU, can (in theory) encode information about the history of the slot all the way back to the beginning of the video. When an object appears again in the video frame, the past information contained in the propagated slot latent will make the slot attend to that object.

Reviewer comment (Novelty):
Existing video object-centric learning methods based on slot attention are developing rapidly, e.g. DINOSAUR (https://arxiv.org/abs/2209.14860), VideoSAUR (https://arxiv.org/pdf/2306.04829.pdf), and perform well on natural scene datasets such as YouTube-VIS, please explain the advantages of the method in this paper compared to methods based on slot attention.

Author response:
The paper already includes DINOSAUR and VideoSAUR as references and we are well aware of their existence. DINOSAUR [Seitzer et al. (2023)] is a SOTA method on object discovery from complex real-world images. Technically, it's a single-image method that cannot handle video data. VideoSAUR [Zadaianchuk et al. (2023)] is an extension to DINOSAUR to handle videos, and it was reviewed in our related work section:

*" ... certain previous works,  such as Zadaianchuk et al. (2023), employ a vision backbone pretrained on large-scale datasets for  
feature extraction from video frames, aiding in the process of object learning. While their objective  remains unsupervised, the use of pretrained features integrates valuable prior knowledge of every-day object appearances.  Consequently, the results they reported, though surpassing what typical unsupervised methods including ours could achieve, hold limited reference value in this context.*"

In a word, pre-trained features obtained from self-supervised methods like DINO [Caron et al. 2021] or MAE [He et al. 2022] are heavily relied on for object discovery by both DINOSAUR and VideoSAUR. These pretrained features representations provide a valuable clue for object discovery from video data. While we totally agree that it is a promising direction to incorporate pretrained prior knowledge into video object learning, the focus of our paper is explore the idea of discovering objects from video in a *tabula rasa* manner, namely, learning from scratch without any pre-existing knowledge or biases, or any kind of supervision such as depth, segmentation or optical flow. This allows us to focus on figuring out what kind of model architecture (instead of prior knowledge) more effectively leads to object emergence from video.

On the other hand, DINOSAUR and VideoSAUR employ standard slot attention techniques [Locatello et al., 2020], with their main innovations lying beyond the enhancement of slot attention itself. Their successes are not mainly attributable to slot attention. Slot attention, produced by a transformer, in general faces a dilemma of attention mask "granularity versus continuity" (fine grained object mask vs. spatially coherent object mask), caused by the attention map resolution. We believe that U-Net can provide a solution to this dilemma thanks to the inductive bias of spatial locality in CNNs. Please see our consolidated response for more details on our motivation of parallel U-Net attention.

Finally, our method VONet, has indeed been empirically compared to several slot-attention methods including SAVI and STEVE, and showed great improvements upon them.

Reviewer comment (Writing):
For writing, the introduction section is short. It is hard to understand the main idea of the whole work.

Author response:
We've extended the introduction section to elaborate the motivation and contributions of this work. Please see our consolidated response and updated paper.