Paper: Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR-2024)

Reviewer comment (Experiments):
The DINO baseline is trained for 100 epochs only which is not the default setting. Additionally, DINO paper reports a performance of 61.8 with a VIT.S/16 on Davis while the paper reports of 54.6 for the same method. I would encourage the authors to report what are the performances of the DINO released models as those models are available.

Author response:
**2. DAVIS numbers**  

DINO (Caron et al.) reports a performance of 61.8 on DAVIS using ViT-S/16 when pretrained on *ImNet* for 300 epochs. In our work, we report that DINO achieves a performance of 54.6 on DAVIS when pretrained on *WT-Venice* for 100 epochs. The difference in performance is from the different pretraining dataset and higher number of pretraining epochs.

In Section C under ``Longer pretraining" and Table 7 in the Appendix, we have added the comparison of DoRA vs DINO when pretrained for 300 epochs on WT-Venice and WT-all.

Reviewer comment (Novelty):
DORA shares some similarity with the VITTO. Both approaches learn from video and use an 'unsupervised' pooling mechanism. However, DORA seems to underperform VITTO on the ADE20K and MS-COCO tasks.

Author response:
**3. DoRA vs VITO**  

VITO is pretrained for 300 epochs using Resnet-50 on VideoNet: a curated dataset of 1 million videos whose distribution is similar to that of ImNet. While DoRA is pretrained for 100 epochs using ViT-S/16 on 1 or 10 long uncurated WTour videos whose distribution is different from ImNet.

Reviewer comment (Experiments):
Missing comparison with more recent baselines. It would be nice to add comparison with DINOv2 and a weakly-supervised baseline OpenCLIP, which are both state-of-art methods.

Author response:
**4. Comparison with Dino-V2 and OpenCLIP**

Thanks for this suggestion. We will add experiments with DINO-v2 and OpenCLIP in the camera ready version

Reviewer comment (Experiments):
Comparing with state-of-art image-baseline such as DINOv2 and CLIP on the different tasks would really highlight the importance of video pretraining.

Author response:
**4. Comparison with Dino-V2 and OpenCLIP**

Thanks for this suggestion. We will add experiments with DINO-v2 and OpenCLIP in the camera ready version

Reviewer comment (Novelty):
It would be useful to discuss in depth the relation with the VITTO approach.

Author response:
**3. DoRA vs VITO**  

VITO is pretrained for 300 epochs using Resnet-50 on VideoNet: a curated dataset of 1 million videos whose distribution is similar to that of ImNet. While DoRA is pretrained for 100 epochs using ViT-S/16 on 1 or 10 long uncurated WTour videos whose distribution is different from ImNet.

Reviewer comment (Experiments):
One minor issue I have about the presentation is the introduction of the tracking module. The tracker is not learned, and it is only used to provide object locations. I would like further discussion on the potential use of the corresponding information. Also, a comparison on the effect of using different types of unsupervised trackers would also help strengthen the work as the major idea seems to be not dependent on a certain type of tracker.

Author response:
**1. Using unsupervised trackers**. 

Thanks for this interesting suggestion. We shall integrate UnSupTrack [1*] with DoRA. We shall first detect the objects using our proposed multi-object masking method, which we then use as input to UnSupTrack. We shall add this result in the camera-ready version.

[1*] Karthik *et al.*, Simple Unsupervised Multi-Object Tracking, ECCV 2020

Reviewer comment (Experiments):
In Eq. (8) the learning is done on a single frame. Because the tracking has already provided corresponding between locations across multiple frames, is there a certain consideration to not use views from multiple frames in this loss function?

Author response:
**2. Views from multiple frames**. 

Thanks for the interesting question. We will apply the loss function between the global crop of the teacher network (from the reference frame $t_0$ that is used in tracking) to multi-object crops of the student network from all other frames $t$ in the mini-batch. Due to time and compute constraints, we will add this experiment to the camera-ready version.

Reviewer comment (Evaluation):
The paper mainly talks about how we can learn discriminative representations from egocentric videos, whereas what kind of egocentric videos are suitable for DoRA is not deeply discussed. It would contribute more to the community if we knew what properties a video should have to be worth learning.

Author response:
**1. Type of egocentric videos**. 

In Table 2(a), we show that the performance of DoRA is not specific to egocentric videos, *e.g.*, WTours and Epic-Kitchens (EK), but also achieves good performance on diverse pretraining datasets like Kinetics-400 (K-400) and Movie videos (Movie$_{\text{rom}}$). Thus, we observe that DoRA is agnostic to type of video pretraining dataset. Our argument for videos like WTours is that they can collected or filmed very easily.

Reviewer comment (Writing):
Some minor writing problems. (1) in Sec. 4 “Discovering objects with multi-head attention”, $\widetilde{Q}$, $\widetilde{K_t}$ are only defined in Fig.3 and are not defined in text. (2) In Fig.3 (Left), the input should be $X_t^{o_i}$.

Author response:
**5. Minor corrections in Figure 3**

Thanks for pointing this out. We have corrected it in the revised version. 
We define $\tilde{Q} \in \mathbb{R}^{n \times d}$ in the sentence above eq(2) and $\tilde{K} \in \mathbb{R}^{n \times d}$ in the sentence below eq (3)

Reviewer comment (Experiments):
**The potential privacy and safety issues of the dataset.** Also see *Details Of Ethics Concerns*. As the paper claims the dataset as a main contribution, I would expect more efforts in assessing the privacy and safety issues in the dataset (e.g., How many clear faces are detected and what are their resolutions? How many harmful scenes are detected to the best effort of the authors?) and clarifying the legal issues and usage restrictions of the dataset (e.g., Is it possible that some videos are taken down upon the request from people appearing in them? Is their usage in some jurisdictions not allowed / limited to non-commercial only? What are some possible negative effects if the models remember the private information in it? What are the possible effects of some common mitigations, like blurring the faces in Google street view?)

Author response:
The reviewer makes a very good point. To address this concern, we use Deface (https://github.com/ORB-HD/deface) to automatically detect and blur faces in WT videos. Using these modified WT videos, we apply DoRA on WT-Venice. We shall report the results once pretraining is completed.

Reviewer comment (Reproducibility):
In Appendix D, are there any other differences between DoRA without tracking and DINO other than the crop generation method?

Author response:
In Appendix D, we apply equation (6) on a single image rather than a set of frames, i.e., there is no tracking involved when DoRA is pretrained on ImNet. Thus, other than the crop generation method, there is no other difference between DoRA* (DoRA without tracking) and DINO.

Reviewer comment (Experiments):
What is the stability like when using SK, aside from the entropy regularisation is some annealing schedule needed that transforms the coupling matrix from soft to hard gradually?

Author response:
**2. Stability of SK**  

SK is an iterative optimization algorithm that indeed converges, with its cost function monotonically decreasing with the number of iterations.

We understand that the reviewer asks about training stability when using SK---if not, please clarify. We empirically find that, when using SK with the features from the last layer of the transformer to compute refined object prototypes $P'$, training is indeed stable.

Using annealing to transform the coupling matrix from soft to hard might be needed if we target hard assignment between object prototypes and patch features. Hard assignment can also be achieved by using a smaller value of $\epsilon$ (coefficient of entropy regularizer), which improves one-to-one matching, although it makes optimization harder. We evaluate the performance of DoRA with a smaller value of $\epsilon$ and increasing the number of iterations to 60 (default iterations is 30), observing sub-optimal results on downstream tasks. This indicates that downstream task performance does not benefit from hard assignment.

Reviewer comment (Experiments):
Is there any intuition why the k-attention maps obtained by projecting the attention map are spatially overlapping? Is there any possibility to use a simple heuristic to avoid it that can be ablated with SK?

Author response:
**3. Overlapping attention maps.**  

The overlap of the attention maps obtained from different heads is commonly observed as there is no supervisory signal during training to ensure diverse attention from different heads. For example, see Fig. 10 of DINO (Appendix), where the authors show overlap for the different heads in the last layer.  

In our previous experiments, we used a simple heuristic inspired from CutLer [2*]. This is an unsupervised object discovery method that iteratively uses normalized cuts on patch affinity matrix to find foreground objects. Similarly, we iteratively removed attended regions to find more, non-overlapping ones. However, on WT videos, we observed that this heuristic did not achieve consistent improvements on all downstream tasks. In particular, its results were sub-optimal on linear probing on Imagenet and unsupervised object discovery on Pascal VOC.

[2*] Wang *et al.*, Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023.