Paper: V-JEPA: Latent Video Prediction for Visual Representation Learning (ICLR-2024)

Reviewer comment (Novelty):
Using an existing image architecture to process videos by flattening the 3D feature maps into 1D does not qualify as an architectural novelty.

Author response:
1) **Novelty:**  We would like to clarify that we don’t claim novelty in terms of model architecture. Rather, this is the first paper to demonstrate that the joint-embedding predictive architecture, which is a learning principle, can be used to learn high-quality representations from video.
    * V-JEPA is the first approach to learn from video without either pixel reconstruction or an invariance-based learning criteria (contrastive) and without using any text. Previous works exploring this class of learning algorithms (I-JEPA, data2vec, etc.) focused only on Image, speech or text modalities.
    * To demonstrate the value and impact of this exploration, we evaluate our approach using a ‘frozen evaluation’ protocol as we are interested in video models that can be applied off-the-shelf to many downstream tasks, similarly to models such as DINOv2 or OpenCLIP in the image domain.
    * In a frozen evaluation setting, V-JEPA achieves (1) the best performance across video models on 6 video and image tasks and (2) outperforms large-scale image-based pretrained models on tasks requiring motion understanding. V-JEPA therefore produces state-of-art video embedding that can be applied off-the-shelf to many video and image tasks without any finetuning.
    * We agree that developing architectures and methods that further take advantage of the video specificities, such as long duration, is a very important direction for research which is complementary to this work. However, the first step to enable this direction was to demonstrate the suitability of JEPA learning for video, which we have accomplished in this paper.

Reviewer comment (Novelty):
Videos have different challenges than images, which could have been tackled as novelties, such as how to cover larger temporal context (>2 seconds) or how to capture long-term dependencies (e.g., H-JEPA?).

Author response:
1) **Novelty:**  We would like to clarify that we don’t claim novelty in terms of model architecture. Rather, this is the first paper to demonstrate that the joint-embedding predictive architecture, which is a learning principle, can be used to learn high-quality representations from video.
    * V-JEPA is the first approach to learn from video without either pixel reconstruction or an invariance-based learning criteria (contrastive) and without using any text. Previous works exploring this class of learning algorithms (I-JEPA, data2vec, etc.) focused only on Image, speech or text modalities.
    * To demonstrate the value and impact of this exploration, we evaluate our approach using a ‘frozen evaluation’ protocol as we are interested in video models that can be applied off-the-shelf to many downstream tasks, similarly to models such as DINOv2 or OpenCLIP in the image domain.
    * In a frozen evaluation setting, V-JEPA achieves (1) the best performance across video models on 6 video and image tasks and (2) outperforms large-scale image-based pretrained models on tasks requiring motion understanding. V-JEPA therefore produces state-of-art video embedding that can be applied off-the-shelf to many video and image tasks without any finetuning.
    * We agree that developing architectures and methods that further take advantage of the video specificities, such as long duration, is a very important direction for research which is complementary to this work. However, the first step to enable this direction was to demonstrate the suitability of JEPA learning for video, which we have accomplished in this paper.

Reviewer comment (Novelty):
For a paper that uses an existing architecture (JEPA), it is expected to see proposals on how to solve challenges of the video domain.

Author response:
1) **Novelty:**  We would like to clarify that we don’t claim novelty in terms of model architecture. Rather, this is the first paper to demonstrate that the joint-embedding predictive architecture, which is a learning principle, can be used to learn high-quality representations from video.
    * V-JEPA is the first approach to learn from video without either pixel reconstruction or an invariance-based learning criteria (contrastive) and without using any text. Previous works exploring this class of learning algorithms (I-JEPA, data2vec, etc.) focused only on Image, speech or text modalities.
    * To demonstrate the value and impact of this exploration, we evaluate our approach using a ‘frozen evaluation’ protocol as we are interested in video models that can be applied off-the-shelf to many downstream tasks, similarly to models such as DINOv2 or OpenCLIP in the image domain.
    * In a frozen evaluation setting, V-JEPA achieves (1) the best performance across video models on 6 video and image tasks and (2) outperforms large-scale image-based pretrained models on tasks requiring motion understanding. V-JEPA therefore produces state-of-art video embedding that can be applied off-the-shelf to many video and image tasks without any finetuning.
    * We agree that developing architectures and methods that further take advantage of the video specificities, such as long duration, is a very important direction for research which is complementary to this work. However, the first step to enable this direction was to demonstrate the suitability of JEPA learning for video, which we have accomplished in this paper.

Reviewer comment (Experiments):
It seems that the experimental protocol is flawed. Neither Table 1 nor Table 2 show the results of the model when trained on the same datasets. This model is trained on more data than all the other models, making a fair comparison impossible. A common approach would be to show the new model results when trained on the same datasets as previous works, then provide the results with the scaled-up dataset (as a plus). It is hard for the reader to validate the model’s performance if more than one thing changes between the rows of a comparison table.

Author response:
2)  **Evaluation:** Thank you for your feedback. To address your concern regarding the evaluation, we pretrained a V-JEPA model with a ViT-L/16 architecture on the K400 dataset only. This experiment uses the same architecture and dataset as the VideoMAE approach allowing for a direct comparison. V-JEPA achieves scores of 78.7, 61.7, and 25.0 on K400, SSv2, and AvA. V-JEPA has a consistent advantage over VideoMAEv1 with an improvement of +0.7, +0.5, and +3.4 points on K400, SSv2, and AvA, respectively.

Reviewer comment (Evaluation):
Looking through the ablations, the only result I can use for comparison is [Table 3 (b) row 1], [Table 1 row VideoMAE], and [Table 2 row VideoMAE]. All three results use the ViT-L/16 model and pretrain on the same K400 data. Results show that on the Video task for SSv2, VideoMAE outperforms the proposed method by (61.2-50.8=10.4), and on the Image task for IN1k, VideoMAE outperforms V-JEPA by (71.9-66.1=5.8). These results do not show that V-JEPA is SOTA when trained fairly on the same datasets as others.

Author response:
3) **Table 3 results:** We apologize for the confusion in the Table 3 ablation experiment. We found a mistake in the reporting of this ablation, which is now corrected. V-JEPA pretrained on K400 achieves a score of 61.7 on SSv2 and 70.4 on ImageNet. While VideoMAE v1 achieves 71.9 on ImageNet (+1.5% compared to V-JEPA pretrained on K400), the V-JEPA K400 model outperforms the identical VideoMAE K400 model on all video tasks.

    Furthermore, V- JEPA ViT-L/16 demonstrates a favorable scaling behavior; by pretraining on VideoMix2M (comprising 2M videos), the performance on ImageNet is boosted by +3.3% top-1. We note that we do not observe a performance improvement when evaluating a VideoMAEv2 model pretrained on the larger UnlabeledHybrid dataset (comprising 1.35M videos).

Reviewer comment (Experiments):
In Paragraph "Comparing with image models, 2nd paragraph": Is it possible that attentive probing improves the performance over linear probing because of the attention mechanism? How does it do with just linear probing?

Author response:
5)  **Linear vs attentive probing:** We perform an extensive analysis on the probing strategies in Appendix C. We found that attentive probing improved the results compared to linear probing for both V-JEPA and the baselines. Using our evaluation protocol, we achieve new state-of-art performance for frozen evaluation using a DINOv2 backbone on three of our evaluation tasks: K400, INaturalist and Places-205 (see Table 9 in Appendix B). Our evaluation protocol also strongly improves the performance of OpenCLIP and VideoMAE models as Tables 8 and 9 show. We therefore choose attentive probing as our default evaluation as it leads to more competitive results among frozen evaluations for all baselines and V-JEPA.

Reviewer comment (Experiments):
Have you considered a multiscale representation, where you concatenate 16 frames with small strides at 224x224 with 16 frames at double the stride at 128x128, and so on?

Author response:
6) **Multiscale representation:** We agree that exploring multiscale representations with various timescales is a very interesting and promising research direction. We did not explore such an approach in the current work, but plan to do so in future work.

Reviewer comment (Experiments):
The results shown in the abstract and Figure 1 are not quite fair actually, since the model sizes are not the same.

Author response:
Thank you for your suggestion. We agree that both the VideoMix2M dataset and proposed V-JEPA methodology contribute to the V-JEPA results shown in Table 1. To disentangle some differences from previous work, we ran an additional experiment where we pretrain a ViT-L/16 on only the K400 dataset (comprising 240K videos) using the same pretraining hyperparameters described in Section 4.1. We compare this model to a VideoMAEv1 ViT-L/16 which was also pretrained only on K400. V-JEPA achieves scores of 78.7, 61.7, and 25.0 on K400, SSv2, and AvA. V-JEPA has a consistent advantage over VideoMAEv1 with an improvement of +0.7, +0.5, and +3.4 points on K400, SSv2, and AvA, respectively.

    Furthermore, our V-JEPA approach demonstrates favorable scaling behavior: by pretraining on VideoMix2M (comprising 2M videos), the performance is boosted by +0.5, +5.3, and +0.6 points, respectively. We note that we do not observe a similar improvement when evaluating the VideoMAEv2 model, which was pre-trained on a dataset of 1.35M videos (including a private subset, preventing us from training V-JEPA models in the same setting).

Reviewer comment (Experiments):
The pretraining data is VideoMix2M, which is different from others. It is a bit doubtful whether the improvement comes from the self-supervised method.

Author response:
Thank you for your suggestion. We agree that both the VideoMix2M dataset and proposed V-JEPA methodology contribute to the V-JEPA results shown in Table 1. To disentangle some differences from previous work, we ran an additional experiment where we pretrain a ViT-L/16 on only the K400 dataset (comprising 240K videos) using the same pretraining hyperparameters described in Section 4.1. We compare this model to a VideoMAEv1 ViT-L/16 which was also pretrained only on K400. V-JEPA achieves scores of 78.7, 61.7, and 25.0 on K400, SSv2, and AvA. V-JEPA has a consistent advantage over VideoMAEv1 with an improvement of +0.7, +0.5, and +3.4 points on K400, SSv2, and AvA, respectively.

    Furthermore, our V-JEPA approach demonstrates favorable scaling behavior: by pretraining on VideoMix2M (comprising 2M videos), the performance is boosted by +0.5, +5.3, and +0.6 points, respectively. We note that we do not observe a similar improvement when evaluating the VideoMAEv2 model, which was pre-trained on a dataset of 1.35M videos (including a private subset, preventing us from training V-JEPA models in the same setting).

Reviewer comment (Writing):
The paper writing should be improved. The methodology part is quite hard to read. The high-level idea (sect. 3.1) is okayish to understand, but the application to video part (sect. 3.2) is not. Concretely, as the input has already been processed by the 3D ConvNet as a 1D feature, I don't know how to obtain the 3D sin-cos positional embedding (mentioned in Predictor section).

Author response:
1) **3D positional embedding:** Thank you for your feedback. We added a new figure (Figure 5 in appendix) and have reworked the methodology section to clarify this point. Specifically, to convert a video clip of size 16x224x224x3 into a 1D token sequence, we apply a 3D convolution comprising d filters of size 2x16x16 with a temporal stride of 2 and a spatial stride of 16, resulting in a tensor of shape 8x14x14xd. Next, we add absolute 3D sin-cos positional embeddings to the spatio-temporal feature map and flatten it, resulting in a 1D token sequence of shape 1568xd.

Reviewer comment (Reproducibility):
After reading the paper, I still don't know how to obtain the self-supervised video features. Is it the output of the target encoder or the output of the context encoder?

Author response:
2) **SSL feature:** The self-supervised learning features are the output of the target encoder, which is then fed to an attentive or linear probe. Please refer to appendix B.1 for more details

Reviewer comment (Experiments):
For Table 1, it is known that joint training on SSv2 and Kinetics will improve both dataset's performance as mentioned in (Zhang, B., Yu, J., Fifty, C., Han, W., Dai, A. M., Pang, R., & Sha, F. (2021). Co-training transformer with videos and images improves action recognition. arXiv preprint arXiv:2112.07175). For a fair comparison with VideoMAE, the proposed approach should be pretrained on the Kinetics400 dataset.

Author response:
3) **Comparison on K400:** Thank you for your suggestion. We added an experiment where we pretrain a ViT-L/16 using V-JEPA on only the K400 dataset (comprising 240K videos) using the same pretraining hyperparameters described in Section 4.1.  We compare this model to a VideoMAE ViT-L/16 also pretrained only on K400. V-JEPA achieves scores of 78.7, 61.7 and 25.0 on K400, SSv2 and AvA. V-JEPA has a consistent advantage over VideoMAE with an improvement of +0.7, +0.5 and +3.4 points on K400, SSv2 and AvA. 
 
    Furthermore, V-JEPA ViT-L/16 demonstrates a favorable scaling behavior; by pretraining on VideoMix2M (comprising 2M videos), the performance is boosted by +0.5, +5.3 and +0.6 points, respectively. Both the VideoMix-2M dataset and the V-JEPA approach therefore contribute to the results. We note that we do not observe a similar performance improvement when evaluating a VideoMAEv2 model that was pretrained on the UnlabeledHybrid dataset (comprising 1.35M videos, including a private subset).

Reviewer comment (Evaluation):
Why could the model achieve much better performance than the VideoMAE? Is it because of the mixture video datasets for pretraining (if so, the whole paper might not be that exciting)? If predicting the embedding is better than predicting the patch, could you provide a conceptual reasoning for that?

Author response:
4) **Importance of latent-space prediction:** By predicting in latent space, our model can potentially eliminate unnecessary pixel-level details when encoding a video with the target encoder. As a result, the model has the flexibility to focus only on predictive features from the videos and discard irrelevant information from the target representation. In contrast, a reconstruction based approach needs to model every input pixel correctly to satisfy its loss.

Reviewer comment (Experiments):
Before reading Table 1 carefully, I thought this paper was quite good. However, after reading Table 1 and comparing VideoMAE and V-JEPA on the ViT-L/16, I am afraid the main performance improvement is not brought by the proposed approach, but by the mixture of video datasets for pretraining.

Author response:
3) **Comparison on K400:** Thank you for your suggestion. We added an experiment where we pretrain a ViT-L/16 using V-JEPA on only the K400 dataset (comprising 240K videos) using the same pretraining hyperparameters described in Section 4.1.  We compare this model to a VideoMAE ViT-L/16 also pretrained only on K400. V-JEPA achieves scores of 78.7, 61.7 and 25.0 on K400, SSv2 and AvA. V-JEPA has a consistent advantage over VideoMAE with an improvement of +0.7, +0.5 and +3.4 points on K400, SSv2 and AvA. 
 
    Furthermore, V-JEPA ViT-L/16 demonstrates a favorable scaling behavior; by pretraining on VideoMix2M (comprising 2M videos), the performance is boosted by +0.5, +5.3 and +0.6 points, respectively. Both the VideoMix-2M dataset and the V-JEPA approach therefore contribute to the results. We note that we do not observe a similar performance improvement when evaluating a VideoMAEv2 model that was pretrained on the UnlabeledHybrid dataset (comprising 1.35M videos, including a private subset).

Reviewer comment (Novelty):
The method proposed in the paper is an extension of I-JEPA to video. The approach used to implement masking across the temporal dimension (``3D-multiblock'') and latent space prediction borrow heavily from I-JEPA. The novelty of the method introduced here is very limited for the reason mentioned above.

Author response:
1) **Novelty:**  V-JEPA is the first paper to demonstrate that self-supervised learning with a joint-embedding predictive architecture works for video. Previous works have only explored joint-embedding predictive architectures with images, text, or speech modalities.

Reviewer comment (Experiments):
A key component that determines the success of VJPEA is multiblock design. I would have liked to have seen careful ablation of the design choices of this block including the scale factors and also aspect ratios used to design the mask.

Author response:
2) **Masking Ablation:** Thank you for your suggestion. We added a new section in Appendix C.4 that ablates different choices regarding the masking strategy. The main findings of this ablation are 1) the importance of high-masking ratio (as highlighted in previous work) 2) masking several blocks performs better than masking one large block, given a fixed masking ratio (recall that each mask is constructed by sampling several (possibly overlapping) blocks and taking their union). 3) sampling several masks with different block sizes for each clip further improves the performance.

Reviewer comment (Evaluation):
The method uses an evaluation scheme (attentive probing) which may be the way to probe representations in foundation models. The biggest problem I see with this scheme is that it may make it hard for readers to compare performance across publications.

Author response:
3)  **Attentive probe comparison with previous work:** For the initial submission we ran the attentive probe protocol with open-source baseline models, and also reported attentive probing results from the VideoGLUE paper, which provides an extensive evaluation of previous video models under frozen evaluation. Previous work VideoMAE v1 reports frozen evaluation with a linear probe on SSv2 (38.9% top-1). We significantly improve upon their reported numbers by evaluating their model using an attentive probe (61.2% top-1 with ViT-L/16). To ease comparison with previous work, we also report fine-tuning evaluations, which we moved from the appendix to the main paper in the revision.

Reviewer comment (Novelty):
Potential missing background/related work: the paper "Siamese Masked Encoders" (https://arxiv.org/abs/2305.1434).

Author response:
3)  **Siamese Masked Encoder:** We have cited Siamese Masked Encoders and discussed this interesting work in the main paper. SME uses a frame-level encoder, and trains a decoder to reconstruct missing pixels in future frames. By contrast, V-JEPA uses a video encoder and does not require reconstructing pixels, but predicts in latent space. It would  be interesting to combine the cross-attentive architecture of Siamese Masked Autoencoder with a JEPA loss in future work!

Reviewer comment (Experiments):
Did you conduct a full ablation of all the design hyperparameters used in multiblock. If so, please include this ablation in the appendix as done in I-JEPA for instance.

Author response:
2) **Masking Ablation:** Thank you for your suggestion. We added a new section in Appendix C.4 that ablates different choices regarding the masking strategy. The main findings of this ablation are 1) the importance of high-masking ratio (as highlighted in previous work) 2) masking several blocks performs better than masking one large block, given a fixed masking ratio (recall that each mask is constructed by sampling several (possibly overlapping) blocks and taking their union). 3) sampling several masks with different block sizes for each clip further improves the performance.

Reviewer comment (Experiments):
Table 3 ablation - is randomly picking images from a video sequence the best approach to create a training dataset for image-based backbones. Also, the backbones used for VJEPA and I-JEPA in Table 3(a) are different. If results are available for the same backbone please consider including those to help a reader understand the design choices better.

Author response:
4)  **Ablation Table 3:** In this ablation, V-JEPA and I-JEPA see the same exact training data (videos) in the same exact sequence. However, because I-JEPA can only process individual frames, we can only compute the I-JEPA loss on individual frames. We did not have enough time to train a smaller I-JEPA backbone on K400, but we are planning to include this additional ablation and finer-grained comparison in the paper.

Reviewer comment (Novelty):
As an extension of I-JEPA to videos, technical novelty is extremely limited.

Author response:
1) **Novelty:**  V-JEPA is the first model to learn from video using a Joint-Embedding Predictive Architecture. As you have pointed out, previous works exploring a similar class of algorithms (I-JEPA, data2vec, etc.) only focused only on image, speech or text modalities.

Reviewer comment (Evaluation):
Evaluations are problematic, by either not comparing to the relevant work or by comparing to prior work in a misleading way, contrary to the evaluation protocols the whole community has been using.

Author response:
2) **Frozen evaluation:** Our motivation is to study the ability to produce video embeddings for off-the-shelf use in downstream tasks, and thus we respectfully disagree that the frozen evaluation protocol is not interesting or misleading. In fact, the frozen evaluation protocol is standard in computer vision, such as in self-supervised learning with images or text. While it is true that the video community has mostly focused on end-to-end fine-tuning, there are many notable works that also explore frozen evaluation of video models/tasks, such as VideoGLUE (Yuan et al., 2023), VITO (Parthasarathy et al., 2022), DINOv2 (Oquab et al., 2023), Unsupervised Video CNNs (Wang & Gupta, 2015), etc. In fact, even the VideoMAE v1 mentioned in the review reports frozen evaluation with a linear probe on SSv2 (38.9% top-1). We significantly improve upon their reported numbers by evaluating their model using an attentive probe (61.2% top-1 with ViT-L/16). We also improve the reported DINOv2 frozen evaluation number on K400 from 74.4 to 84.4 top-1 by using an attentive probe.

    Having said that, we hope to address your concern and have updated the main paper to state explicitly that absolute state-of-art is obtained by models that are fine-tuned on each task individually and report the best performances on video tasks. We are happy to revise our paper further to clarify those points.

Reviewer comment (Evaluation):
Most recent related work has adopted a finetuning evaluation protocol (e.g. VideoMAE, VideoMAEv2, ST-MAE, Hiera, MaskFeat, OmniMAE, MCMAE) and yet such evaluation is almost entirely absent from this work (except for highly limited comparison in Table 11, where V-JEPA performs worse than prior work despite a misleading comparison*), portraying a highly misleading picture of not only the state of the literature but also of the capabilities of the V-JEPA models in comparison. It is well known since even the original MAE work that such models do not have the best performance when frozen, but even finetuning 1 or a small number of layers makes a large difference, demonstrating how frozen evaluation can be quite misleading. Classification aside, on action recognition, the gap between frozen and finetuning can be particularly large, which can again be quite misleading.

Author response:
3) **Finetuning evaluation:** We thank the reviewer for their suggestion, and have added the VideoMAEv1 baseline in our Finetuning Evaluation table. While VideoMAEv1 uses a smaller model than VideoMAE v2 and is trained with far fewer samples, its cost per iteration is higher than VideoMAEv2, since v2 uses a much shallower decoder and aggressive decoder masking. We find V-JEPA to outperform VideoMAEv1 finetuning by +0.1% on K400 and +0.5% on SSv2, while also training for only ~60% of the VideoMAEv1 training time.

Reviewer comment (Experiments):
Notably missing are references/discussion and comparison to some of the stronger recent work (e.g. UnMasked Teacher, Li et al 2023; Hiera, Ryali et al 2023; MCMAE, Gao et al, 2022, etc).

Author response:
4) **Other papers:**  Thank you for pointing out those works, we have added citations to these papers and discussed them in the main text as well. We have also
a) added the suggested evaluation of Hiera in Table 1. A V-JEPA L/16 trained on K400 outperforms a bigger Hiera-H model (also trained on K400) by +5.4% on K400, +1.0% on SSv2, and +7.5% on AVA.
b) We cite and discuss VITO in the main paper. Note that the VITO video model reports a frozen evaluation on ImageNet of 66.2 with a ResNet50 and a linear probe. Their model is not publicly available so we cannot validate this result or rerun the frozen evaluation, but our smallest V-JEPA model, a ViT-L/16 with an attentive probe, achieves a top-1 of 73.7 , which is +7.5% higher.

Reviewer comment (Experiments):
Transfer to images is explored, but comparison to prior work is problematic, e.g. there is no comparison to VITO, Parthasarathy et al, 2023.

Author response:
4) **Other papers:**  Thank you for pointing out those works, we have added citations to these papers and discussed them in the main text as well. We have also
a) added the suggested evaluation of Hiera in Table 1. A V-JEPA L/16 trained on K400 outperforms a bigger Hiera-H model (also trained on K400) by +5.4% on K400, +1.0% on SSv2, and +7.5% on AVA.
b) We cite and discuss VITO in the main paper. Note that the VITO video model reports a frozen evaluation on ImageNet of 66.2 with a ResNet50 and a linear probe. Their model is not publicly available so we cannot validate this result or rerun the frozen evaluation, but our smallest V-JEPA model, a ViT-L/16 with an attentive probe, achieves a top-1 of 73.7 , which is +7.5% higher.