Paper: MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features (ICLR-2024)

Reviewer comment (Experiments):
The biggest weaknesses are in the experimental setup and missing comparisons with SoTA.

Author response:
**The biggest weaknesses are in the experimental setup and missing comparisons with SoTA.**

We incorporated a new experiment with a ConvNeXt-B backbone, as well as comparison with iBOT.

Reviewer comment (Experiments):
It is very hard to validate the performance of the proposed model when using a different backbone from the comparison methods. The only valid comparison is VICReg Vs. MC-JEPA because they both use CNX-T.

Author response:
**It is very hard to validate the performance of the proposed model when using a different backbone from the comparison methods. The only valid comparison is VICReg Vs. MC-JEPA because they both use CNX-T.**

Our flow estimation method is incompatible with using a vision transformer. Indeed, the core principle is to first estimate a flow at a very coarse resolution using the deepest features of a convnet, and to progressively refine this flow by doubling the resolution at every layer, using the corresponding features that also double in resolution. The idea between ConvNeXt is to propose an architecture that is comparable in terms of size, number of parameters, throughput and performance, to recent vision transformer architecture. We therefore thought this would give a fair comparison with ViTs of the same size.

Reviewer comment (Experiments):
Some methods, such as iBOT (ICLR22), are not reported. iBOT’s performance on ADE20k with ViT-S is 45.4, which outperforms MC-JEPA. Also if we compare against ViT-B of DINO or IBOT, MC-JEPA is outperformed on many benchmarks. It makes the paper stronger more relevant backbones and methods are included.

Author response:
**Some methods, such as iBOT (ICLR22), are not reported. iBOT’s performance on ADE20k with ViT-S is 45.4, which outperforms MC-JEPA. Also if we compare against ViT-B of DINO or IBOT, MC-JEPA is outperformed on many benchmarks. It makes the paper stronger more relevant backbones and methods are included.**

We will incorporate all our new results, as well as the iBOT performance in the main table.

Reviewer comment (Experiments):
Qualitative results of optical flow only shown against one relatively weaker method (as shown in Table 1). Results from SMURF and UPFlow could also be shown to visualize fail cases and limitations of this model.

Author response:
**Qualitative results of optical flow only shown against one relatively weaker method (as shown in Table 1). Results from SMURF and UPFlow could also be shown to visualize fail cases and limitations of this model.**

As we mentioned in our answer to reviewer arP2, the goal of this paper is to demonstrate the usefulness of learning motion for SSL, which has never been demonstrated yet. Optical flow estimation is simply a pretext task and the goal is not to build a highly specialized flow estimator and produce SOTA flow estimation, but to produce high-quality features with a lot of semantic content. We refer to our answer to reviewer arP2 for a full justification.

Reviewer comment (Experiments):
This paper does not show any results of scaling up the model or training data; all backbones reported in table 4 are lightweight. I imagine that training two tasks like this would require a bigger backbone to handle multitasking. A study showing how performance scales with the model size would be useful.

Author response:
**This paper does not show any results of scaling up the model or training data; all backbones reported in table 4 are lightweight. I imagine that training two tasks like this would require a bigger backbone to handle multitasking. A study showing how performance scales with the model size would be useful.**

We refer to our new result with a CNX-B backbone, which shows that the trend we observed at a lower scale is still valid with scaling.

Reviewer comment (Evaluation):
In section 4.3 (backbone), the authors mention that PWC is not adapted to learn good content features, which explains the low performance in Table 4. However, the method MCRW (in Table 1) uses a PWC backbone and performs much higher than the results of MC-JEPA with a PWC backbone in Table 4. Any reasons behind this significant difference in performance?

Author response:
**In section 4.3 (backbone), the authors mention that PWC is not adapted to learn good content features, which explains the low performance in Table 4. However, the method MCRW (in Table 1) uses a PWC backbone and performs much higher than the results of MC-JEPA with a PWC backbone in Table 4. Any reasons behind this significant difference in performance?**

We investigated this difference of performance and realized that the number reported in Table 4 corresponds to a model only pretrained on Flow and not in our multi-task setup. The new values for ISeg and VSeg in the multi-task setup are 41.8 and 54.7, which is roughly in the same range as the numbers from MCRW. We will update Table 4.

Reviewer comment (Evaluation):
The proposed method performs on par or slightly worse against other flow estimation methods on Sintel benchmark and Kitti, which can not demonstrate the benefit of the proposed ''Joint-Embedding Predictive Architecture''. The authors don't give convincing analysis for this issue.

Author response:
Contrary to other flow estimation methods, which have the objective of producing the best quality flow possible, our goal is to leverage flow estimation as a pretext task in order to learn self-supervised representations from images. We therefore choose to keep the flow estimation method simple, and to only make the necessary adjustments that we describe in our answer to reviewer q59r to make it work in a multi-task setup with regular self-supervised learning of semantic content features. We compare the quality of our produced flow to show that it is of comparable quality than popular unsupervised flow estimation methods, without using a complex list of tricks.

Reviewer comment (Experiments):
Unfair comparison. The proposed method brings more training data compared with the other content methods. It can be seen from Table 2 that the performance in segmentation is likely to degradation as the decrease of motion datasets.

Author response:
The other content methods are only trained on ImageNet, which has no motion information. The goal of this paper is to demonstrate the usefulness of learning motion for SSL, and thus bringing optical flow estimation from video datasets, and this is demonstrated by comparing MC-JEPA to VICReg and showing strong improvement on segmentation tasks. The ablation of Table 2 further shows that regardless of the specific flow estimation dataset, a strong improvement is still observed. Indeed for example on Pascal VOC linear frozen segmentation, VICReg performs 60.1 mIoU. MC-JEPA variants with different flow datasets perform between 66.8 and 67.1 which is a very strong improvement, with a very low variance when the flow dataset changes.

Reviewer comment (Experiments):
Missing analysis. From Table 4, we can see that the model's performance is quite sensitive to the used backbone (more than 10% between Rsenet50 and ConvNext). However, the authors didn't give explanation. Besides, the proposed model uses six loss terms in total for both flow estimation and content learning. I am wondering how to decide the trade-offs and if they would affect the final results.

Author response:
In Table 4 we compare 3 backbones, PWC-Net, ResNet-50 and ConvNeXt-T. For flow estimation all these backbones perform similarly. However, for semantic content learning, PWC-Net, which is a very shallow network specialized for flow estimation, and it therefore does not perform well on segmentation downstream tasks. The gap in performance between ResNet-50 and ConvNeXt-T is not something specific to this paper, and is observed even in supervised learning on ImageNet. ConvNeXt is an architecture that was designed to perform as good as modern vision transformers and it is not surprising to see such gaps for segmentation tasks. We indeed choose ConvNeXt over ResNet from the beginning for this specific reason.

We refer to the explanation we gave to reviewer q59r on how we tune the coefficients of our losses.

Reviewer comment (Presentation):
The captions of the figures and tables are too detailed, and it is better to condense the captions.

Author response:
The objective was to have self-contained captions, which could be fully understood without diving into the main text. We will change the approach and simplify the captions.

Reviewer comment (Experiments):
In multi-task learning, how are the weights for different loss functions chosen and adjusted? Will adjusting coefficients for the six different loss functions for each task introduce a significant tuning cost during training? Would it be possible to draw comparisons between this tuning approach and other methodologies in the field of multi-task learning?

Author response:
We refer to the explanation from reviewer q59r on how we tune the coefficients of our losses. Our approach is simple and does not require a lot of tuning.

Reviewer comment (Experiments):
Although some experiments have already been conducted to demonstrate improvements in Optical Flow Estimation, Image Segmentation, and Video Segmentation, additional experiments could be included to further validate the effectiveness of the proposed methods. For example, more experiments on video object segmentation (YoutubeVOS), video semantic segmentation, and video panoptic segmentation (Cityscapes-VPS, VIPSeg).

Author response:
We decided to focus on linear frozen segmentation (Pascal, Cityscapes, ADE20k) and non-parametric evaluation (DAVIS 2017), which we believe are more suited to evaluate SSL representations. We nevertheless provide new experiments were we fine-tune our model on YoutubeVOS and Cityscapes-VPS:
			
|			YoutubeVOC (mJ&F)		 | Cityscapes-VPS (VPQ) |
| -- | -- |
|VICReg	 |	80.4				               | 58.6 |
|DINO	 |	83.4				               | 61.1 |
|MC-JEPA |	84.5				                | 61.9 |

Our results further demonstrate the superiority of the flow estimation multi-task over regular semantic content learning.

Reviewer comment (Theory):
What is the motivation for the multi-task learning? Learning optical flow seems to be a low-level visual understanding task; why does it benefit semantic segmentation?

Author response:
Optical flow estimation is indeed a low-level task, and the goal of this paper is to test the following hypothesis: can a low-level task help ground self-supervised training to help learn higher quality high-level representations ? To our surprise, the answer is yes, which we show by comparing MC-JEPA with VICReg and demonstrating strong improvements across all segmentations downstream tasks. Our intuition is that optical flow estimation in this multi-task setup helps structuring the representations with a locality bias, and observing with which pixel is moving where, helps understanding to which object this pixel belongs.