Paper: Exploring Target Representations for Masked Autoencoders (ICLR-2024)

Reviewer comment (Novelty):
Regarding target representation, is the bottleneck of scalable visual representation learning not about the methodology but more about data?

Author response:
In our paper, we conducted experiments with the same dataset (IN-1K) and the same setting (multi-stage distillation). We specifically focused on masked knowledge distillation, which utilizes masked image modeling as the pretext task for self-supervised training. We observed that the target representation is not crucial for mask knowledge distillation when distillation is performed in stages. However, we believe that new sophisticated pretext tasks can further enhance the performance. The bottleneck of scalable visual representation learning arises from both methodology and data.

Reviewer comment (Experiments):
From the perspective of asymmetric masking, another way to understand the observed phenomenon is due to the limited scalability [1] of the asymmetric masking architecture proposed in MAE, which is also utilized as the main architecture in this work. While in Swinv2 and EVA, the MIM pre-training has been quite important for ViT training of giant sizes. Does the implementation of masking also affect the scalability of pre-training?

Author response:
The masking operation is indeed the key in masked knowledge distillation for self-supervised learning. Through the same pre-training task, the target representation is not important in multi-stage distillation, and dBOT can achieve a similar weight distribution. The implementation of masking also affects the scalability of pre-training. Like MAE, even changing the mask ratio can affect the model performance, as shown in our ablation study.

Reviewer comment (Theory):
Moreover, in the context of asymmetric masking, does the invariance of target representation suggest masking operation is the key in MKD, which might also explain why the learnt representation has similar weight distribution?

Author response:
The masking operation is indeed the key in masked knowledge distillation for self-supervised learning. Through the same pre-training task, the target representation is not important in multi-stage distillation, and dBOT can achieve a similar weight distribution. The implementation of masking also affects the scalability of pre-training. Like MAE, even changing the mask ratio can affect the model performance, as shown in our ablation study.

Reviewer comment (Experiments):
Regarding backbone architecture, throughout the whole paper, the authors utilize the vanilla ViT of different sizes for both the student and teacher networks, while DeiT has shown that architecture discrepancy exists when distilling between different architectures like CNN and ViT. Will that be the same for MKD, i.e., does the phenomenon still hold with a ConvNext teacher?

Author response:
In our experiment, we used the same architecture for both the student and teacher networks, as the teacher's weights are loaded from the pre-stage student's weights in multi-stage distillation. Therefore, the teacher and student networks should be the same. Previous works such as DINO and DALL-E were pre-trained on vanilla VIT, which facilitated our study on masked knowledge distillation with vanilla VIT as the backbone. As you mentioned ConvNext, a possible experiment would involve performing DINO and DALL-E on ConvNext first, and then applying masked knowledge distillation. However, whether DINO and DALL-E are suitable for ConvNext is a question that goes beyond the scope of our paper. We would like to leave this as an open-ended question.

Reviewer comment (Writing):
Typo: "self-supervised learning" instead of "self-supervised learninf" in Keywords.

Author response:
We will fix the typos.

Reviewer comment (Writing):
The conclusion of the paper is limited to Masked auto-encoders where the teacher and student models are pre-trained with the same data (IN-1k). However, this is not clearly stated in the main paper.

Author response:
In our pre-training and fine-tuning, we use IN-1K as the dataset and masked autoencoder as the student models. This is described in section 3 (common setup). We will clearly state the technical details in our revised version.

Reviewer comment (Experiments):
While the authors mention the instability and sensitivity of other methods, it should be noted that MKD also requires careful pipeline design and hyper-parameter selection. For example, MAE with fixed m (momentum updating teacher’s param) can achieve 84.3 at stage 2, however, in most cases in the ablation, MKD only outperforms MAE when the stage split number, epochs for each stage, and momentum parameter (m) and others are correctly set. This limits the practical application of MKD, as one could simply distill MAE for two stages with less time consumption and similar performance.

Author response:
The setting for distilling MAE in the two-stage process is selected based on the ablation in section 5.4. In Table 1, all the results are set the same for a fair comparison. Distilling the mask autoencoder for two stages with any teacher achieves similar performance in classification, object detection, and segmentation.

Reviewer comment (Experiments):
The paper only explores the fine-tune setting of semi-supervised learning (SSL), which diminishes the differences between pre-training models. How about the linear-probe setting? Would teachers also matter? Additionally, it is unclear whether the models mentioned in Section 6 are supervisedly fine-tuned on IN or pre-trained models before fine-tuning.

Author response:
The linear probing results of dBOT using DINO as the teacher are 77.35%, while dBOT using other teachers achieves similar results (69.1%). We attribute this to the multi-crop technique and contrastive learning used in DINO. In section 6, all the models are pre-trained models before fine-tuning.

Reviewer comment (Experiments):
A minor weakness of the paper is that only l2 distillation is performed, neglecting other potential distillation methods.

Author response:
We are investigating dBOT's effectiveness in other potential distillation methods except for l2 distillation. We appreciate your valuable feedback and will continue to explore and expand the scope of our work.

Reviewer comment (Evaluation):
**dBOT does not scale with more training?** Results in Table 1 show a drop in performance on adding more stages, thus requiring a good stopping condition.

Author response:
We perform multi-stage training using pixels as the target. The fine-tuning performance on IN-1K at the 0th, 1st, 2nd, and 3rd stages is 83.6, 84.3, 84.4, and 84.3, respectively. The performance saturates after two-stage distillation and fluctuates around the saturation performance.

Reviewer comment (Evaluation):
**Different models for different tasks** as opposed to prior works that typically release a single general model. The purpose of SSL is to learn a single model that generalize across tasks. Given that the performance on different downstream tasks peaks at different stages (Table 1), it is not clear how to choose the number of distillation stages to arrive at a single general model that can work across tasks (even beyond the ones discussed - e.g., for embodied navigation [1]).

Author response:
As we explained in response to Q1, the performance will fluctuate around the saturation performance. Downstream users can load any model after two-stage distillation with a slight performance difference.

Reviewer comment (Experiments):
**Do the trends hold across random seeds?** For a single random initialized teacher, Table 1 shows 2 stage being better for classification and stage 3 being better for detection/segmentation. It is not clear if trends like this (and the performance) hold across random seeds.

Author response:
During our experiments, we found that the performance of classification and detection is quite stable, except for the segmentation tasks. The segmentation performance reported in our paper is averaged over 5 random seeds, which we will clearly state in our revised version.

Reviewer comment (Experiments):
In context of the weaknesses above, do authors have evidence that suggests that performance saturates and not drops with more stages and training?

Author response:
We perform multi-stage training using pixels as the target. The fine-tuning performance on IN-1K at the 0th, 1st, 2nd, and 3rd stages is 83.6, 84.3, 84.4, and 84.3, respectively. The performance saturates after two-stage distillation and fluctuates around the saturation performance.

Reviewer comment (Experiments):
**Sample efficiency.** SSL methods have been shown to improve with more training (Figure 7 of MAE [1]). Is the total number of pre-training iterations controlled across different methods in Table 2, 3, 4 and 5?

Author response:
In Table 5, we conducted ablation studies to choose the best settings for dBOT. Then, the best setting was used in Tables 2, 3, and 4. The model performance saturates after multi-stage distillation.

Reviewer comment (Evaluation):
Table 6: Why is MAE faster than dBOT when both use an asymmetric encoder-decoder architecture?

Author response:
MAE uses the original pixel as the target, which does not involve a teacher network, while dBOT uses an extra teacher to extract the reconstructed token.

Reviewer comment (Evaluation):
"Additionally dBOT achieves more local attention than previous works". Can authors clarify this comment in context of Figure 2? It seems like MAE's attention distance plot is similar to the distilled models'.

Author response:
They are similar but there are differences between them. Taking the last layer as an example, the maximum average attention distance for MAE is 85, while it is 70 for dBOT. The minimum average attention distance for MAE is 40, while it is 30 for dBOT. dBOT achieves more local attention than previous works.

Reviewer comment (Theory):
Section 6. SVD line 5: Shouldn't lesser correlation in model's output result in lower redundancy?

Author response:
We apologize for this mistake and will correct it in the revised version.

Reviewer comment (Theory):
C.1. I think I understand why the need for a decoder could be "eased when target contains high-level semantics". But can the authors elaborate a bit on why the "existence of decoder" may hurt?

Author response:
The encoder and decoder are both used in the pre-training stage, but the decoder is discarded in the fine-tuning stage. When the target is low-level, the decoder is needed to decode the high-level representations encoded by the encoder. However, when the target contains high-level semantics, the existence of the decoder may even restrain the encoder from fully grasping knowledge from the teacher, leading to degraded performance.

Reviewer comment (Novelty):
The observation of the insignificance of the choice of teacher network is somewhat aligned to the observation of MAE, which simply opts for raw image pixels and surpasses previous methods like BeiT or MaskFeat with more sophisticated target representation by a non-trivial margin. This is also validated from the results in appendix B.1, which shows *using the patch token obtained by a randomly initialized network as the target can achieve comparable results with a pixel as a target.*

Author response:
**Q2: The observation is somewhat aligned with the observation of MAE.**

In the early stages of Masked Image Modeling (MIM), researchers used different targets for performing MIM, such as pixels, Histogram of Oriented Gradients (HOG), and tokens from a pre-trained teacher. Researchers have different opinions on target selection and different results are achieved with different targets. MAE simplifies MIM by using pixels as the target and achieves better results than Beit and maskfeat, which demonstrates that using pixels is better than HOG and tokens from a pre-trained teacher. However, our paper finds that the choice of the teacher network is not crucial if the distillation is done in stages. In other words, similar results can be achieved by using any target (as shown in Table 1), even a randomly initialized teacher. This is the main difference between dBOT and MAE.

Reviewer comment (Novelty):
The authors claim that *Using a random model as teachers not only avoids an extra pre-training stage, but also alleviates the painstaking selection of the target representations*. But I think this has already been achieved by MAE, since reconstrucing raw image pixels does not involve pretraining, and there is no need to select target representations. Thus, I am not sure what is the practical value of the observation of this work.

Author response:
**Q3: Reconstructing raw pixels.**

MAE achieves good results by simply using pixels as a target. However, later works, like [1], achieve better results by using tokens from pre-trained teachers as targets. Different results are achieved with different targets. The motivation of our paper is that there is no work that provides a system-level study on the importance of how to choose an adequate target representation or teacher network to guide the learning of Masked Knowledge Distillation (MKD). Our work reveals that the choice of teacher is not crucial if the distillation is done in stages. In particular, we find that using a randomly initialized model as a teacher can achieve similar results through multi-stage distillation, which we formulate as dBOT. This alleviates the painstaking selection of target representations since we achieve similar performances with different target representations in multi-stage distillation.

Reviewer comment (Evaluation):
In table 1, I find a randomly initialized teacher network could achieve 77.3 accuracy on ImageNet (and similar for other datasets), which really seems impossible to me. Am I missing something here?

Author response:
**Q4: The results of randomly initialized teacher.**

The results of the 0th stage are the initialized teacher, and the results of the 1st and 2nd stages are the distilled student, which is described in Section 3.1. Therefore, 77.1 is the result of fine-tuning a randomly initialized model on IN-1K.