Paper: Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR-2024)

Reviewer comment (Experiments):
It seems that using two teachers for distillation really boosts the performances on downstream tasks. Will it take a longer pre-training time compared to only using CLIP as the teacher? I hope the authors can show the training time and the memory usage during distillation.

Author response:
**Q1: [The training time and memory usage.]:**

Please refer to Q1 for the general response, we claim that introducing additional teacher models inevitably brings more overhead, but the increase is tolerable. Besides, Hybrid Distill **enjoys shorter one-epoch training time and lower memory usage compared to dBOT[1]**. Hence, we believe that the computational cost of Hybrid Distill is acceptable compared to other distillation methods.

Reviewer comment (Experiments):
Previous CL-based methods outperform MAE on linear probing because they own strong discrimination abilities. Since the hybrid distillation keeps this ability, can authors show the linear probing performances on ImageNet-1k?

Author response:
**Q2: [Linear probing results on ImageNet-1K.]:**

The linear probing performances on ImageNet-1k are shown in the table below. The subpar performance of dBOT further validates our perspective discussed in Section 2.3, which suggests that its increase in diversity comes at the expense of sacrificing discrimination. In contrast, Hybrid Distill consistently maintains outstanding linear probing performance, thus confirming the effectiveness of our approach in simultaneously promoting diversity and discrimination.

**Results for linear probing. $\star$: using MAE+DeiT teachers. $\dagger$: using MAE+CLIP teachers.**

| Method                   | Linear Probing |
| :----------------------- | :------------: | 
| MAE                      |      67.8      |
| dBOT                     |      67.9      |
| Hybrid Distill $\star$   |      80.9      |
| Hybrid Distill $\dagger$ |      80.1      |

[1] Exploring target representations for masked autoencoders, 2022.

Reviewer comment (Evaluation):
Sec 2 does not discuss performance, ie the "discrimination" aspect. It is unclear if this is because the two paradigms work equally well on downstream tasks? Probably that's not exactly the case, and generalization is in the end what we care about, ie what we use these features for. Also, wrt 2.2, studies (eg Wang et al and Sariyildiz et al below as well as the papers that introduce such projectors) have shown that the size of projector/"asymmetric decoder" matters wrt to downstream task behaviour.

Author response:
**Q1: [Sec 2 does not discuss performance.]:** 

i)  We have demonstrated the inherent difficulty of achieving both diversity and discrimination through direct feature distillation in Section 2. Therefore, discussing the performance in Section 2 is essentially equivalent to **comparing the superiority of these two properties with respect to different downstream tasks since the student can only guarantee one of these properties, and this comparison has already been discussed in the second paragraph of Section 1 and in previous related works [1-2]**. Specifically, models with better discrimination exhibit advantages in recognition tasks, particularly in linear probing, while models with better diversity excel in dense-level tasks. 

ii) In Q2 ii) of the general response, we have verified that models with better discrimination indeed exhibit superior linear probing performance. However, according to previous works [2-3] and our experience, longer pre-training or distillation periods (e.g., 1600 epochs in [2-3]) are required to achieve sufficient diversity and reap dense-level downstream benefits. In Section 2, we only conduct 100 epoch distillation to compare the distillation trends, which is insufficient to fully reflect the downstream advantages of diversity. Therefore, performance comparisons at this stage have limited value. Nonetheless, our Hybrid Distill design is based on the analysis presented in Section 2, and we consider its effectiveness on dense-level tasks as reliable evidence supporting this analysis. 

iii)  Regarding the two papers you mentioned, they primarily discuss the impact of projectors on CL and Supervised models, whereas our focus is on their influence on the trend of distillation. As a result, they have different motivations with our studies.

Reviewer comment (Experiments):
The method proposed in Sec 3 tries to get the best of both worlds by using two teachers, something that makes training harder and requires more memory.

Author response:
**Q2: [The method makes training harder and requires more memory.]:**

i) Please refer to Q1 of the general response, Hybrid Distill **enjoys lower memory usage compared to MAE and dBOT**. Hence, we believe that the computational cost of Hybrid Distill is acceptable compared to other distillation methods.

ii) Regarding the training difficulty, we argue that Hybrid Distill actually mitigates this issue since the required properties can be directly obtained from teacher models. This claim is supported by the results presented in Table 10, as well as the discussion provided in Section E of the appendix. Specifically, Hybrid Distill can achieve better performance with much fewer training epochs compared to dBOT.

Reviewer comment (Presentation):
More of a note than a weakness: It is now hard to compare the relevant rows in Tables 1 and 2. Tab 1 would be more easy to understand and compare if the two sections are wrt the two different backbones (ie ViT-B on top, ViT-L on the bottom). Inside each section, a separate split should be wrt what is the teacher. The models that distill clip should be clearly separated, cause they start from a much stronger teacher. Same for the ViT-L from the authors, that distills im21k. These results are not comparable to the rest, and the im21k-distilled row seems out of place in both tables. Please make sure to separate accordingly.

Author response:
**Q3: [Hard to compare the relevant rows in Tables 1 and 2.]:** 

Thanks for your suggestion and we have reorganized Table 1 and Table 2 in the revised version.

Reviewer comment (Writing):
What do the authors mean by "s. For ViT-L, the distillation is based on both ImageNet-1K and ImageNet-21K"? It is for different models or for the same model?

Author response:
**Q4: [The distillation schedule when using ViT-L.]:** 

For ViT-L, we offer two distinct models that are derived through different distillation processes. The first model is obtained through a 100-epoch distillation process using ImageNet-1k, while the second model is derived from a 40-epoch distillation process with ImageNet-21k. We have made revisions to the description in order to improve its clarity.

Reviewer comment (Experiments):
Unfair comparison. Table 1 compares to many vanilla training methods on ImageNet-1K but Hybrid Distill exploits CLIP targets and ImageNet-21K for training. Other methods exploiting the joint objectives, better models, or larger scale data are not compared.

Author response:
i) Since the main purpose of our work is to reveal the incompleteness of previous works and to verify the benefits of promoting diversity and discrimination simultaneously, **we conduct most of our experiments in a basic setting**. As shown in Table 1 and Table 9 of our appendix, we **conduct 300 epochs of distillation utilizing the ImageNet-1K dataset and ViT-B/L as backbones**. We only report the results with ImageNet-21K in the last row of Table 1 to illustrate the potential benefits of larger-scale distillation, **while all other results are reported within the basic setting for fair comparisons**.

ii) Besides the CLIP+MAE targets you mentioned, **Hybrid Distill also provides results of using DeiT+MAE targets** and both results showcase consistent benefits towards teacher models and distillation baselines on dense-level detection and segmentation tasks, illustrating the effectiveness of our Hybrid Distill. We have reorganized Table 1 to make the relevant comparisons easy to understand.

Reviewer comment (Novelty):
The two metrics of average attention distance and NMI are existing metrics. They provide insights to models but are still neither necessary or sufficient condition for a good representation. Existing works such as dBOT also visualized the average attention distance differences in existing methods (DeiT, DINO, MAE, etc.) and even different distillation stages and observed similar conclusion as well. Existing joint learning work with the distillation objective is producing good representation already.

Author response:
We agree that average attention distance and NMI do not directly indicate good representation, as they are merely tools used to assess model behavior. However, **we argue that these two metrics have been inadequately interpreted in previous works**. In fact, our work is dedicated to **solving the limitations of the distillation works you mentioned, including dBOT [1]**. We present evidence that the diversity brought by dBOT is derived from its asymmetric decoder design (Section 2.2), while the asymmetric decoder design de facto hinders discrimination (Section 2.3). **This loss in discrimination also explains the phenomenon you mentioned, where models with different teachers (DeiT, DINO, MAE, etc.) exhibit similar behavior in dBOT after multi-stage distillation**. In summary, in Section 2, we have confirmed that **existing works are still far from achieving good representation from analyzing the diversity and discrimination perspectives**.

Reviewer comment (Experiments):
HybridDistill is complicated and designed specifically to ViT architecture with global attention map. Therefore, it looks challenging to apply to other architectures such as Swin or MViT or CNNs.

Author response:
The core of our Hybrid Distill is to utilize MAE and Supervised/CL teachers to ensure both diversity and discrimination. Since **these two kinds of teachers have been confirmed to demonstrate similar behavior in hierarchical transformer structures (e.g., Swin) [2]** when compared to the plain ViT structure, we believe that our designs can be effectively adapted to these hierarchical structures as well. However, our analysis relies on the available pretrained models for efficiency, and the challenge lies in that previous methods have not provided pre-training weights (e.g., CLIP) or baseline results (e.g., [1,3-5]) for these structures. Therefore, further comprehensive validation will be conducted in the future, in parallel with the research progress of the community.

Reviewer comment (Evaluation):
Marginal performance gain compared with existing solutions in fair comparison. E.g. SdAE in Table 1 achieves 84.1, iBOT achieves 84.0, etc. with ViT-B without CLIP or ImageNet-21K, but Hybrid Distill achieving 83.7 lags behind. Similarly limited gain is observed with CLIP targets, e.g. dBOT 87.8% vs. this paper 87.6% with ViT-L ImageNet-1K CLIP targets, and Cascaded detection settings and ADE20K segmentation.

Author response:
i)  **Hybrid Distill achieves noticeable gains in the dense-level downstream for its diversity advantage**. Specifically, when distilled with DeiT+MAE teachers, Hybrid Distill results in a growth of 1.4 $AP_{box}$ and 1.2 $AP_{mask}$ compared to SdAE (r.f., Table 1). In comparison to iBOT, Hybrid Distill demonstrates gains of 1.7 $AP_{box}$ and 1.3 $AP_{mask}$ when using Cascaded Mask-RCNN for downstream adaption  (r.f., Table 10 of the appendix). These improvements are in line with the analysis in previous works [2,6], which indicates that diversity is mainly beneficial for dense-level downstream. While for classification tasks, performance on ImageNet is affected by the saturation of the DeiT teacher. Hence, we focus more on its ability to generalize to other datasets (r.f., Table 2). Specifically, Hybrid Distill achieves 1.4% and 1.2% average accuracy gains towards DeiT and MAE teachers, respectively.

ii)  **The performance comparison you mentioned between dBOT and Hybrid Distill is not conducted under fair conditions**, as Hybrid Distill only uses a basic 300-epoch distillation setting while dBOT distills the model for 1600 epochs.  Nevertheless, we believe that the fact that **300 epoch** Hybrid Distill can achieve comparable or even better performance (r.f., Table 10 of our appendix) compared to **1600 epochs** dBOT already demonstrates its effectiveness. Besides, we have provided 300 epoch dBOT distillation results in Table 9 of the appendix, and also supplement the 600 epoch dBOT distillation results in the table below. **In these fairer comparisons, the advantages of our Hybrid Distill are more pronounced**.


| Method         |   Teacher    |  Epoch  | $AP_{box}$ | $AP_{mask}$ |
| :------------- | :----------: | :-----: | ---------: | ----------: |
| dBOT           |     DeiT     | 300/600 |  47.5/47.7 |   41.9/42.0 |
| dBOT           |     MAE      | 300/600 |  49.3/49.6 |   43.5/43.7 |
| dBOT           |     CLIP     | 300/600 |  49.5/49.8 |   43.5/43.8 |
| Hybrid Distill | MAE and DeiT |   300   |       50.3 |        44.2 |
| Hybrid Distill | MAE and CLIP |   300   |       50.6 |        44.4 |

Reviewer comment (Evaluation):
Runtime compared with baselines. Since Hybrid Distill requires inferencing with multiple backbones and uses less aggressive masking strategy, what is the runtime of one epoch compared with vanilla MAE?

Author response:
Please refer to Q1 of the general response, Hybrid Distill **enjoys shorter one-epoch training time compared to dBOT and lower memory usage compared to MAE and dBOT**. Hence, we believe that the computational cost of Hybrid Distill is acceptable compared to other distillation methods.

Reviewer comment (Writing):
About "diversity": This submission claims that "diversity" means that the model pays attention to both local and global information and can achieve more evenly distributed representations. What is "evenly distributed representations"? Specifically, which does it mean by "distributed representation"?

Author response:
**[The definition of diversity.]** 

The term 'evenly distributed representations' is derived from [1], indicating that the representations are acquired through more evenly distributed attention across different attention distances. Your interpretation is also accurate, as diversity does represent the existence of 'different, or diverse, per-head attention distances'. In the revised version, we have adjusted the description for clarity.

Reviewer comment (Writing):
About "diversity": My understanding is that the "diversity" is quantitively measured by both average attention distance and NMI. I am not sure if "diversity" is a good terminology in this case, which oversimplified many necessary context. I suggest, for example, "different per-head attention distance", to more accurately describe the discussed property. For simplicity, I will still use "diversity" in my review.

Author response:
**[The definition of diversity.]** 

The term 'evenly distributed representations' is derived from [1], indicating that the representations are acquired through more evenly distributed attention across different attention distances. Your interpretation is also accurate, as diversity does represent the existence of 'different, or diverse, per-head attention distances'. In the revised version, we have adjusted the description for clarity.

Reviewer comment (Experiments):
Questions on results: Why MAE's COCO results in the main Table (Table 1) are lower than the results reported in MAE's original paper? For example, in MAE, a ViT-B is reported to get 50.3 APbox and 44.9 APmask, but 48.4 APbox and 42.6 APmask in this submission? Please highlight the experimental setting differences to MAE and other compared baselines in the paper if this is the reason. However, if the setting difference is the reason for performance difference on COCO, then why ADE20K results are matched?

Author response:
**Q2: [MAE results. ]:** 

Sorry for the confusion, the difference between reported performance and original values in the MAE paper is attributed to **variations in downstream settings**.  We follow the settings of recent works  [3-5] (which are also widely adopted currently) for employing pretrained models in downstream tasks of COCO detection and ADE20K segmentation. For COCO detection:

i) **The MAE paper finetunes the pretrained model for 100 epochs**, which is time-consuming and not a common downstream setting. Besides, MAE does not release the official codes for COCO detection and some fine-tuning details are missing. Therefore, we follow the common practice in [3-5] that **employs a 1x schedule (12 epochs)** for fine-tuning, and our reported MAE results align with these papers (48.4 $AP_{box}$ and 42.6 $AP_{mask}$). 

ii) In Table 10 of our appendix, **we also provide another set of COCO detection results using Cascaded Mask-RCNN for comprehensiveness**. According to [2], the reported MAE result under this setting is 50.6 $AP_{box}$, while our Hybrid Distill achieves 53.0 $AP_{box}$, which is 2.4 points higher than MAE. This clearly demonstrates the advantages of our method compared to MAE across different settings.

Reviewer comment (Evaluation):
Questions on results: If directly distilling CLIP's feature harms dense tasks for its inadequate "diversity" in self-attention distance, then why results reported in MILAN outperform the proposed method on COCO detection/instance segmentation and ADE20K semantic segmentation? I think MILAN directly predicts CLIP's features. Also, MILAN is not cited and compared in the submission. [MILAN] Hou et al, MILAN: Masked Image Pretraining on Language Assisted Representation.

Author response:
**Q3: [MILAN is not cited and compared in the submission.]:**

i) **MILAN does not directly distill features**. Instead, it introduces a prompting decoder design that involves freezing the unmasked tokens. These tokens are then used as keys and values to interact with learnable mask tokens in the decoder. In our updated supplementary materials, we have verified that this design can alleviate the discrimination loss caused by the asymmetric decoder, resulting in improved performance. Therefore, the reason for its performance gain compared to direct distillation can be explained by the metrics in our paper, i.e.,  it successfully improves diversity while still maintaining a certain level of discrimination. We have cited this paper in the revised version.  

ii) MILAN follows the downstream setting of MAE and also does not disclose its details. Therefore, **we re-evaluate the performance of MILAN under our 1x downstream setting for a fair comparison, and our method outperforms MILAN by 0.9 $AP_{box}$ and 0.7 $AP_{mask}$**, as demonstrated in the table below. Evidence for this improvement can be found in the supplementary materials, i.e., MILAN does not exhibit the same level of diversity gain and discrimination preservation capabilities as our Hybrid Distill.

**Results compared with MILAN**

| Method         |   Teacher    | $AP_{box}$ | $AP_{mask}$ |
| :------------- | :----------: | :--------: | ----------: |
| MILAN          |     CLIP     |    49.7    |        43.7 |
| Hybrid Distill | MAE and DeiT |    50.3    |        44.2 |
| Hybrid Distill | MAE and CLIP |    50.6    |        44.4 |