Paper: CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction (ICLR-2024)

Reviewer comment (Experiments):
This paper does not include any studies on windowed attention based methods, such as Swin V2. This seems to have weakened the case for CLIPSelf.

Author response:
**R4-1: Swin-based Backbones** Directly verifying CLIPSelf on Swin backbones is impractical as there is no published VLM based on a Swin backbone, which is agreed by **Reviewer yRsy**. Therefore, we opt for the alternative proposed by **Reviewer yRsy** to replace the global attention in CLIP ViTs with local window attention. The results are as follows.

|     |        Method        |  Model   |   Boxes   | Thing Masks | Stuff Masks |  OV-COCO   |
|:---:|:--------------------:|:--------:|:---------:|:-----------:|:-----------:|:----------:|
|  1  | GlobalAttn(original) | ViT-B-16 | 18.2 |  20.6  |  18.4  | 17.5  |
|  2  |      WindowAttn      | ViT-B-16 | 34.7 |  40.6  |  30.9  | 19.4  |
|  3  | GlobalAttn+CLIPSelf  | ViT-B-16 | 72.1 |  74.4  |  46.8  | 33.6  |
|  4  | WindowAttn+CLIPSelf  | ViT-B-16 | 73.3 |  74.9  |  48.6  | 33.6  |

For a CLIP ViT enhanced with window attention, the performances on the zero-shot box/mask recognition
and open-vocabulary detection are slightly improved (#1 and #2) but still lag behind CLIPSelf by a large margin (#2 and #3). Moreover, we show that CLIPSelf also applies to the window attention-based CLIP. Therefore, CLIPSelf is also applicable to ViT models equipped with local window attentions.

Reviewer comment (Experiments):
The analysis that compares CLIP ViT and CLIP CNN can also be interpreted as "CNN is more suited for dense prediction tasks." Recent empirical studies overwhelmingly show that transformer-based models outperform CNN-based models in many computer vision tasks, including dense prediction tasks. From the analysis, it appears that CLIPSelf is most helpful with backbones that do not have translation equivariance built-in. However, many (if not most) of the best models for dense prediction tasks still use backbones that have this useful property for dense predictions (e.g., Swin V2). This is the choice for some of the best recent open-vocabulary detectors, such as GLIP (v2) and Grounding DINO. The choice to only study ViT backbones leaves two obvious questions open: (1) Can CLIPSelf similarly benefit Swin-based backbones? (2) How does the current method compare to SOTA methods based on Swin V2, such as GLIP and Grounding DINO?

Author response:
**R4-1: Swin-based Backbones** Directly verifying CLIPSelf on Swin backbones is impractical as there is no published VLM based on a Swin backbone, which is agreed by **Reviewer yRsy**. Therefore, we opt for the alternative proposed by **Reviewer yRsy** to replace the global attention in CLIP ViTs with local window attention. The results are as follows.

|     |        Method        |  Model   |   Boxes   | Thing Masks | Stuff Masks |  OV-COCO   |
|:---:|:--------------------:|:--------:|:---------:|:-----------:|:-----------:|:----------:|
|  1  | GlobalAttn(original) | ViT-B-16 | 18.2 |  20.6  |  18.4  | 17.5  |
|  2  |      WindowAttn      | ViT-B-16 | 34.7 |  40.6  |  30.9  | 19.4  |
|  3  | GlobalAttn+CLIPSelf  | ViT-B-16 | 72.1 |  74.4  |  46.8  | 33.6  |
|  4  | WindowAttn+CLIPSelf  | ViT-B-16 | 73.3 |  74.9  |  48.6  | 33.6  |

For a CLIP ViT enhanced with window attention, the performances on the zero-shot box/mask recognition
and open-vocabulary detection are slightly improved (#1 and #2) but still lag behind CLIPSelf by a large margin (#2 and #3). Moreover, we show that CLIPSelf also applies to the window attention-based CLIP. Therefore, CLIPSelf is also applicable to ViT models equipped with local window attentions.

Reviewer comment (Experiments):
The focus solely on ViT backbones makes it harder, in certain cases presented in the experiments section, to interpret the comparison with prior methods. For example, in Table 3, CLIPSelf uses either ViT-B/16 or ViT-L/14 backbones. However, most of the cited methods are either from CNN backbones or Swin V2. In the few instances where the prior methods are based on ViT, the particular ViT variant is different from the two used with CLIPSelf.

Author response:
**R4-3: Comparison With Prior Methods** We have made the benchmark comparison as fair as possible by limiting the data of distillation to only COCO and LVIS datasets. The compared ViTs (in RO-ViT) are of different architectures because they include special designs for object detection. We outperform them with Valina ViT architectures, which reflects the superiority of CLIPSelf. And please kindly note Table 3(b), where our ViT-L/14 model outperforms the larger ViT-H/16 model of RO-ViT. This is also acknowledged by reviewer **Reviewer CnGz**, who commented "More notably, CLIPSelf outperforms the recent work of RO-ViT which proposes training a new backbone catered to downstream task from scratch. I find this result particularly appealing as it shows that CLIPSelf can lightly adapt a wide range of publicly available checkpoints that were trained with relatively less consideration for the particular task of open-vocabulary object detection".

To make the benchmark comparison more relevant to recent ViT-based approaches, we include the following ViT-based methods in our revised version, which are highlighted in the modified Table 3. "L" stands for the same model size (24 layers of attention modules) so the following comparison is built on the same / similar backbone compacity.


|     |  Method  |  Model   | OV-COCO  | OV-LVIS  |
|:---:|:--------:|:--------:|:--------:|:--------:|
|  1  |  OW-ViT [e] | ViT-L/14 |    -     |   25.6   |
|  2  |  RO-ViT [h] | ViT-L/16 |   33.0   |   32.1   |
|  2  | CFM-ViT [i] | ViT-L/16 |   34.1   |   33.9   |
|  3 | CLIPSelf | ViT-L/14 | **44.3** | **34.9** |


[e] Simple Open-Vocabulary Object Detection with Vision Transformers, Minderer et.al., ECCV 2022

[h] Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers, Kim et.al., CVPR 2023

[i] Contrastive Feature Masking Open-Vocabulary Vision Transformer, Kim et.al., ICCV 2023

Reviewer comment (Experiments):
I would like to see comparisons with GLIP, GLIP v2 and Grounding DINO.

Author response:
**R4-2: Comparisons with GLIP, GLIP v2, and Grounding DINO** Although we appreciate the reviewer's suggestion on comparison with GLIP-like methods, we clarify that such comparison is infeasible and would be under an unfair setting. CLIPSelf follows the strict definition of open-vocabulary object detection that does not incorporate external region-level annotations. The only box annotations are the boxes of base categories defined in the COCO or LVIS dataset. Such a setting is widely adopted by mainstream open-vocabulary approaches [a,b,c,d,e,f,g,h, i]. In contrast, the so-called open-vocabulary object detection in GLIP, GLIP v2, and Grounding DINO is under a much looser definition, which allows training on the large detection dataset Objects365 and various grounding datasets such as Flickr30K, VG Caption, GQA, and RefCOCO. Therefore, CLIPSelf and GLIP-like methods belong to different research tracks and the comparison would not make much sense. We list some results here for reference but we do not plan to add them to the final version of our paper.


|     |         Method         |  Model   | Box AP on LVIS |
|:---:|:----------------------:|:--------:|:--------------:|
|  1  |         GLIP-L         |  Swin-L  |      26.9      |
|  2  |    Grounding-DINO-L    |  Swin-L  |      33.9      |
|  3  |        CLIPSelf        | ViT-L/14 |      37.3      |

For more details on the setting of open-vocabulary object detection, please refer to the very first paper in this direction OV-RCNN [a].


[a] Open-Vocabulary Object Detection Using Captions, Zareian et.al., CVPR 2021.

[b] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, Gu et.al., ICLR 2022

[c] Open-Vocabulary DETR with Conditional Matching, Zang et.al., ECCV 2022.

[d] Detecting Twenty-thousand Classes using Image-level Supervision, Zhou et.al., ECCV 2022

[e] Simple Open-Vocabulary Object Detection with Vision Transformers, Minderer et.al., ECCV 2022


[f] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models, Kuo et.al., ICLR 2023

[g] Aligning Bag of Regions for Open-Vocabulary Object Detection, Wu et.al., CVPR 2023


[h] Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers, Kim et.al., CVPR 2023

[i] Contrastive Feature Masking Open-Vocabulary Vision Transformer, Kim et.al., ICCV 2023

Reviewer comment (Experiments):
I would like to see if the proposed method can only improve on ViT-based backbones.

Author response:
**R4-4: If the proposed method can only improve on ViT-based backbones** Conceptually, the proposed method applies to any VLM that has inferior region-language alignment. As the published models are mainly CNNs and valina ViTs, we only analyze these two variants in the paper as shown in Figure 1 and Figure A1. CNN-based CLIP models have shown impressive region recognition ability, so we only apply CLIPSelf to ViTs. In addition, CLIPSelf also applies to the CLIP model enhanced with local window attention which is supplemented in the rebuttal. Moreover, it is noteworthy that a valina transformer serves as a general architecture that unifies both vision and language processing without task-specific designs. Recently, there has been a trend of applying such frozen ViT-based VLMs to downstream tasks like multi-modal LLMs. In the context of adopting the freezing strategy to preserve VLM's generalization ability, CLIPSelf exhibits its advantage in preserving zero-shot ability and requiring only image data.

Reviewer comment (Experiments):
**Is self-distillation on task dataset (COCO) necessary?** Main CLIPSelf experiments use images from COCO/LVIS dataset to perform self-distillation. The authors should consider disentangling the impact of self-distillation in isolation, as compared to self-distillation using images from the target downstream task. If CLIPSelf does not _require_ COCO images, then it strengthens the contribution -- the authors may call for future works to "fine-tune" their CLIP ViT for an additional epoch to improve their out-of-the-box usability for dense prediction tasks. It would be even better if CLIP ViTs retain their original zero-shot classification and retrieval performance after such self-distillation. If the authors wish to do this experiment in the rebuttal, they may consider a large dataset of diverse unlabeled images (e.g. images from datasets with approx. 10-20 million samples like Conceptual Captions, RedCaps, YFCC-15M, Segment Anything dataset, etc.)

Author response:
**R3-1: Distillation on Non-COCO Dataset** We do the distillation only on the COCO/LVIS dataset mainly to ensure a fair comparison with existing methods, especially those distillation-based and frozen CLIP-based approaches. Here we add the experiment on the CC3M dataset, which includes 28 million images. The results are as follows.


|     | Distillation Dataset |        Model        | Boxes  | Thing Masks | Stuff Masks | OV-COCO | OV-LVIS |
|:---:|:--------------------:|:-------------------:|:------:|:-----------:|:-----------:|:-------:|:-------:|
|  1  |          -           | ViT-B-16 (EVA-CLIP) |  18.2  |    20.6     |    18.4     |  17.5   |  11.5   |
|  2  |         COCO         | ViT-B-16 (CLIPSelf) |  72.1  |    74.4     |    46.8     |  33.6   |  25.3   |
|  3  |         CC3M         | ViT-B-16 (CLIPSelf) |  72.1  |    74.5     |    49.5     |  35.8   |  26.6   |

As shown in the table, CLIPSelf exhibits consistent improvement on both zero-shot box and mask recognition as well as open-vocabulary detection on the COCO/LVIS dataset.

Reviewer comment (Experiments):
**Accounting for the 'padding' patches during distillation.** Current state-of-the-art object detectors using plain ViTs (e.g. ViTDet and SAM) typically pad input images to a square. Any non-square input images are made square by adding zero pixels to the right and bottom after color normalization. Pre-trained CLIP ViTs have not seen such patches so they encounter a train-test mismatch if they are kept frozen in the object detector. Have the authors considered randomly adding padding regions to image patches while performing self-distillation? I believe that accounting for this detail will ease the adaptation of CLIP ViTs for open-vocabulary object detection.

Author response:
**R3-2: 'Padding' Patches** Thanks for the suggestion. We have already done this in both self-distillation and detection training by adding zero pixels to the right and bottom of non-square images after color normalization. We'll make this clear in the implementation details.

Reviewer comment (Writing):
Minor question: Figure 3 (b) non-standard word use: "initiate" -> "initialize"

Author response:
**R3-3: Typos** Thanks for pointing out the typos in Figure 3 and Figure A2. We'll correct them in the final version.

Reviewer comment (Presentation):
Minor question: Figure A2 caption is incorrect: "image segmentation" -> "object detection"

Author response:
**R3-3: Typos** Thanks for pointing out the typos in Figure 3 and Figure A2. We'll correct them in the final version.

Reviewer comment (Novelty):
There are some related works that already explored the way of distilling patch-level representations to regional feature-level representations, such as ZeroSeg proposed in "Exploring Open-Vocabulary Semantic Segmentation without Human Labels. Chen et al. ICCV 2023". The overall methodology proposed in this work resembles the one in ZeroSeg, in that both attempted to distill the CLIP knowledge from patch to feature map and for open-vocabulary segmentation tasks.

Author response:
**R2-1: Discussion on ZeroSeg** Although ZeroSeg and CLIPSelf are similar in that both approaches perform distillation based on image patches instead of human annotations, we would like to categorize ZeroSeg to distillation-based open-vocabulary approaches that typically transfer the knowledge of CLIP models to downstream object detection models or image segmentation models, including ViLD[a], and the follow-ups like OV-DETR[b] and BARON[c]. Therefore, such approaches are quite task-specific as the students vary in tasks and architectures. In contrast, CLIPSelf is generally applicable to various downstream tasks with the teacher and student being the exactly same model. And the zero-shot recognition ability is better-preserved thanks to this self-distillation design, which is verified by the benchmark results. Abstractly, CLIPSelf is at a point between upstream vision-language pretraining and downstream detection/segmentation applications like ZeroSeg and ViLd.

In addition, there are also clear differences in the technical details. As the segment tokens in ZeroSeg do not preserve explicit location information, ZeroSeg applies both a global-local distillation loss and a segment-matching loss. In comparison, the design of CLIPSelf is much simpler due to the direct correspondence between image crops and locations on the dense feature map, with a single loss that simply maximizes cosine similarities between dense features and the corresponding crop features. Moreover, the distillation of CLIPSelf would be more precise as we do not require the noisy matching process that is necessary for ZeroSeg.

[a] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, Gu et.al., ICLR 2022

[b] Open-Vocabulary DETR with Conditional Matching, Zang et.al., ECCV 2022.

[c] Aligning Bag of Regions for Open-Vocabulary Object Detection, Wu et.al., CVPR 2023

Reviewer comment (Experiments):
It turns out that the authors only used the COCO training set for distillation. A doubt on this setting is that it is probably very favorable to downstream tasks like COCO detection and segmentation, and ADE20K as well. Though this in-domain distillation brings a significant boost, the authors should have more studies and evaluations on more downstream datasets from various domains and also try using non-COCO training sets for distillation, which seems to be more realistic.

Author response:
**R2-2: Non-COCO datasets** Thanks for the suggestion. We do the distillation only on the COCO/LVIS dataset mainly to ensure a fair comparison with existing methods, especially those distillation-based and frozen CLIP-based approaches. For various downstream datasets, we already include such results in Table 4,5,6, where the LVIS-trained or COCO-trained models are evaluated on other datasets. For a non-COCO dataset in self-distillation, we supplement the experiment using the CC3M dataset and the results are as follows.


|     | Distillation Dataset |        Model        | Boxes  | Thing Masks | Stuff Masks | OV-COCO | OV-LVIS |
|:---:|:--------------------:|:-------------------:|:------:|:-----------:|:-----------:|:-------:|:-------:|
|  1  |          -           | ViT-B-16 (EVA-CLIP) |  18.2  |    20.6     |    18.4     |  17.5   |  11.5   |
|  2  |         COCO         | ViT-B-16 (CLIPSelf) |  72.1  |    74.4     |    46.8     |  33.6   |  25.3   |
|  3  |         CC3M         | ViT-B-16 (CLIPSelf) |  72.1  |    74.5     |    49.5     |  35.8   |  26.6   |

As shown in the table, CLIPSelf exhibits consistent improvement on both zero-shot box and mask recognition as well as open-vocabulary detection on the COCO/LVIS dataset. We're glad that this experiment is also advised by **Reviewer CnGz**, who suggested that the experiment "strengthens the contribution -- the authors may call for future works to 'fine-tune' their CLIP ViT for an additional epoch to improve their out-of-the-box usability for dense prediction tasks".

Reviewer comment (Novelty):
How to distinguish this work from ZeroSeg regarding the core learning algorithm?

Author response:
**R2-1: Discussion on ZeroSeg** Although ZeroSeg and CLIPSelf are similar in that both approaches perform distillation based on image patches instead of human annotations, we would like to categorize ZeroSeg to distillation-based open-vocabulary approaches that typically transfer the knowledge of CLIP models to downstream object detection models or image segmentation models, including ViLD[a], and the follow-ups like OV-DETR[b] and BARON[c]. Therefore, such approaches are quite task-specific as the students vary in tasks and architectures. In contrast, CLIPSelf is generally applicable to various downstream tasks with the teacher and student being the exactly same model. And the zero-shot recognition ability is better-preserved thanks to this self-distillation design, which is verified by the benchmark results. Abstractly, CLIPSelf is at a point between upstream vision-language pretraining and downstream detection/segmentation applications like ZeroSeg and ViLd.

In addition, there are also clear differences in the technical details. As the segment tokens in ZeroSeg do not preserve explicit location information, ZeroSeg applies both a global-local distillation loss and a segment-matching loss. In comparison, the design of CLIPSelf is much simpler due to the direct correspondence between image crops and locations on the dense feature map, with a single loss that simply maximizes cosine similarities between dense features and the corresponding crop features. Moreover, the distillation of CLIPSelf would be more precise as we do not require the noisy matching process that is necessary for ZeroSeg.

[a] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, Gu et.al., ICLR 2022

[b] Open-Vocabulary DETR with Conditional Matching, Zang et.al., ECCV 2022.

[c] Aligning Bag of Regions for Open-Vocabulary Object Detection, Wu et.al., CVPR 2023

Reviewer comment (Reproducibility):
How to adapt the VIT models to different input image resolutions?

Author response:
**R2-3: Different Resolutions** The attention modules are agnostic to input size, we only need to upsample the positional embeddings to fit different image resolutions.

Reviewer comment (Evaluation):
This speculation is partially supported by Fig 1 (b), where the gap between dense features and cropped image features is negligible for CNN (where the features are localized), yet rather significant for ViT. This difference is attributed in part to the design of ViT using global attention, where the dense features are diffused so that each spot encodes a global image representation.

Author response:
**R1-1: Retrieval Experiment** We greatly appreciate the reviewer's valuable insight that "the dense features are diffused in a way that each spot encodes a global image representation". And we conduct the retrieval experiment proposed by the reviewer.

Specifically, we first encode images and regions annotated in COCO's validation split. Then we extract dense feature maps of the images and let each location in the feature map retrieve the image and region that it belongs to by calculating cosine similarity between the location feature and the encoded image or region features. For each retrieval, we provide 50 samples (1 positive and 49 negative samples) and calculate recall at 1, 5, and 10, respectively. The results are as follows.


|        Model        | Image R@1 | Image R@5 | Image R@10 | Region R@1 | Region R@5 | Region R@10 |
|:-------------------:|:---------:|:---------:|:----------:|:----------:|:----------:|:-----------:|
| ViT-B-16 (EVA-CLIP) |   67.0    |   85.6    |    91.4    |    34.7    |    60.5    |    71.4     |
| ViT-B-16 (CLIPSelf) |   28.4    |   50.3    |    62.5    |    57.1    |    80.1    |    85.8     |

The experimental results are a direct verification of the reviewer's hypothesis. For the original CLIP model, the dense features are well-matched with the corresponding images, indicating that "each spot encodes a global image representation". The model refined by CLIPSelf yields dense features that match the corresponding regions instead of images, revealing the transfer of knowledge from global image representation to local region representation. We add this experiment to the paper to better interpret our intuition.

Reviewer comment (Experiments):
There might be ways to validate this explanation. For example, one possibility is a retrieval experiment using features on the dense maps. Features on dense maps from pre-trained CLIP ViT might yield similar images instead of similar regions, while features after distillation may better retrieve regions. The reviewer requests that the authors comment on this reasoning and discuss the key intuition more explicitly in the paper.

Author response:
**R1-1: Retrieval Experiment** We greatly appreciate the reviewer's valuable insight that "the dense features are diffused in a way that each spot encodes a global image representation". And we conduct the retrieval experiment proposed by the reviewer.

Specifically, we first encode images and regions annotated in COCO's validation split. Then we extract dense feature maps of the images and let each location in the feature map retrieve the image and region that it belongs to by calculating cosine similarity between the location feature and the encoded image or region features. For each retrieval, we provide 50 samples (1 positive and 49 negative samples) and calculate recall at 1, 5, and 10, respectively. The results are as follows.


|        Model        | Image R@1 | Image R@5 | Image R@10 | Region R@1 | Region R@5 | Region R@10 |
|:-------------------:|:---------:|:---------:|:----------:|:----------:|:----------:|:-----------:|
| ViT-B-16 (EVA-CLIP) |   67.0    |   85.6    |    91.4    |    34.7    |    60.5    |    71.4     |
| ViT-B-16 (CLIPSelf) |   28.4    |   50.3    |    62.5    |    57.1    |    80.1    |    85.8     |

The experimental results are a direct verification of the reviewer's hypothesis. For the original CLIP model, the dense features are well-matched with the corresponding images, indicating that "each spot encodes a global image representation". The model refined by CLIPSelf yields dense features that match the corresponding regions instead of images, revealing the transfer of knowledge from global image representation to local region representation. We add this experiment to the paper to better interpret our intuition.

Reviewer comment (Experiments):
Another question is whether the proposed method is a remedy to a particular design (ViT with global self-attention). The reviewer wonders what would happen if a CLIP model is trained using ViT with local self-attention (e.g., using Swin Transformers), and whether such models would still benefit from the proposed method. While this is hard to validate directly due to the lack of pre-trained models, one possible alternative is modifying a pre-trained ViT by replacing global attention with local ones, and then training for those open-vocabulary tasks. The reviewer notes that MaskCLIP can be seen as a special case of this modification, where the global attention at the end is replaced by a local attention of size 1x1, and expresses interest in how the proposed method compares to this alternative.

Author response:
**R1-3: Window Attention** Thanks for the insightful suggestion, we replace the global attention in original CLIP models with local window attention, which slightly improves the recognition ability of the dense features as shown in the following table (#1 and #2). We also train open-vocabulary detection using the window attention (the rightmost column).


|     |        Method        |  Model   |   Boxes   | Thing Masks | Stuff Masks |  OV-COCO   |
|:---:|:--------------------:|:--------:|:---------:|:-----------:|:-----------:|:----------:|
|  1  | GlobalAttn(original) | ViT-B-16 | 18.2  |  20.6 |  18.4 | 17.5  |
|  2  |      WindowAttn      | ViT-B-16 | 34.7  |  40.6 |  30.9 | 19.4  |
|  3  | GlobalAttn+CLIPSelf  | ViT-B-16 | 72.1  |  74.4 |  46.8 | 33.6  |
|  4  | WindowAttn+CLIPSelf  | ViT-B-16 | 73.3  |  74.9 |  48.6 | 33.6  |

The performance of simply replacing global attention with local window attention lags largely behind using CLIPSelf for self-distillation (#2 and #3). Moreover, we find CLIPSelf can consistently improve the CLIP model whose global attention is replaced with window attention (#2 and #4).