Original: Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision
Revised: Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Contrastive Language-Image Pretraining (CLIP) has emerged as a novel paradigm to learn visual models from language supervision.
Revised: Contrastive Language-Image Pretraining (CLIP) has emerged as a novel paradigm to learn visual models from language supervision.

Original: While researchers continue to push the frontier of CLIP, reproducing these works remains challenging.
Revised: While researchers continue to push the frontier of CLIP, reproducing these works remains challenging.

Original: This is because researchers do not choose consistent training recipes and even use different data, hampering the fair comparison between different methods.
Revised: This is because researchers do not choose consistent training recipes and even use different data, hampering the fair comparison between different methods.

Original: In this work, we propose CLIPbenchmark, a first attempt to evaluate, analyze, and benchmark CLIP and its variants.
Revised: In this work, we propose CLIPbenchmark, a first attempt to evaluate, analyze, and benchmark CLIP and its variants.

Original: We conduct a comprehensive analysis of three key factors: data, supervision, and model architecture.
Revised: We conduct a comprehensive analysis of three key factors: data, supervision, and model architecture.

Original: We find considerable intuitive or counter-intuitive insights: (1).
Revised: We find considerable intuitive or counter-intuitive insights: (1).

Original: Data quality has a significant impact on performance.
Revised: Data quality has a significant impact on performance.

Original: Certain supervision has different effects for Convolutional Networks (ConvNets) and Vision Transformers (ViT).
Revised: Certain supervision has different effects for Convolutional Networks (ConvNets) and Vision Transformers (ViT).

Original: Curtailing the text encoder reduces the training cost but not much affect the final performance.
Revised: Curtailing the text encoder reduces the training cost but not much affect the final performance.

Original: Moreover, we further combine DeCLIP (Li et al., 2021) with FILIP (Yao et al., 2021), bringing us the strongest variant DeFILIP.
Revised: Moreover, we further combine DeCLIP (Li et al., 2021) with FILIP (Yao et al., 2021), bringing us the strongest variant DeFILIP.

Original: The CLIP-benchmark would be released to the public for future CLIP research.
Revised: The CLIPbenchmark is released at: https://github. com/Sense-GVT/DeCLIP for future CLIP research.

Original: 1. Introduction
Revised: 1. Introduction

Original: Over the past few years, supervised pre-training on welllabeled ImageNet (Deng et al., 2009) and then transferred to downstream tasks (Girshick et al., 2014;
Revised: Over the past few years, supervised pre-training on welllabeled ImageNet (Deng et al., 2009) and then transferred to downstream tasks (Girshick et al., 2014;

Original: Long et al., 2015;
Revised: Long et al., 2015;

Original: Vinyals et al., 2015) has greatly transformed the computer vision (CV) community.
Revised: Vinyals et al., 2015) has greatly transformed the computer vision (CV) community.

Original: However, supervised pre-training is hard to scale since we need arduous human labeling to specify new visual concepts.
Revised: However, supervised pre-training is hard to scale since we need arduous human labeling to specify new visual concepts.

Original: More Recently, Contrastive
Revised: More Recently, Contrastive

Original: Language-Image Pretraining (CLIP) (Radford et al., 2021) has emerged as a scalable pre-training paradigm via learning visual models from language supervision, or more specifically, image-text pairs.
Revised: Language-Image Pretraining (CLIP) (Radford et al., 2021) has emerged as a scalable pre-training paradigm via learning visual models from language supervision, or more specifically, image-text pairs.

Original: Basically, CLIP adopts the contrastive loss to push the embeddings of matched image-text pairs together while pushing those of non-matched pairs apart.
Revised: Basically, CLIP adopts the contrastive loss to push the embeddings of matched image-text pairs together while pushing those of non-matched pairs apart.

Original: Benefiting from abundant image-text pairs on the
Revised: Benefiting from abundant image-text pairs on the

Original: Internet, CLIP learns general visual features that could perform zero-shot recognition, i .
Revised: Internet, CLIP learns general visual features that could perform zero-shot recognition, i .

Original: e ., predict an image’s category without seeing a single labeled example.
Revised: e ., predict an image’s category without seeing a single labeled example.

Original: CLIP’s transferable features could also be well transferred to various downstream tasks.
Revised: CLIP’s transferable features could also be well transferred to various downstream tasks.

Original: Witnessing its great success, researchers continue to push the frontier of CLIP.
Revised: Witnessing its great success, researchers continue to push the frontier of CLIP.

Original: For instance, SLIP (Mu et al., 2021),
Revised: For instance, SLIP (Mu et al., 2021),

Original: DeCLIP (Li et al., 2021) and FILIP (Yao et al., 2021) achieve considerable improvements via embracing different kinds of supervision within the image-text pairs.
Revised: DeCLIP (Li et al., 2021) and FILIP (Yao et al., 2021) achieve considerable improvements via embracing different kinds of supervision within the image-text pairs.

Original: However, it remains challenging to make fair comparison between these methods.
Revised: However, it remains challenging to make fair comparison between these methods.

Original: This is because they do not choose consistent training recipes and even use different data.
Revised: This is because they do not choose consistent training recipes and even use different data.

Original: Although
Revised: Although

Original: CLIP (Radford et al., 2021), DeCLIP (Li et al., 2021) and
Revised: CLIP (Radford et al., 2021), DeCLIP (Li et al., 2021) and

Original: SLIP (Mu et al., 2021) use the same amount of 15 million data from YFCC (Thomee et al., 2016), they adopt different filtering strategies.
Revised: SLIP (Mu et al., 2021) use the same amount of 15 million data from YFCC (Thomee et al., 2016), they adopt different filtering strategies.

Original: Moreover, methods (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Yao et al., 2021; Pham et al., 2021) crawl their datasets from the Internet, making the fair comparison more difficult.
Revised: Moreover, methods (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Yao et al., 2021; Pham et al., 2021) crawl their datasets from the Internet, making the fair comparison more difficult.

Original: This paper aims to democratize large-scale CLIP, i .
Revised: This paper aims to democratize large-scale CLIP, i .

Original: e ., to build a fair and reproducible CLIP community.
Revised: e ., to build a fair and reproducible CLIP community.

Original: We propose CLIP-benchmark, a first attempt to evaluate, analyze, and benchmark CLIP and its variants.
Revised: We propose CLIP-benchmark, a first attempt to evaluate, analyze, and benchmark CLIP and its variants.

Original: We do a comprehensive empirical study on three key factors: data, supervision, and model architecture.
Revised: We do a comprehensive empirical study on three key factors: data, supervision, and model architecture.

Original: We find considerable intuitive or counter-intuitive insights:
Revised: We find considerable intuitive or counter-intuitive insights:

Original: Data : Mid-scale 15M data is a good balance of the training cost and performance.
Revised: Data : Mid-scale 15M data is a good balance of the training cost and performance.

Original: Thus, most methods use a 15M subset from YFCC (Thomee et al., 2016) to verify the effectiveness of their methods.
Revised: Thus, most methods use a 15M subset from YFCC (Thomee et al., 2016) to verify the effectiveness of their methods.

Original: We carefully compare the current two
Revised: We carefully compare the current two

Original: YFCC15M versions, V1 from CLIP (Radford et al., 2021) and V2 from DeCLIP (Li et al., 2021).
Revised: YFCC15M versions, V1 from CLIP (Radford et al., 2021) and V2 from DeCLIP (Li et al., 2021).

Original: Interestingly, we find that in terms of the zero-shot performance on ImageNet, V is much better than V1 (details in Tab. 2).
Revised: Interestingly, we find that in terms of the zero-shot performance on ImageNet, V is much better than V1 (details in Tab. 2).

Original: We conjecture that
Revised: We conjecture that

Original: V2 includes a more meticulous filtering strategy, making its ① Img. enc. Textenc.
Revised: V2 includes a more meticulous filtering strategy, making its data quality better than V1.

Original: … 𝐿 !"

Original: " : Image Self-Sup.

Original: ①

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: ④

Original: [Figure]

Original: [Figure]

Original: ⓪

Original: [Figure]

Original: … … … … … … ⓪ … … … … … … ...

Original: ... a cute white cat.

Original: ... cute white kitty.

Original: … … … … … … ② … … … … … … … … … … … … …

Original: 𝑧 #$ 𝑧 %$ 𝑧 &$ … 𝑧 #!

Original: 𝑧 %!

Original: 𝑧 &!

Original: ̃𝑧 #$ ̃𝑧 % $ ̃𝑧 &$ ̃𝑧 &!

Original: ̃𝑧 #!

Original: ̃𝑧 %!

Original: 𝑧 #!"

Original: 𝑧 %!"

Original: 𝑧 &!"

Original: ② 𝐿 )"" : Text Self-Sup.

Original: ③ ⑤ ③ ③ ③ ④ ④ … … … … … … ⑤ ⑤

Original: [Figure]

Original: : Finer Alignment Sup.

Original: data quality better than V1.

Original: This also helps us conclude that data quality is crucial in CLIP training.
Revised: This also helps us conclude that data quality is crucial in CLIP training.

Original: Supervision : We first reproduce all the methods using a unified training recipe (details in Tab. 3).
Revised: Supervision : We first reproduce all the methods using a unified training recipe (details in Tab. 3).

Original: We find that fine-grained alignment supervision (Yao et al., 2021) could benefit ViT image encoder but hurts ConvNets.
Revised: We find that fine-grained alignment supervision (Yao et al., 2021) could benefit ViT image encoder but hurts ConvNets.

Original: Intuitively, fine-grained alignment needs the image features to be nonoverlapped, which is unachievable for ConvNets.
Revised: Intuitively, fine-grained alignment needs the image features to be nonoverlapped, which is unachievable for ConvNets.

Original: For ViT image encoder, aggregating self-supervision (Mu et al., 2021; Li et al., 2021), multi-view supervision (Li et al., 2021), nearest-neighbor supervision (Li et al., 2021) and fine-grained alignment supervision (Yao et al., 2021) brings us the strongest variant DeFILIP.
Revised: For ViT image encoder, aggregating self-supervision (Mu et al., 2021; Li et al., 2021), multi-view supervision (Li et al., 2021), nearest-neighbor supervision (Li et al., 2021) and fine-grained alignment supervision (Yao et al., 2021) brings us the strongest variant DeFILIP.

Original: Model : While most attention is paid to image encoders, little research is conducted on text encoders.
Revised: Model : While most attention is paid to image encoders, little research is conducted on text encoders.

Original: Most literature follows the exact setting from CLIP, i .
Revised: Most literature follows the exact setting from CLIP, i .

Original: e ., a 12-layer transformer (Radford et al., 2019).
Revised: e ., a 12-layer transformer (Radford et al., 2019).

Original: We find that CLIP’s text encoder is not necessary to be so much deep; a 3-layer transformer performs even better than the default 12-layer setting under the mid-scale data scenarios (details in Tab. 4).
Revised: We find that CLIP’s text encoder is not necessary to be so much deep; a 3-layer transformer performs even better than the default 12-layer setting under the mid-scale data scenarios (details in Tab. 4).

Original: Therefore, pay attention to the text encoder when designing your CLIP models.
Revised: Therefore, pay attention to the text encoder when designing your CLIP models.

Original: In a nutshell, this paper proposes the first CLIP-benchmark that includes the state-of-the-art methods.
Revised: In a nutshell, this paper proposes the first CLIP-benchmark that includes the state-of-the-art methods.

Original: We benchmark these methods under the same training recipe using the same data.
Revised: We benchmark these methods under the same training recipe using the same data.

Original: Our CLIP-benchmark also brings some insights about data, supervision and model.
Revised: Our CLIP-benchmark also brings some insights about data, supervision and model.

Original: The CLIP-benchmark would be released to the public for future research.
Revised: The CLIP-benchmark would be released to the public for future research.

Original: 2. Methods
Revised: 2. Related Work

Revised: Concurrently to this work, many researchers continue to push the frontier of CLIP (Radford et al., 2021).

Revised: SLIP (Mu et al., 2021) introduces self-supervision to Contrastive

Revised: Language-Image Pretraining.

Revised: DeCLIP (Li et al., 2021) utilizes widespread supervision among the image-text pairs.

Revised: FILIP (Yao et al., 2021) leverages the finer-grained alignment between image patches and textual words.

Revised: LiT (Zhai et al., 2021) adopt contrastive-tuning to tune the text tower using image-text data while using a pre-trained, strong image model as the image tower.

Revised: OTTER (Wu et al., 2021) uses online entropic optimal transport to find a soft imagetext match as labels for contrastive learning.

Revised: The representations learned by CLIP have shown excellent transferability over various tasks.

Revised: CLIP2Video (Fang et al., 2021) and CLIP4Clip (Luo et al., 2021) apply CLIP to video retrieval task.

Revised: ActionCLIP (Wang et al., 2021) utilizes CLIP for action recognition task.

Revised: More works about improving image captioning with CLIP, e .

Revised: g .CLIPCap

Revised: (Mokady et al., 2021), CLIP4Caption (Tang et al., 2021).

Revised: Interestingly

Revised: CLIP can be even used in text-guided image generation task ( StyleCLIP (Patashnik et al., 2021)) and Embodied

Revised: AI ( EmbCLIP (Khandelwal et al., 2021)).

Revised: CLIP has also contributed to the development of general vision (Shao et al., 2021).

Revised: Witnessing CLIP’s active community and wide applications, we propose the first work to benchmark CLIP.

Original: CLIP (Radford et al., 2021) and its variants ( e . g ., DeCLIP (Li et al., 2021), SLIP (Mu et al., 2021), and
Revised: CLIP (Radford et al., 2021) and its variants ( e . g ., DeCLIP (Li et al., 2021), SLIP (Mu et al., 2021), and

Original: FILIP (Yao et al., 2021)) follow a common high-level structure(see Fig. 1).
Revised: FILIP (Yao et al., 2021)) follow a common high-level structure(see Fig. 1).

Original: The model consists of an image encoder ( e .
Revised: The model consists of an image encoder ( e .

Original: g .
Revised: g .

Original: , ResNet (He et al., 2016) or ViT (Dosovitskiy et al., 2020)) and a text encoder( e .
Revised: , ResNet (He et al., 2016) or ViT (Dosovitskiy et al., 2020)) and a text encoder( e .

Original: g ., transformer (Vaswani et al., 2017)), with a multimodal interaction at the top.
Revised: g ., transformer (Vaswani et al., 2017)), with a multimodal interaction at the top.

Original: Take the most straightforward CLIP as an example, the image encoder (the text encoder) extracts the image embedding (the text embedding) based on the input image-text pair.
Revised: Take the most straightforward CLIP as an example, the image encoder (the text encoder) extracts the image embedding (the text embedding) based on the input image-text pair.

Original: A contrastive objective is used to push the embeddings of matched image-text pairs together while pushing non-matched pairs apart.
Revised: A contrastive objective is used to push the embeddings of matched image-text pairs together while pushing non-matched pairs apart.

Original: At the test phase, the learned text encoder synthesizes a zero-shot linear classifier by embedding the arbitrary categories of the test dataset.
Revised: At the test phase, the learned text encoder synthesizes a zero-shot linear classifier by embedding the arbitrary categories of the test dataset.

Original: Because it is rare in the dataset that image caption is just a single word, CLIP uses prompts to make up the context of the category { label } , such as "a photo of a { label } " .
Revised: Because it is rare in the dataset that image caption is just a single word, CLIPs use prompts to make up the context of the category { label } , such as "a photo of a { label } " .

Original: As shown in the Fig.1, different variants further explore the widespread supervised signal of the image-text pair for better visual representations.
Revised: As shown in the Fig.1, different variants further explore the widespread supervised signal of the image-text pair for better visual representations.

Original: This section will briefly introduce the above CLIP variants and bring the strongest variant DeFILIP. 
Revised: This section will briefly introduce the above CLIP variants and bring the strongest variant DeFILIP. 3.1. CLIP

Original: CLIP (Radford et al., 2021) only uses the original imagetext supervsion.
Revised: CLIP (Radford et al., 2021) only uses the original image-text supervsion.

Original: In a batch of N image-text pairs { ( x Ii , x Ti ) } , we denote x Ii and x Ti as image and text of the i th pair.
Revised: In a batch of N image-text pairs { ( x Ii , x Ti ) } , we denote x Ii and x Ti as image and text of the i th pair.

Original: Let z Ii and z
Revised: Let z Ii and z

Original: Tj be the normalized embedding of the i th image and j th text, respectively.
Revised: Tj be the normalized embedding of the i th image and j th text, respectively.

Original: CLIP uses InfoNCE loss (Van den Oord et al., 2018).
Revised: CLIP uses InfoNCE loss (Van den Oord et al., 2018).

Original: The loss for the image encoder can be denoted as Eq. 1.
Revised: The loss for the image encoder can be denoted as Eq. 1.

Original: [Equation]
Revised: [Equation]

Original: Here, the similarity function sim( , ) is measured by dot product, and τ is a learnable temperature variable to scale the logits.
Revised: Here, the similarity function sim( , ) is measured by dot product, and τ is a learnable temperature variable to scale the logits.

Original: We have a symmetrical loss for image and text encoder; thus, the overall loss function L CLIP is the average of L I and L T . 
Revised: We have a symmetrical loss for image and text encoder; thus, the overall loss function L CLIP is the average of L I and L T . ① Img. enc. Textenc.

Original: SLIP (Mu et al., 2021) introduces self-supervision to CLIP for better visual representations.
Revised: … 𝐿 !" " : Image Self-Sup. ① 𝐿 #$" : Multi-View Sup. 𝐿 %%" : Nearest-Neighbor Sup. ④

Revised: Feature Queue … 𝑧 !"

Revised: = 𝑁𝑁(𝑧 ! )

Revised: ⓪ 𝐿 &’!

Revised: ( : Original Img-text Sup .

Revised: … … … … … … ⓪ … … … … … … ...

Revised: ... a cute white cat.

Revised: ... cute white kitty.

Revised: … … … … … … ② … … … … … … … … … … … … …

Revised: 𝑧 #$ 𝑧 %$ 𝑧 &$ … 𝑧 #!

Revised: 𝑧 %!

Revised: 𝑧 &!

Revised: ̃𝑧 #$ ̃𝑧 % $ ̃𝑧 &$ ̃𝑧 &!

Revised: ̃𝑧 #!

Revised: ̃𝑧 %!

Revised: 𝑧 #!"

Revised: 𝑧 %!"

Revised: 𝑧 &!"

Revised: ② 𝐿 )"" : Text Self-Sup.

Revised: ③ ⑤ ③ ③ ③ ④ ④ … … … … … … ⑤ ⑤ … 𝑧 !,!

Revised: # 𝑧 !,$# 𝑧 !

Revised: %# … 𝑧 !,!& 𝑧 !,$& 𝑧 !,%& a c u t e w h

Revised: i t e c a t 𝐿 +," : Finer Alignment Sup.

Original:  Built upon CLIP, SLIP gets two more strong augmented views for image self-supervised contrastive loss L ISS .
Revised: [Equation] SLIP SLIP (Mu et al., 2021) introduces self-supervision to CLIP for better visual representations. Built upon CLIP, SLIP gets two more strong augmented views for image self-supervised contrastive loss L ISS .

Original: SLIP further compares different image self-supervised methods and finally selected SimCLR (Chen et al., 2020) for the final framework. 
Revised: SLIP further compares different image self-supervised methods and finally selected SimCLR (Chen et al., 2020) for the final framework. The overall loss function of SLIP is shown in Eq.

Original: FILIP (Yao et al., 2021) achieves finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
Revised: α is the scale of self-supervision and is set to 1. [Equation]

Original: FILIP  perform finer-grained alignment supervision on token level rather than image-text level by L FAS .
Revised: FILIP (Yao et al., 2021) perform finer-grained alignment supervision on token level rather than image-text level.

Original: In the L FAS , the similarity { sim ( z Ii , Ti ) } of the i th image and j th textin is improved to token-wise maximum similarity.
Revised: The similarity { sim ( z Ii , Ti ) } of the i th image and j th text is improved to token-wise maximum similarity which is calculated as: [Equation] Where m Ik = argmax 0 <r<n 2 z Ii, k z

Revised: Tj,r and m Tk = argmax 0 <r<n 1 z Ii,r z Tj,k .

Original: DeCLIP (Li et al., 2021) utilizes widespread supervision among  the image-text pairs, including Self-Supervision(SS),
Revised: FILIP achieves finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. Though the late cross-modal interaction can capture finergrained features, it relies on the token-wise representations of both modalities and can be inefficient in terms of communication, memory, and computation. To alleviate this problem, the authors carefully reduce the precision and embedding size of the model and further select the 25% tokens with the highest token-wise maximum similarity score among all texts ( resp , images) in the same local worker before node communication. Denoting the loss of fine-grained alignment supervision as L FAS , The overall loss function of FILIP is shown in Eq.

Revised: [Equation]

Revised: 3.4. DeCLIP

Revised: DeCLIP (Li et al., 2021) utilizes widespread supervision among the image-text pairs, including Self-Supervision(SS),

Original: Multi-View Supervision(MVS), and Nearest-Neighbor Supervision(NNS).
Revised: Multi-View Supervision(MVS), and Nearest-Neighbor Supervision(NNS).

Original: DeCLIP contains image SS and text SS:
Revised: DeCLIP contains image SS and text SS:

Original: Image SS maximizes the similarity between two augmented views of the same instance while text SS leverages Masked
Revised: Image SS maximizes the similarity between two augmented views of the same instance while text SS leverages Masked

Original: Language Modeling(MLM) within a text sentence.
Revised: Language Modeling(MLM) within a text sentence.

Original: For
Revised: For

Original: MVS, DeCLIP has two augmented views of both image and text, then contrasts the 2 × 2 image-text pairs.
Revised: MVS, DeCLIP has two augmented views of both image and text, then contrasts the 2 × 2 image-text pairs.

Original: For NNS, 
Revised: For NNS, De-

Original: DeCLIP sample text Nearest-Neighbor in the embedding space as additional supervision.
Revised: CLIP sample text NN in the embedding space as additional supervision.

Original: In summary, DeCLIP denote L ISS and L TSS as the loss function of image SS and text SS, respectively.
Revised: In summary, DeCLIP denote L ISS and L TSS as the loss function of image SS and text SS, respectively.

Original: L MV S is multi-view loss, and L NNS is nearest-neighbor loss.
Revised: L MV S is multi-view loss, and L NNS is nearest-neighbor loss.

Revised: The overall loss function of DeCLIP is shown in Eq.

Revised: α, β, γ are the loss scales and are both set to 0.2.

Revised: [Equation]

Original:  DeFILIP.
Revised: 3.5. DeFILIP

Original: By introducing the above methods, we can find a large number of possible supervision signals in the imagetext pairs, which can improve the efficiency of training and generalization ability.
Revised: By introducing the above methods, we can find a large number of possible supervision signals in the image-text pairs, which can improve the efficiency of training and generalization ability.

Original: In order to learn better visual representations and improve the data efficiency of the model, we further combine DeCLIP (Li et al., 2021) with FILIP (Yao et al., 2021), bringing us the strongest variant DeFILIP. 
Revised: In order to learn better visual representations and improve the data efficiency of the model, we further combine DeCLIP (Li et al., 2021) with FILIP (Yao et al., 2021), bringing us the strongest variant DeFILIP. The overall loss function of DeFILIP is shown in Eq.

Original:  L FAS is applied to improve fine-grained learning of visual representations further.
Revised: [Equation] L FAS is applied to improve fine-grained learning of visual representations further.

Revised: The loss weight λ is set to 0.2 in this work.

Original: As shown in fig.
Revised: As shown in fig.

Original: 1, our DeFILIP is a summary and development of the existing SOTA methods, which applies the existing supervision and achieves a new state-of-the-art performance.[Table]
Revised: 1, our DeFILIP is a summary and development of the existing SOTA methods, which applies the existing supervision and achieves a new state-of-the-art performance.

Original: 3. CLIP-Benchmark 3.1. Setup
Revised: 4. CLIP-Benchmark 4.1. Setup

Original: Evaluation Metric:
Revised: Evaluation Metric

Original: In this paper, we mainly evaluate zeroshot performance of different models on ImageNet (Deng et al., 2009), which is regarded as the main feature of
Revised: In this paper, we mainly evaluate zeroshot performance of different models on ImageNet (Deng et al., 2009), which is regarded as the main feature of

Original: CLIP methods.
Revised: CLIP methods.

Original: We perform prompt ensemble by averaging the caption embeddings for each class across the prompt templates.
Revised: We perform prompt ensemble by averaging the caption embeddings for each class across the prompt templates.

Original: The prompts are the same as proposed in
Revised: The prompts are the same as proposed in

Original: CLIP(Radford et al., 2021).
Revised: CLIP(Radford et al., 2021).

Original: Implementation details: The models in this work are trained and tested in the same codebase.
Revised: Implementation details The models in this work are trained and tested in the same codebase.

Revised: Unless otherwise specified, all models are realized with Pytorch, and are trained with 32 NVIDIA A100 GPUs.

Original: When pretraining, we use an AdamW optimizer (Loshchilov & Hutter, 2017) with a total batch size of 4,096 (single GPU batch size 128 with 32 NVIDIA A100 GPUs).
Revised: When pretraining, we use an AdamW optimizer (Loshchilov & Hutter, 2017) with a total batch size of 4,096 (single GPU batch size [Table] [Table] with 32 NVIDIA A100 GPUs).

Original: Starting with a learning rate (LR) of 0.0001, we linearly warm-yp the LR to 0.
Revised: Starting with a learning rate (LR) of 0.0001, we linearly warm-yp the LR to 0.

Original: in one epoch, and then we use the cosine anneal LR decay strategy (Loshchilov & Hutter, 2016) to decrease the LR.
Revised: in one epoch, and then we use the cosine anneal LR decay strategy (Loshchilov & Hutter, 2016) to decrease the LR.

Original: The weight decay rate is set to 0.1.
Revised: The weight decay rate is set to 0.1.

Original: The input resolution of the image encoder is 224 × 224, and the maximum context length of the text encoder is 76.
Revised: The input resolution of the image encoder is 224 × 224, and the maximum context length of the text encoder is 76.

Original: The learnable temperature parameter τ in Eq.1 is initialized to 0.07.
Revised: The learnable temperature parameter τ in Eq.1 is initialized to 0.07.

Original: All models are trained from scratch for 32 epochs.
Revised: All models are trained from scratch for 32 epochs.

Original: 3.2. Data
Revised: 4.2. Data

Original: Data is a crucial part of CLIP.
Revised: Data is a crucial part of CLIP.

Original: This section does a holistic study of two mid-scale YFCC15M versions.
Revised: This section does a holistic study of two mid-scale YFCC15M versions.

Original: V1 from
Revised: V1 from

Original: CLIP (Radford et al., 2021) and V2 from DeCLIP (Li et al., 2021).
Revised: CLIP (Radford et al., 2021) and V2 from DeCLIP (Li et al., 2021).

Original: Data statistics: We present statistics of two versions
Revised: Data statistics We present statistics of two versions

Original: YFCC15M on examples number, mean/std of caption length, mean English word ratio, and the vocabulary size (unique tokens) in Table 1.
Revised: YFCC15M on examples number, mean/std of caption length, mean English word ratio, and the vocabulary size (unique tokens) in Table 1.

Original: The V2 consists of 15.4M image-text pairs, 0.6M(3%) more than V1.
Revised: The V2 consists of 15.4M image-text pairs, 0.6M(3%) more than V1.

Original: V2 is generally shorter and more evenly distributed than V1 regarding the caption length.
Revised: V2 is generally shorter and more evenly distributed than V1 regarding the caption length.

Original: The English word ratio ( i .
Revised: The English word ratio ( i .

Original: e ., # of English words divided by # of all words) of V2 is about 0.92, which is significantly better than V1’s 0.72.
Revised: e ., # of English words divided by # of all words) of V2 is about 0.92, which is significantly better than V1’s 0.72.

Original: For the vocabulary size (unique tokens), V1 is one order larger than V2 mainly because V1 contains many non-English characters.
Revised: For the vocabulary size (unique tokens), V1 is one order larger than V2 mainly because V1 contains many non-English characters.

Original: We can infer from these statistics that V2 has better quality than
Revised: We can infer from these statistics that V2 has better quality than

Original: V1 because V2 is more evenly distributed and has fewer non-English characters.
Revised: V1 because V2 is more evenly distributed and has fewer non-English characters.

Original: We believe that V2 includes a more

Original: [Table]

Original: [Table] [Table] meticulous filtering strategy, making its data quality better than V1.
Revised: We believe that V2 includes a more meticulous filtering strategy, making its data quality better than V1.

Original: Performance over V1-V2: To further evaluate the quality of the two YFCC15M versions and explore the impact of data quality on CLIP, we perform a comparison with
Revised: Performance over V1-V2 . To further evaluate the quality of the two YFCC15M versions and explore the impact of data quality on CLIP, we perform a comparison with

Original: V1 (Radford et al., 2021) and V2 (Li et al., 2021), using the same methodology.
Revised: V1 (Radford et al., 2021) and V2 (Li et al., 2021), using the same methodology.

Original: As shown in Tab. 2, training with V  leads to a better zero-shot performance than V1 under the same experimental setup.
Revised: As shown in Tab. 2, training with V [Table] leads to a better zero-shot performance than V1 under the same experimental setup.

Original: On the one hand, it proves that the data quality of V2 is better regarding final performance.
Revised: On the one hand, it proves that the data quality of V2 is better regarding final performance.

Original: On the other hand, it also proves that data quality significantly impacts the performance of CLIP methods.
Revised: On the other hand, it also proves that data quality significantly impacts the performance of CLIP methods.

Original: 3.3. Supervision
Revised: 4.3. Supervision

Original: We perform a comprehensive comparison of our reimplemented pretraining methods (Radford et al., 2021;
Revised: We perform a comprehensive comparison of our reimplemented pretraining methods (Radford et al., 2021;

Original: Li et al., 2021; Mu et al., 2021; Yao et al., 2021) to benchmark these methods under the same training recipe.
Revised: Li et al., 2021; Mu et al., 2021; Yao et al., 2021) to benchmark these methods under the same training recipe.

Original: We report the zero-shot top-1 accuracy on ImageNet in Tab.
Revised: We report the zero-shot top-1 accuracy on ImageNet in Tab.

Original: When the image encoder is ViT, all supervision is proved to be effective.
Revised: When the image encoder is ViT, all supervision is proved to be effective.

Original: DeCLIP, which utilizes the maximum supervision, obtains the best results.
Revised: DeCLIP, which utilizes the maximum supervision, obtains the best results.

Original: Moreover, we further integrate the existing supervision to make the strongest variant, named DeFILIP.
Revised: Moreover, we further integrate the existing supervision to make the strongest variant, named DeFILIP.

Original: Our proposed DeFILIP reaches 45.0% accuracy, surpassing the CLIP baseline by a considerable 12.2% margin.
Revised: Our proposed DeFILIP reaches 45.0% accuracy, surpassing the CLIP baseline by a considerable 12.2% margin.

Original: When we use ResNet as the image encoder, some methods seem cannot preserve the improvement.
Revised: When we use ResNet as the image encoder, some methods seem cannot preserve the improvement.

Original: Worth mentioning,
Revised: Worth mentioning,

Original: SLIP (Mu et al., 2021) and FILIP (Yao et al., 2021) do not report the results of ResNet models.
Revised: SLIP (Mu et al., 2021) and FILIP (Yao et al., 2021) do not report the results of ResNet models.

Original: We conjecture there[Table] are two reasons: 1) ResNet models might need more dedicated hyper parameter tuning.
Revised: We conjecture there are two reasons: 1) ResNet models might need more dedicated hyper parameter tuning.

Original: Fine-grained alignment requires the image features to be non-overlapped, which is unachievable for ConvNets.
Revised: Fine-grained alignment requires the image features to be non-overlapped, which is unachievable for ConvNets.

Original: However, DeCLIP can still brings 7.2% improvement over the CLIP baseline.
Revised: However, DeCLIP can still brings 7.2% improvement over the CLIP baseline.

Original: 3.4. Model
Revised: 4.4. Model

Original: While most attention is paid to image encoders, little research is conducted on text encoders.
Revised: While most attention is paid to image encoders, little research is conducted on text encoders.

Original: Most literature follows the exact setting from CLIP, i .
Revised: Most literature follows the exact setting from CLIP, i .

Original: e ., a 12-layer transformer.
Revised: e ., a 12-layer transformer.

Original: Therefore, we expect to study the role of the text encoder, and further explore whether the training efficiency can be improved by reducing the parameters of the text encoder without affecting the performance.
Revised: Therefore, we expect to study the role of the text encoder, and further explore whether the training efficiency can be improved by reducing the parameters of the text encoder without affecting the performance.

Original:  As shown in Tab. 4, we try 1/3/6/12-layer transformer for
Revised: [Table] As shown in Tab. 4, we try 1/3/6/12-layer transformer for

Original: CLIP-ViTB32 and DeFILIP-ViTB32.
Revised: CLIP-ViTB32 and DeFILIP-ViTB32.

Original: Surprisingly, we find that (1) For the primitive CLIP method, text encoders with 6 layers of transformers achieve the best results instead of the default 12 layers.
Revised: Surprisingly, we find that (1) For the primitive CLIP method, text encoders with 6 layers of transformers achieve the best results instead of the default 12 layers.

Original: A 3-layers transformer is enough to achieve high results.
Revised: A 3-layers transformer is enough to achieve high results.

Original: (2) For the DeCLIP, which applies more supervision, the text encoder is more critical.
Revised: (2) For the DeCLIP, which applies more supervision, the text encoder is more critical.

Original: However, even if half the number of layers, it does not significantly affect the final performance.
Revised: However, even if half the number of layers, it does not significantly affect the final performance.

Original: Such an exciting result shows that curtailing the text-encoder is an efficient approach to reducing training costs.
Revised: Such an exciting result shows that curtailing the text-encoder is an efficient approach to reducing training costs.

Original: 4. Conclusions
Revised: 5. Conclusions

Original: In this paper, we propose the first CLIP-benchmark that includes state-of-the-art methods.
Revised: In this paper, we propose the first CLIP-benchmark that includes state-of-the-art methods.

Original: We benchmark these methods under the same training recipe using the same data.
Revised: We benchmark these methods under the same training recipe using the same data.

Original: Our CLIP-benchmark also brings some insights about data, supervision, and model.
Revised: Our CLIP-benchmark also brings some insights about data, supervision, and model.

Original: Moreover, we further propose DeFILIP to make a stronger baseline for this task.
Revised: Moreover, we further propose DeFILIP to make a stronger baseline for this task.

Original: The CLIPbenchmark would be released to the public for future research.
Revised: The CLIPbenchmark would be released to the public for future research.

Original: We hope this technical report could avoid duplicate data cleaning efforts and provide a consistent benchmark to facilitate fair comparisons.
Revised: We hope this technical report could avoid duplicate data cleaning efforts and provide a consistent benchmark to facilitate fair comparisons