Original: Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching
Revised: Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Despite surprising performance on zero-shot transfer, pre-training a large-scalemultimodal model is often prohibitive as it requires a huge amount of data andcomputing resources.
Revised: Despite surprising performance on zero-shot transfer, pre-training a large-scale multimodal model is often prohibitive as it requires a huge amount of data and computing resources.

Original: In this paper, we propose a method ( BeamCLIP ) that caneffectively transfer the representations of a large pre-trained multimodal model(CLIP-ViT) into a small target model (e.g., ResNet-18).
Revised: In this paper, we propose a method ( BeamCLIP ) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP-ViT) into a small target model (e.g., ResNet-18).

Original: For unsupervised transfer,we introduce cross-modal similarity matching (CSM) that enables a student modelto learn the representations of a teacher model by matching the relative similaritydistribution across text prompt embeddings.
Revised: For unsupervised transfer, we introduce cross-modal similarity matching (CSM) that enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings.

Original: To better encode the text prompts,we design context-based prompt augmentation (CPA) that can alleviate the lexicalambiguity of input text prompts.
Revised: To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input text prompts.

Original: Our experiments show that unsupervised representation transfer of a pre-trained vision-language model enables a small ResNet-18 toachieve a better ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-onlyself-supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%),while closing the gap with supervised learning (69.8%).
Revised: Our experiments show that unsupervised representation transfer of a pre-trained vision-language model enables a small ResNet-18 to achieve a better ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-only self-supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%), while closing the gap with supervised learning (69.8%).

Original:  Introduction
Revised:  Introduction

Original: Learning transferable representations is crucialfor successful downstream applications.
Revised: Learning transferable representations is crucial for successful downstream tasks.

Original: Contrastive learning such as SimCLR  and MoCo-v2have shown notable success by forcing featuresof individual classes to be clustered and sufficiently scattered [41].
Revised: Contrastive learning such as SimCLR [4] and MoCo-v2 [6] have shown notable success by forcing features of individual classes to be clustered and sufficiently scattered [41].

Original: But their downstreamperformances are still far behind the supervisedtraining as shown in Figure 1.
Revised: But their linear probe performances are still far behind the supervised learning as shown in Figure 1.

Original: Recently, largescale vision and language pre-trained (VLP)models provide highly transferable visual representations via language supervision.
Revised: Recently, largescale vision and language pre-trained (VLP) models provide highly transferable visual representations via language supervision.

Original: However,learning VLP models fromthe scratch is prohibitive as it requires large amounts of training data and computing resource.
Revised: However, learning VLP models from scratch is prohibitive as it requires large amounts of training data and computing resources.

Original: For example, training CLIP [32] requires 400M pairedimage-text data and several hundreds of GPUs.
Revised: For example, training CLIP [32] requires 400M paired image-text data and several hundreds of GPUs.

Original: ALIGN [22] further scales up to leverage alternative texts specified for descriptions of webimages.
Revised: ALIGN [22] further scales up to leverage alternative texts specified for descriptions of web images.

Original: While these models are often based onlarge Transformers [40], small ConvNets suchas ResNet-50 [17] and MobileNet [20] are still widely used in practice [1] and even more crucial for low-resource  environments.
Revised: While these models are often based on large Transformers [40], small ConvNets such as ResNet-50 [17] and MobileNet [20] are still widely used in practice [1] and even more crucial for low-resource 36th Conference on Neural Information Processing Systems (NeurIPS 2022). environments.

Original: We reformulate representation learning in terms of knowledge transferfrom a large pre-trained model to a small practical model.
Revised: We reformulate representation learning in terms of knowledge transfer from a large pre-trained model to a small practical model.

Original: Large pre-trained models exhibit strong alignments between different modalities.
Revised: Large-scale vision-language pre-trained models exhibit strong alignments between different modalities.

Original: CLIP  learns visualconcepts from natural language supervision, mapping image and text into the same vector space.
Revised: CLIP [32] learns visual concepts from natural language supervision, mapping image and text into the same vector space.

Original: Astheir training data is not only huge but inaccessible, however, conventional knowledge distillationbased on instance similarity is no longer a viable option.
Revised: As their training data is not only huge but inaccessible, however, conventional knowledge distillation [19] based on the source training data is no longer a viable option.

Original: Instead, we propose cross-modal similaritymatching (CSM).
Revised: Instead, we propose cross-modal similarity matching (CSM).

Original: Imagine your goal is to learn a high quality representation for an input dog imagein CIFAR-10  as in Figure 2.
Revised: Imagine your goal is to learn a high quality representation for an input dog image in CIFAR-10 [23] as in Figure 2.

Original: Since CLIP was trained on numerous image-caption pairs, angulardistances from the dog image embedding to the caption embeddings of other anchor prompt textssuch as “a photo of cat” or “a photo of horse” must comprehensively preserve their visual differences.
Revised: Since CLIP was trained on numerous image-caption pairs, angular distances from the dog image embedding to the caption embeddings of other anchor prompt texts such as "A photo of cat" or "A photo of horse" must comprehensively preserve their visual differences.

Original: By training the student to preserve angular relations witnessed from the teacher, our model achievesnear benchmark performance without accessing the original data for training CLIP.
Revised: By training the student to preserve angular relations witnessed from the teacher, our model achieves near benchmark performance without accessing the original data for training CLIP.

Original: To measure the effects of different amounts of supervisions, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of the input text prompts.
Revised: To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of the input text prompts.

Original: We find that lexicalambiguity in prompt texts can lead to semantically incorrect text embeddings.
Revised: We find that lexical ambiguity in prompt texts can lead to semantically incorrect text embeddings.

Original: This may result inunexpected discrepancies of image-text alignment in the teacher’s embedding space.
Revised: This may result in unexpected discrepancies of image-text alignment in the teacher’s embedding space.

Original: Also, it is knownthat the zero-shot performance of CLIP can be improved by designing task-specific prompt texts.
Revised: Also, it is known that the zero-shot performance of CLIP can be improved by designing task-specific prompt texts.

Original: Inspired by this, we design CPA that extends the basic prompt of CLIP to better encode prototypicalanchor representations.
Revised: Inspired by this, we design CPA that extends the basic prompt of CLIP to better encode prototypical anchor representations.

Original: Our experimental result shows that our BeamCLIP  achieves the strongest and near benchmarkperformance on ImageNet-1K Top-1 linear probe accuracy when using most popular  ResNet-50 asthe student network.
Revised: Our experimental results show that the BeamCLIP ("beam" means to transmit) achieves the strongest and near benchmark performance on ImageNet-1K [10] top-1 linear probe accuracy when using most popular ResNet-18 and ResNet-50 as the student network.

Original: We also compare the effectiveness of  BeamCLIP against zero-shot transferlearning.
Revised: We also compare the effectiveness of the BeamCLIP against zero-shot transfer learning.

Original: Further, we provide ablation study results to show how much each component contribute tothe performance.
Revised: Further, we provide ablation study results to show how much each component contributes to the performance.

Original: Contributions of this paper can be summarized as follows:
Revised: Contributions of this paper can be summarized as follows:

Original: • We propose a method ( BeamCLIP ) that can effectively transfer the representations of a largepre-trained multimodal model (e.g., CLIP-ViT) into a small target model (e.g., ResNet-50or ResNet-18).
Revised: • We propose a method ( BeamCLIP ) that can effectively transfer the representations of a large pre-trained multimodal model (e.g., CLIP-ViT) into a small target model (e.g., ResNetor ResNet-50).

Original: To achieve this, we introduce cross-modal similarity matching (CSM) and context-based prompt augmentation (CPA).
Revised: To achieve this, we introduce cross-modal similarity matching (CSM) and context-based prompt augmentation (CPA).

Original: (Figure 2).
Revised: (Figure 2).

Original: • We empirically show that BeamCLIP enables a small target model (e.g., ResNet-18) to achieve a better ImageNet linear probe accuracy than vision-only self-supervised learning (SSL) methods, by effectively transferring CLIP-ViT representations.
Revised: • We empirically show that BeamCLIP enables a small target model (e.g., ResNet-18) to achieve a better ImageNet-1K linear probe accuracy than vision-only self-supervised learning (SSL) methods, by effectively transferring CLIP-ViT representations.

Original: (Figure 1, Table 2,and Table 3.
Revised: (Figure 1, Table 2, and Table 3.

Original: • We also explore the zero-shot capability of the BeamCLIP (Table 5) and analyze the effectiveness of the BeamCLIP on various target datasets (Table 6 and Table 7)
Revised: • We also explore the zero-shot capability of the BeamCLIP (Table 5) and analyze the effectiveness of the BeamCLIP on various target datasets (Table 6 and Table 7)

Original:  Related Work
Revised:  Related Work

Original: Vision and language pre-trainig.
Revised: Vision and language pre-trainig.

Original: Vision and language pre-training (VLP) aims to jointly learnvision and language representations that can be transferred to the downstream tasks such as visualquestion answering (VQA), image captioning, and vision and language navigation (VLN).
Revised: Vision and language pre-training (VLP) aims to jointly learn vision and language representations that can be transferred to the downstream tasks such as visual question answering (VQA), image captioning, and vision and language navigation (VLN).

Original: There are
Revised: There are

Original: BERT-based vision and language models such as VLBERT [35], ViLBERT [26], and UNITER [7].
Revised: BERT-based vision and language models such as VLBERT [35], ViLBERT [26], and UNITER [7].

Original: Also, there are contrastive learning-based models such as CLIP [32] and ALIGN [22].
Revised: Also, there are contrastive learning-based models such as CLIP [32] and ALIGN [22].

Original: These modelsuse contrastive loss [29] to learn aligned vision and language representations by performing a task ofmatching a large-scale image and text pairs.
Revised: These models use contrastive loss [30] to learn aligned vision and language representations by performing a task of matching a large-scale image and text pairs.

Original: The BeamCLIP aims to transfer the rich representationsof large-scale vision and language pre-trained models such as CLIP and ALIGN to a small targetmodel.
Revised: The BeamCLIP aims to transfer the rich representations of large-scale vision and language pre-trained models such as CLIP and ALIGN to a small target model.

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: Self-supervised learning (SSL) aims to learn highly transferable representations by using unlabeled data.
Revised: Self-supervised learning (SSL) aims to learn highly transferable representations by using unlabeled data.

Original: In computer vision, at the early stage, task-specific self-supervisedmethods were introduced.
Revised: In computer vision, at the early stage, task-specific self-supervised methods were introduced.

Original: These include Context Prediction [11], Rotation Prediction [14], and Colorization [43].
Revised: These include Context Prediction [11], Rotation Prediction [14], and Colorization [43].

Original: More recently, contrastive learning-based methods were introduced as a task-agnosticapproach.
Revised: More recently, contrastive learning-based methods were introduced as a task-agnostic approach.

Original: These include SimCLR [4] and MoCo-v2 [18].
Revised: These include SimCLR [4] and MoCo-v2 [6].

Original: However, since contrastive self-supervisedmethods require a large batch size, non-contrastive methods have been introduced.
Revised: However, since contrastive self-supervised methods require a large batch size, non-contrastive methods have been introduced.

Original: These include
Revised: These include

Original: SwAV [3], BYOL [16], and SimSiam [5].
Revised: SwAV [3], BYOL [16], and SimSiam [5].

Original: In this paper, we empirically show that the BeamCLIP can provide better visual representations than the state-of-the-art SSL methods by leveraging a large-scalepre-trained multimodal model.
Revised: In this paper, we empirically show that the BeamCLIP can provide better visual representations than the state-of-the-art SSL methods by leveraging a large-scale pre-trained multimodal model.

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [19] aims to transfer rich knowledge from astrong teacher model to a target student model.
Revised: [19] aims to transfer rich knowledge from a strong teacher model to a target student model.

Original: In a conventional setting, it encourages the studentmodel to mimic the task-specific prediction of the teacher model.
Revised: In a conventional setting, it encourages the student model to mimic the task-specific prediction of the teacher model.

Original: As the student model is trainedto predict the same probability distribution over pre-defined classes as the teacher model’s, using
Revised: As the student model is trained to predict the same probability distribution over pre-defined classes as the teacher model’s, using

Original: Kullback-Leibler (KL) divergence is a natural metric to measure the error between the two models.
Revised: Kullback-Leibler (KL) divergence is a natural metric to measure the error between the two models.

Original: For a classification task, the loss function can be formulated as follows:
Revised: For a classification task, the loss function can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: The first term indicates the supervised loss, where p i denotes the one-hot labels and H ( p, q ) denotescross-entropy.
Revised: The first term indicates the supervised loss, where p i denotes the one-hot labels and H ( p, q ) denotes cross-entropy.

Original: The second term is the distillation loss, where p Ti and p Si are the softmax predictionsof the teacher and student models, respectively.
Revised: The second term is the distillation loss, where p Ti and p Si are the softmax predictions of the teacher and student models, respectively.

Original: Similarity-based knowledge distillation.
Revised: Similarity-based knowledge distillation.

Original: Recently, similarity-based knowledge distillation suchas SEED [13], OSS [8], and ISD [37] was introduced in the context of self-supervised learning(SSL).
Revised: Recently, similarity-based knowledge distillation such as SEED [13], OSS [8], and ISD [37] was introduced in the context of self-supervised learning (SSL).

Original: SEED [13] showed that the linear probe accuracy of a small student (ResNet-18) can beimproved by transferring the representations of a larger teacher (ResNet-50) pre-trained by SSLmethods such as MoCo-v2 [6].
Revised: SEED [13] showed that the linear probe accuracy of a small student (ResNet-18) can be improved by transferring the representations of a larger teacher (ResNet-50) pre-trained by SSL methods such as MoCo-v2 [6].

Original: Unlike this, OSS [8] aims to transfer representations of an evolvingteacher (ResNet-50) into a smaller student (ResNet-18) on the fly.
Revised: Unlike this, OSS [8] aims to transfer representations of an evolving teacher (ResNet-50) into a smaller student (ResNet-18) on the fly.

Original: Unlike SEED and OSS, ISD [37]considered the same size student and teacher network (ResNet-18), and showed a student can learnvisual representations by iterativly distilling the similarity of teacher’s representations.
Revised: Unlike SEED and OSS, ISD [37] considered the same size student and teacher network (ResNet-18), and showed a student can learn visual representations by iterativly distilling the similarity of teacher’s representations.

Original: These worksare closely related to our work.
Revised: These works are closely related to our work.

Original: Unlike these works, the BeamCLIP aims to transfer rich vision andlanguage representations of large-scale pre-trained models such as CLIP-ViT/16 [32] into a smallernetwork such as ResNet-50.
Revised: Unlike these works, the BeamCLIP aims to transfer rich vision and language representations of large-scale pre-trained models such as CLIP-ViT/16 [32] into a smaller network such as ResNet-18.

Original: Prompt engineering.
Revised: Prompt engineering.

Original: Recently, researchers showed that prompt engineering [2] is surprisinglyeffective at improving the performance of large-scale language models (LLMs) on downstreamtasks without fine-tuning.
Revised: Recently, researchers showed that prompt engineering [2] is surprisingly effective at improving the performance of large-scale language models (LLMs) on downstream tasks without fine-tuning.

Original: Prompts are input texts of language models that usually consists of atask description or several examples.
Revised: Prompts are input texts of language models that usually consist of a task description or several examples.

Original: To further simplify prompt engineering, prompt tuning [24]proposed to add k learnable tokens to the input texts, while having language models frozen.
Revised: To further simplify prompt engineering, prompt tuning [24] proposed to add k learnable tokens to the input texts, while having language models frozen.

Original: Similarto GPT-3, it is known that the zero-shot performance of CLIP [32] can be improved by designingthe prompt texts to each task.
Revised: Similar to GPT-3, it is known that the zero-shot performance of CLIP [32] can be improved by designing the prompt texts to each task.

Original: For example, on satellite image classification datasets, "A satellitephoto of a {label}" provides better performance than the default "A photo of a {label}" .
Revised: For example, on satellite image classification datasets, "A satellite photo of a {label}" provides better performance than the default "A photo of a {label}" .

Original: Inspired by this, we propose context-based prompt augmentation that extends the basic prompt of CLIP to better encode prototypical text anchor representations by alleviating the lexical ambiguity ofclass label texts.
Revised: Inspired by this, we propose context-based prompt augmentation that extends the basic prompt of CLIP to better encode prototypical text anchor representations by alleviating the lexical ambiguity of class label texts.

Original:  Method
Revised:  Method

Original: Problem formulation.
Revised: Problem formulation.

Original: Formally, our problem is to transfer aligned cross-modal representations ofa strong teacher model f Tϕ ( · ) into a target student model f Sθ ( · ) with unlabeled data D u
Revised: Formally, our problem is to transfer aligned cross-modal representations of a strong teacher model f Tϕ ( · ) into a target student model f Sθ ( · ) with unlabeled data D u

Original: = { x i } Ni =1 .
Revised: = { x i } Ni =1 .

Original: Given each unlabeled image x i , we formulate representation transfer as a regression task that matchesteacher representations f
Revised: Given each unlabeled image x i , we formulate representation transfer as a regression task that matches teacher representations f

Original: Tϕ ( x i ) to a student’s f
Revised: Tϕ ( x i ) to a student’s f

Original: Sθ ( x i ) .
Revised: Sθ ( x i ) .

Original: As the student network is parameterized by θ ,the learning objective is
Revised: As the student network is parameterized by θ , the learning objective is

Original: [Equation]
Revised: [Equation]

Original: Normalizing the representations via l 2 -normalization ( i.e., q i =
Revised: Normalizing the representations via l 2 -normalization ( i.e., q i =

Original: f Sθ ( x i ) /
Revised: f Sθ ( x i ) /

Original: ∥
Revised: ∥

Original: f Sθ ( x i ) ∥ 2 and k i =
Revised: f Sθ ( x i ) ∥ 2 and k i =

Original: f Tϕ ( x i ) /
Revised: f Tϕ ( x i ) /

Original: ∥
Revised: ∥

Original: f Tϕ ( x i ) ∥ 2 ) leads to the following simplification:
Revised: f Tϕ ( x i ) ∥ 2 ) leads to the following simplification:

Original: [Equation]
Revised: [Equation]

Original: The problem now involves maximizing the cosinesimilarity between l 2 -normalized representationsfrom teacher and student models.
Revised: The problem now involves maximizing the cosine

Original: The BeamCLIP first measures the normalized cross-modal similarity of the query image compared to anchor text representations in the teacher’s embedding space.
Revised: The BeamCLIP first measures the normalized cross-modal similarity of the query image compared to anchor text representations in the teacher’s embedding space

Original: Then, it encourages the student to mimic the same cross-modal similarity in the student’s embedding space.
Revised: Then, it encourages the student to mimic the same cross-modal similarity in the student’s embedding space.

Original: To better align image representations, our method uses self-supervised pre-training of the student model.
Revised: To better align image representations, our method uses self-supervised pre-training of the student model.

Original: Finally, to avoid text ambiguity, we uses context-based prompt augmentation.
Revised: Finally, to avoid text ambiguity, we uses context-based prompt augmentation.

Original: Method overview.
Revised: Method overview.

Original: The overview of BeamCLIP is shown in Figure 2.
Revised: The overview of BeamCLIP is shown in Figure 2.

Original: The teacher model ofthe BeamCLIP consists of an image encoder f
Revised: The teacher model of the BeamCLIP consists of an image encoder f

Original: Tϕ ( · ) and a text encoder g Tψ ( · ) .
Revised: Tϕ ( · ) and a text encoder g Tψ ( · ) .

Original: These encoders arepre-trained under a simple task of matching images to texts with large-scale corpora.
Revised: These encoders are pre-trained under a simple task of matching images to texts with large-scale corpora.

Original: Image representations f
Revised: Image representations f

Original: Tϕ ( x i ) and text representations g Tψ ( t i ) are thus well-aligned within a cross-modal embeddingspace.
Revised: Tϕ ( x i ) and text representations g Tψ ( t i ) are thus well-aligned within a cross-modal embedding space.

Original: We provide the details of the BeamCLIP in the following sections.
Revised: We provide the details of the BeamCLIP in the following sections.

Original: More specifically, wedescribe how to extend the basic problem setting by leveraging the unique features of CLIP wherevision and language representations are precisely aligned.
Revised: More specifically, we describe how to extend the basic problem setting by leveraging the unique features of CLIP where vision and language representations are precisely aligned.

Original: Throughout the paper, we use the notation
Revised: Throughout the paper, we use the notation

Original: CLIP-ViT/16 to denote the CLIP [32] model that uses Vision Transformer (ViT) [12] with the patchsize of 16x16 as the image encoder.
Revised: CLIP-ViT/16 to denote the CLIP [32] model that uses Vision Transformer (ViT) [12] with the patch size of 16x16 as the image encoder.

Original: Similar to this, CLIP-RN50 denotes the CLIP model with
Revised: Similar to this, CLIP-RN50 denotes the CLIP model with

Original: ResNet-50 [17] as the image encoder.
Revised: ResNet-50 [17] as the image encoder.

Original: 3.1 Similarity-based cross-modal representation transfer
Revised: 3.1 Similarity-based cross-modal representation transfer

Original: To effectively distill cross-modal representations, we use similarity-based matching as describedabove.
Revised: To effectively distill cross-modal representations, we use similarity-based matching as described above.

Original: Our similarity-based representation transfer utilizes two carefully designed loss functions: (1)instance similarity matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.
Revised: Our similarity-based representation transfer utilizes two carefully designed loss functions: (1) instance similarity matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.

Original: Instance similarity matching.
Revised: Instance similarity matching.

Original: This objective is directly derived from Eq.
Revised: This objective is directly derived from Eq.

Original: Given a query imagex i , it encourages the student image encoder f Sθ ( · ) to regress the representation of the teacher imageencoder f T ϕ ( · ) .
Revised: Given a query image x i , it encourages the student image encoder f Sθ ( · ) to regress the representation of the teacher image encoder f T ϕ ( · ) .

Original: We apply conventional image augmentations (see Appendix B) on a query imagex i , and the same augmented image ˆ x i is fed to both the teacher and student image encoders.
Revised: We apply conventional image augmentations (see Appendix B.1) on a query image x i , and the same augmented image ˆ x i is fed to both the teacher and student image encoders.

Original: Givenunlabeled query images D u
Revised: Given unlabeled query images D u

Original: = { x i } Ni =1 , it is formulated as follows:
Revised: = { x i } Ni =1 , it is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: However, the similarity signal from a single instance is not enough to constraint the student representations.
Revised: However, the similarity signal from a single instance is not enough to constraint the student representations.

Original: For example, the topological ambiguity may occur in image encoding, since two symmetricrepresentations have the same cosine similarity compared to a single teacher representation.
Revised: For example, the topological ambiguity may occur in image encoding, since two symmetric representations have the same cosine similarity compared to a single teacher representation (see Appendix B.2).

Original: Weconjecture that this can be mitigated by incorporating multiple anchor points to the query points.
Revised: We conjecture that this can be mitigated by incorporating multiple anchor points to the query points.

Original: Based on this idea, we introduce cross-modal similarity matching loss.
Revised: Based on this idea, we introduce cross-modal similarity matching loss.

Original: Cross-modal similarity matching.
Revised: Cross-modal similarity matching.

Original: To better align a student representation q i with the teacherrepresentation k i , we introduce cross-modal similarity matching (CSM) loss.
Revised: To better align a student representation q i with the teacher representation k i , we introduce cross-modal similarity matching (CSM) loss.

Original: We use multiple anchorpoints to cope with the ambiguity problem mentioned above.
Revised: We use multiple anchor points to cope with the ambiguity problem mentioned above.

Original: Further, we use text representations asanchor points, since we can easily generate prototypical anchor points by using text prompts and class
Revised: Further, we use text representations as anchor points, since we can easily generate prototypical anchor points by using text prompts and class

Original: [Table]
Revised: [Table]

Original: labels.
Revised: labels.

Original: Since image and text representations are precisely aligned in CLIP, we can effectively applythis approach.
Revised: Since image and text representations are precisely aligned in CLIP, we can effectively apply this approach.

Original: More specifically, the BeamCLIP first measures the normalized image-text similarityof the query image compared to prototypical text points in the teacher’s embedding space.
Revised: More specifically, the BeamCLIP first measures the normalized image-text similarity of the query image compared to prototypical text points in the teacher’s embedding space.

Original: Then, itencourages the student to mimic the same image-text similarity in the student’s embedding space.
Revised: Then, it encourages the student to mimic the same image-text similarity in the student’s embedding space.

Original: More formally, we generate multiple anchor representations A
Revised: More formally, we generate multiple anchor representations A

Original: = { a j } M j =1 by encoding class texts
Revised: = { a j } M j =1 by encoding class texts

Original: C = { c j } Mj =1 with the teacher text encoder g Tψ ( · ) (in other words, a j = g Tψ ( c j ) ).
Revised: C = { c j } Mj =1 with the teacher text encoder g Tψ ( · ) (in other words, a j = g Tψ ( c j ) ).

Original: To measure thesimilarity regarding to multiple anchor representations A , we define the normalized cross-modalsimilarity as follows:
Revised: To measure the similarity regarding to multiple anchor representations A , we define the normalized cross-modal similarity as follows:

Original: [Equation]
Revised: [Equation]

Original: (5) where τ is a temperature hyperparameter that is set to 0.01 in our experiments.
Revised: where τ is a temperature hyperparameter that is set to 0.01 in our experiments.

Original: Then, we evaluate the cross-modal similarity distribution by using a set of normalized cross-modalsimilarities:
Revised: Then, we evaluate the cross-modal similarity distribution by using a set of normalized cross-modal similarities:

Original: [Equation]
Revised: [Equation]

Original: Then, the student model is optimized to mimic the normalized cross-modal similarity of the teacher’sembedding space by minimizing the cross entropy, i.e., H ( P ( k i | A ) , P ( q i | A )) .
Revised: Then, the student model is optimized to mimic the normalized cross-modal similarity of the teacher’s embedding space by minimizing the cross entropy, i.e., H ( P ( k i | A ) , P [Equation]

Original:  We further minimize the entropy of normalized cross-modal similarities in the student embeddingspace i.e., H ( P ( q i | A )) .
Revised: q i | A )) . We further minimize the entropy of normalized cross-modal similarities in the student embedding space i.e., H ( P ( q i | A )) .

Original: This minimization helps the student provide query representations q i that aremore attracted to anchor representations A
Revised: This minimization helps the student provide query representations q i that are more attracted to anchor representations A

Original: = { a j } Mj =1 .
Revised: = { a j } Mj =1 .

Original: This entropy minimization is also known tobe effective in other domains such as semi-supervised learning [15].
Revised: This entropy minimization is also known to be effective in other domains such as semi-supervised learning [15, 29].

Original: Altogether, the CSM loss is formulated as follows:
Revised: Altogether, the CSM loss is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Final Loss.
Revised: Final Loss.

Original: The final loss of the BeamCLIP is formulated as follows:
Revised: The final loss of the BeamCLIP is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ISM is the scale hyperparameter that is set to 10 in our experiments.
Revised: where λ ISM is the scale hyperparameter that is set to 10 in our experiments.

Original: 3.2 Context-based prompt augmentation
Revised: 3.2 Context-based prompt augmentation

Original: We found that lexical ambiguity in prompt texts can lead to semantically incorrect text embeddings.
Revised: We found that lexical ambiguity in prompt texts can lead to semantically incorrect text embeddings.

Original: This may result in an unexpected discrepancy of image-text alignment in the teacher’s embeddingspace.
Revised: This may result in an unexpected discrepancy of image-text alignment in the teacher’s embedding space.

Original: For example, Flowers102 [28] dataset has some classes with unusual and ambiguous flowernames, such as “snapdragon”, “bird of paradise”, and “colt’s foot”.
Revised: For example, Flowers102 [28] dataset has some classes with unusual and ambiguous flower names, such as “snapdragon”, “bird of paradise”, and “colt’s foot”.

Original: Therefore, incorrect prototypicalanchor points might be compared with a query image.
Revised: Therefore, incorrect prototypical anchor points might be compared with a query image.

Original: To address this issue of semantic ambiguitiesin the text, we introduce context-based prompt augmentation (CPA), a data-driven approach thataugments basic prompts with contextual text such as Wikipedia descriptions or hierarchical labels.
Revised: To address this issue of semantic ambiguities in the text, we introduce context-based prompt augmentation (CPA), a data-driven approach that augments basic prompts with contextual text such as Wikipedia descriptions or hierarchical labels.

Original: For prompt tuning with Wikipedia descriptions, we use the template "A photo of a {label}.{Wikipedia description} ".
Revised: For prompt tuning with Wikipedia descriptions, we use the template "A photo of a {label}. {Wikipedia description} ".

Original: We use this template for Flowers102 and Pets37 in our experiments.
Revised: We use this template for Flowers102 and Pets37 in our experiments.

Original: We provide some examples from the Flowers102 dataset in Table 1.
Revised: We provide some examples from the Flowers102 dataset in Table 1.

Original: For prompt tuning with hierarchical labels, we use the template "A photo of a {fine label}, categorized as {coarselabel} ".
Revised: For prompt tuning with hierarchical labels, we use the template "A photo of a {fine label}, categorized as {coarse label} ".

Original: We use this template for CIFAR100 and ImageNet in our experiments.
Revised: We use this template for CIFAR100 and ImageNet in our experiments.

Original: Analogous examplesfrom CIFAR100 can be found in Table 8 of Appendix B.3.
Revised: Analogous examples from CIFAR100 can be found in Table 11 in Appendix B.3.

Original: 3.3 Other details
Revised: 3.3 Other details

Original: Self-supervised pre
Revised: Self-supervised pre

Original: -training of student.
Revised: -training of student.

Original: To help the student mimic the teacher’s cross-modalembedding space better, we pre-train the student image encoder with a self-supervised method.
Revised: To help the student mimic the teacher’s cross-modal embedding space better, we pre-train the student image encoder with a self-supervised method.

Original: Sinceself-supervised pre -training such as SimCLR [4], MoCo-V2 [18], and SwAV [3] provides a weaklyclustered embedding space based on similarities, it can be used as a better initial state for the studentto mimic the teacher’s embedding space.
Revised: Since self-supervised pre-training such as SimCLR [4], MoCo-v2 [6], and SwAV [3] provides a weakly clustered embedding space based on similarities, it can be used as a better initial state for the student to mimic the teacher’s embedding space.

Revised: The details can be found in B.4.

Original: We show the effect of SSL pre-training of the student in theexperiment section (see Table 4 and 7).
Revised: We show the effect of SSL pre-training of the student in the experiment section (see Table 4 and 7).

Original: Optimization.
Revised: Optimization.

Original: For optimization we use SGD with cosine annealing schedule (SGDR) [25].
Revised: For optimization we use SGD with cosine annealing schedule (SGDR) [25].

Original: Tostabilize training, we use a momentum encoder that updates its weights via exponential movingaverage (EMA) [18, 16].
Revised: To stabilize training, we use a momentum encoder that updates its weights via exponential moving average (EMA) [18, 16].

Original: The momentum encoder of a student θ ˆ S is updated using the following rule:
Revised: The momentum encoder of a student θ ˆ S is updated using the following rule:

Original: [Equation]
Revised: [Equation]

Original: (9) where θ S is the image encoder of a student model and m is a momentum hyperparameter that is set0.99 in our experiments.
Revised: where θ S is the image encoder of a student model and m is a momentum hyperparameter that is set 0.99 in our experiments.

Original: The model hyperparameters are summarized in Table 9 of Appendix B.5.
Revised: The model hyperparameters are summarized in Table 12 in Appendix B.5.

Original:  Experiments
Revised:  Experiments

Original: Downstream datasets.
Revised: Downstream datasets.

Original: We evaluate the BeamCLIP on six standard benchmark datasets: CIFAR10[23], CIFAR100 [23], STL10 [9], Flowers102 [28], Pets37 [31], and ImageNet-1K [10].
Revised: We evaluate the BeamCLIP on six standard benchmark datasets: CIFAR [23], CIFAR100 [23], STL10 [9], Flowers102 [28], Pets37 [31], and ImageNet-1K [10].

Original: Followingconvention, we split the datasets into train, validation, and test sets.
Revised: Following convention, we split the datasets into train, validation, and test sets.

Original: Then, we use train set for transfer,and test set for evaluation.
Revised: Then, we use train set for transfer, and test set for evaluation.

Original: For ImageNet, we use the validation set as a test set, since its test set doesnot provide labels.
Revised: For ImageNet, we use the validation set as a test set, since its test set does not provide labels.

Original: More details on the datasets are summarized in Table 5 of Appendix A.
Revised: More details on the datasets are summarized in Table 8 in Appendix A.

Original: 4.1 Representation transfer with unlabeled target data
Revised: 4.1 Representation transfer with unlabeled target data

Original: Setting.
Revised: Setting.

Original: We compare the BeamCLIP with various self-supervised methods in terms of linear probeaccuracy on ImageNet-1K. Following the conventional protocol, we use ResNet-50 [17] as thebase encoder and evaluate the learned representations by using logistic regression.
Revised: We compare the BeamCLIP with various self-supervised methods in terms of linear probe accuracy on ImageNet-1K. Following the conventional protocol, we use ResNet-18 and ResNet- [17] as the base encoder and evaluate the learned representations by using logistic regression.

Original: We use LBFGSalgorithm [44] for logistic regression.
Revised: We use LBFGS algorithm [44] for logistic regression.

Original: Its hyperparameter C is determined through coarse-grainedhyperparameter search on the validation split.
Revised: Its hyperparameter C is determined through coarse-grained hyperparameter search on the validation split.

Original: And, the accuracy is evaluated in the test split.
Revised: And, the accuracy is evaluated in the test split.

Original: Wefound that it provides the best linear probe accuracy when C is set to 100.
Revised: We found that it provides the best linear probe accuracy when C is set to 30.

Original: We perform our experimentson 8 NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training.
Revised: We perform our experiments on 8 NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Transfer to ResNet-50.
Revised: Transfer to ResNet-50.

Original: Table 2 shows the comparison of the BeamCLIP with vision-only selfsupervised methods such as SimCLR [4], MoCo-v2 [6], SwAV [3], BYOL [16]
Revised: Table 2 shows the comparison of the BeamCLIP with vision-only selfsupervised methods such as SimCLR [4], MoCo-v2 [6], SwAV [3], BYOL [16]

Original: [Table]
Revised: [Table]

Original: and SimSiam[5].
Revised: and SimSiam [5].

Original: The BeamCLIP provides better visual representation by achieving 74.8% top-1 linear probeaccuracy on ImageNet-1K [10].
Revised: The BeamCLIP provides better visual representation by achieving 74.8% top-1 linear probe accuracy on ImageNet-1K [10].

Original: While self-supervised methods take long training epochs to achievecomparable accuracy, BeamCLIP-RN50 achieves better accuracy with less training epochs.
Revised: While self-supervised methods take long training epochs to achieve comparable accuracy, BeamCLIP-RN50 achieves better accuracy with less training epochs.

Original: Also, notethat BeamCLIP-RN50’s representations provide better accuracy than CLIP-RN50’s representations.
Revised: Also, note that BeamCLIP-RN50’s representations provide better accuracy than CLIP-RN50’s representations.

Original: Transfer to ResNet-18.
Revised: Transfer to ResNet-18.

Original: To check if the BeamCLIP can transfer CLIP representations into smallermodels than ResNet-50 (24M), we also measure ImageNet-1K Top-1 linear probe accuracy on
Revised: To check if the BeamCLIP can transfer CLIP representations into smaller models than ResNet-50 (24M), we also measure ImageNet-1K top-1 linear probe accuracy on

Original: ResNet-18 (11M).
Revised: ResNet-18 (11M).

Original: ResNet-18 is trained from scratch (not self-supervised pre-trained with SimCLR),while transferring CLIP ViT-B/16 representations.
Revised: ResNet-18 is trained from scratch (not self-supervised pre-trained with SimCLR), while transferring CLIP ViT-B/16 representations.

Original: As shown in Table 3, BeamCLIP learns betterrepresentations than SSL methods such as SimCLR [4], MoCo-v2 [6], BYOL [16], and SwAV [3].
Revised: As shown in Table 3, BeamCLIP learns better representations than SSL methods such as SimCLR [4], MoCo-v2 [6], BYOL [16], and SwAV [3].

Original: More importantly, the BeamCLIP provides better performance than OSS [8] that simultaneouslylearns and transfers representations from ResNet-50.
Revised: More importantly, the BeamCLIP provides better performance than OSS [8] that simultaneously learns and transfers representations from ResNet-50.

Original: The learning curve is presented in Figure 6 of
Revised: The learning curve is presented in Figure 8 in

Original: Appendix C.1.
Revised: Appendix C.1.

Original: Effect of self-supervised pre-training.
Revised: Effect of self-supervised pre-training.

Original: Table 4 shows ImageNet-1K top-1 linear probe accuracy on
Revised: Table 4 shows ImageNet-1K top-1 linear probe accuracy on

Original: BeamCLIP-RN50 representations by using different SSL pre-training.
Revised: BeamCLIP-RN50 representations by using different SSL pre-training.

Original: With the better SSL method(SwAV [3] > SimCLR [4]), the BeamCLIP can learn better representations with an increased linearprobe accuracy.
Revised: With the better SSL method (SwAV [3] > SimCLR [4]), the BeamCLIP can learn better representations with an increased linear probe accuracy.

Original: 4.2 Representation transfer with unlabeled non-target data
Revised: 4.2 Representation transfer with unlabeled non-target data

Original: To check if the BeamCLIP also inherits the powerful zero-shot capability of CLIP, we comparezero-shot accuracy of CLIP variants on ImageNet-1K. For zero-shot measure, we use CC-3M [34]and ImageNet-21K (12M samples) [33] that do not have overlap with Imagenet-1K. Table 5 shows thecomparison of zero-shot accuracy.
Revised: To check if the BeamCLIP also inherits the powerful zero-shot capability of CLIP, we compare zeroshot accuracy of CLIP variants on ImageNet-1K. For zero-shot measure, we use CC-3M [34] and ImageNet-21K (12M samples) [33] that do not have overlap with ImageNet-1K. Table 5 shows the comparison of zero-shot accuracy.

Original: The BeamCLIP-RN50 achieves about 57.5% zero-shot accuracythat is highly comparable with CLIP RN-50 (59.6%).
Revised: The BeamCLIP-RN50 achieves about 57.5% zero-shot accuracy that is highly comparable with CLIP RN-50 (59.6%).

Original: The learning curve is presented in Figure 7 of
Revised: The learning curve is presented in Figure 9 of

Original: Appendix C.1.
Revised: Appendix C.1.

Original: 4.3 Transfer learning accuracy on various target datasets
Revised: 4.3 Transfer learning accuracy on various target datasets

Original: Setting.
Revised: Setting.

Original: We evaluate how effectively the BeamCLIP transfers CLIP-ViT representations into astudent model by evaluating classification accuracy on various datasets (see Table 6).
Revised: We evaluate how effectively the BeamCLIP transfers CLIP-ViT representations into a student model by evaluating classification accuracy on various datasets (see Table 6).

Original: We choose
Revised: We choose

Original: ResNet-50 [17] as the student model to compare the distilled target model with CLIP-RN50.
Revised: ResNet-50 [17] as the student model to compare the distilled target model with CLIP-RN50.

Original: Also,
Revised: Also,

Original: ResNet-50 is conventionally used in evaluating linear probe accuracy on ImageNet-1K.
Revised: ResNet-50 is conventionally used in evaluating linear probe accuracy on ImageNet-1K.

Original: However, the
Revised: However, the

Original: BeamCLIP canadopt any architecture, not just ResNet-50.
Revised: BeamCLIP can any architecture, not just ResNet-50.

Revised: To achieve such a high zero-shot accuracy

Original: [Table]
Revised: [Table]

Revised: CLIP uses very large image-text pair data (WIT-400M).

Revised: Instead

Revised: [Table]

Revised: the BeamCLIP can achieve the comparable zero-shot accuracy by effectively transferring the teacher’s representations, while using only 3% data (ImageNet-21K (12M)).

Revised: Note that OpenCLIP provides only about

Revised: 5% zero-shot accuracy with the similar amount of data (CC-12M)

Revised: [Table]

Original: As baselines
Revised: As baselines

Original: [Table]
Revised: [Table]

Original: we choose two representative distillation methods among many methods: (1) conventional knowledge distillation (KD) [19] and (2) contrastive representation distillation (CRD) [38].
Revised: we choose two representative distillation methods among many methods: (1) conventional knowledge distillation (KD) [19] and (2) contrastive representation distillation (CRD) [38].

Original: Since conventional KD aims to mimic the task-specific predictions of the teacher model unlike the
Revised: Since conventional KD aims to mimic the task-specific predictions of the teacher model unlike the

Original: BeamCLIP , we apply the KL divergence on minimizing the cross-modal similarity distribution ( i.e.,
Revised: BeamCLIP [Table] we apply the KL divergence on minimizing the cross-modal similarity distribution ( i.e.,

Original: P ( q i | A ) and P ( k i | A ) ), instead of the Cross-entropy (CE).
Revised: P ( q i | A ) and P ( k i | A ) ), instead of the Cross-entropy (CE).

Original: CRD proposes a variant of InfoNCE lossfor representation distillation, which we apply on normalized representations ( i.e., q i
Revised: CRD proposes a variant of InfoNCE loss for representation distillation, which we apply on normalized representations ( i.e., q i

Original: and
Revised: and

Original: k i ).
Revised: k i ).

Original: Thedetails of each method can be found in the related work section 2.
Revised: The details of each method can be found in the related work section 2.

Original: Results.
Revised: Results.

Original: Table 6 shows a comparison of teacher and student accuracy on various datasets.
Revised: Table 6 shows a comparison of teacher and student accuracy on various datasets.

Original: We empirically demonstrate that the BeamCLIP can effectively transfer vision and language representationsof a large teacher model (CLIP ViT-B/16) into a small student model (ResNet-50).
Revised: We empirically demonstrate that the BeamCLIP can effectively transfer vision and language representations of a large teacher model (CLIP ViT-B/16) into a small student model (ResNet-50).

Original: We find thatthe KL divergence used in conventional knowledge distillation (KD) is not effective in transferring
Revised: We find that the KL divergence used in conventional knowledge distillation (KD) is not effective in transferring

Original: CLIP-ViT representations.
Revised: CLIP-ViT representations.

Original: Also, the contrastive learning-based approach is not effective.
Revised: Also, the contrastive learning-based approach is not effective.

Original: Unlike this,the BeamCLIP can effectively transfer CLIP ViT-B/16 representations into ResNet-50, achieving veryhigh accuracy that is comparable or better than the teacher accuracy.
Revised: Unlike this, the BeamCLIP can effectively transfer CLIP ViT-B/16 representations into ResNet-50, achieving very high accuracy that is comparable or better than the teacher accuracy.

Original: KD simply minimizes the errorbetween single instances.
Revised: KD simply minimizes the error between single instances.

Original: We conjecture that the cross-modal similarity to multiple anchor pointsintroduced in the BeamCLIP helps the student preserve the topology of the teacher’s embeddingspace.
Revised: We conjecture that the cross-modal similarity to multiple anchor points introduced in the BeamCLIP helps the student preserve the topology of the teacher’s embedding space.

Original: Also, note that context-based prompt augmentation helps achieve better accuracy after representationtransfer.
Revised: Also, note that context-based prompt augmentation helps achieve better accuracy after representation transfer.

Original: Since Flowers102 has many ambiguous labels, our experiment shows that text promptaugmentation significantly increases the student’s accuracy compared to the teacher’s accuracy.[Table] [Table]
Revised: Since Flowers102 has many ambiguous labels, our experiment shows that text prompt augmentation significantly increases the student’s accuracy compared to the teacher’s accuracy.

Original: Ablation study.
Revised: Ablation study.

Original: Table 7 shows the ablation study results of the BeamCLIP .
Revised: Table 7 shows the ablation study results of the BeamCLIP .

Original: Our empirical findingsare as follows: (1) Instance similarity matching (ISM) is not enough by itself to preserve the topologyof the teacher’s embedding space.
Revised: Our empirical findings are as follows: (1) Instance similarity matching (ISM) is not enough by itself to preserve the topology [Figure] yp e I m g .

Revised: E n c .

Original: (2) Cross-modal similarity matching (SCM) compared to multipleanchor points helps the student mimic the teacher’s embedding space.
Revised: P a r a m . S i [Table] C

Revised: I F

Revised: A

Revised: R

Revised: C

Revised: I F

Revised: A

Revised: R

Revised: S TL

Revised: F l o w e r s

Revised: P e t s

Revised: I m a g e

Revised: N e t1 K

Revised: [Table]

Revised: [Table]

Revised: image retrieval results.

Revised: A red rectangle denotes an incorrect result.

Revised: CLIP-RN50 provides many incorrect results, since its zero-shot accuracy is relatively low.

Revised: In contrast, BeamCLIP-RN50 provides much improved results, since it is transferred from CLIP-ViT/16 with higher zero-shot accuracy.

Revised: of the teacher’s embedding space.

Revised: (2) Cross-modal similarity matching (SCM) compared to multiple anchor points helps the student mimic the teacher’s embedding space.

Original: (3) Self-supervised pretraining of the student (SSL PT) helps the student mimic the teacher’s embedding space.
Revised: (3) Self-supervised pretraining of the student (SSL PT) helps the student mimic the teacher’s embedding space.

Original: (4) Entropyminimization (EntMin) helps to improve the accuracy.
Revised: (4) Entropy minimization (EntMin) helps to improve the accuracy.

Original: (5) Context-based prompt augmentation (CPA)helps measure the similarity more precisely.
Revised: (5) Context-based prompt augmentation (CPA) helps measure the similarity more precisely.

Original: As shown in the table, Flowers102 dataset is sensitive toself-supervised pre-training of student.
Revised: As shown in the table, Flowers102 dataset is sensitive to self-supervised pre-training of student.

Original: We conjecture that since Flowers102 dataset has only 1020training samples for the 102 classes, it is not enough to probe the teacher’s representation space.
Revised: We conjecture that since Flowers102 dataset has onlytraining samples for the 102 classes, it is not enough to probe the teacher’s representation space.

Revised: Qualitative result.

Revised: To see the quality of the transferred representations, we analysed text-image retrieval results on the Flowers102 dataset.

Original: Conclusion and Limitations
Revised: Figure 3 compares the top-5 text-image retrieval results between CLIP-RN50 and BeamCLIP-RN50. A red rectangle denotes an incorrect result. Compared to CLIP-RN50, BeamCLIP-RN50 provides much improved results, since its representations are transferred from CLIP-ViT/16 with higher zero-shot accuracy. More interestingly, BeamCLIP-RN provides surprisingly good text-image retrieval results, even though unseen text prompts such as "a photo of {pink rose}" or "a photo of {yellow rose}" are given.

Original: Conclusion.
Revised: [Figure] 4.4 Effect of random text prompts We measured how effective the BeamCLIP is in cases where the class names of the target dataset are not perfectly given. Figure 4 shows the effect of the randomly sampled text prompts on CIFAR100.

Original: In this paper, we provide the BeamCLIP than can effectively transfer large pre-trainedvision-language model (e.g., CLIP-ViT) into a small target model (e.g., ResNet-18) with cross-modalsimilarity matching (CSM) and context-based prompt augmentation (CPA).
Revised: We can see that the BeamCLIP is still effective, even when (a) the subset of the 100 class names of CIFAR100 are given as the text prompts, or (b) the text prompts are randomly sampled from a non-target dataset (ImageNet-1K). The exact values in Figure 4 are presented in Table 15 and Table 16 in Appendix C.2. Also, the additional results on CIFAR10 are provided in Appendix C.2.  Limitations and Conclusion

Original: We empirically show thatthe BeamCLIP can learn bettervisual representations than vision-only self-supervised learning (SSL)methods, by leveraging a pre-trained vision-language model (CLIP). The BeamCLIP is not intendedto another CLIP, but a effective CLIP student.
Revised: Limitations. With the help of rich representations of pre-trained CLIP, the BeamCLIP can learn better representations than SSL methods. However, since SSL methods can increase the performance at longer training epochs, the performance margin may be decreased in such a setting.

Revised: Another shortcoming is that context-based prompt augmentations may require additional engineering efforts.

Original: Limitations.
Revised: Conclusion.

Revised: In this paper, we provide the BeamCLIP that can effectively transfer large pre-trained vision-language model (e.g., CLIP-ViT) into a small target model (e.g., ResNet-18) with cross-modal similarity matching (CSM) and context-based prompt augmentation (CPA).

Original: With the help of rich representations of pre-trained CLIP, the BeamCLIP can learnbetter representations than SSL methods. However, since SSL methods can increase the performanceat longer training epochs, the performance margin may be decreased in such a setting. Anothershortcoming is that context-based prompt augmentations may require additional engineering efforts.
Revised: We empirically show that the BeamCLIP can learn better visual representations than vision-only self-supervised learning (SSL) methods, by leveraging a pre-trained vision-language model (CLIP). The BeamCLIP is not intended to be another CLIP, but an effective CLIP student.

Original: Broader impact
Revised: Broader impact

Original: This research aims to provide an simple and effective way to leverage CLIP for representationlearning.
Revised: This research aims to provide a simple and effective way to leverage CLIP for representation learning.

Original: With the help of CLIP, the BeamCLIP can learn better representations than self-supervisedlearning (SSL) methods.
Revised: With the help of CLIP, the BeamCLIP can learn better representations than self-supervised learning (SSL) methods.

Original: Since training CLIP requires very large data and hundreds of GPUs, it isimportant to provide a way to effectively reuse the pre-trained CLIP rather than training from scratchon a target model.
Revised: Since training CLIP requires very large data and hundreds of GPUs, it is important to provide a way to effectively reuse the pre-trained CLIP rather than training from scratch on a target model.

Original: We believe that  BeamCLIP can help to save cost and time.Checklist
Revised: We believe that the BeamCLIP can help to save cost and time.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4 and Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results? [No] Our work doesnot include theoretical results. (b) Did you include complete proofs of all theoretical results? [No] Our work does notinclude theoretical results.3.
Revised: Acknowledgement

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide the

Original: code, data, and instructions in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4 and

Original: Appendix B.

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 4.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.4.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 2,Section 3, and Section 4.(b)

Original: Did you mention the license of the assets?

Original: [No] We only used public benchmark

Original: datasets and open-sourced software.

Original: See Section 4. (c) Did you include any new assets either in the supplemental material or as a URL? [No] We did not use new assets.
Revised: We thank anonymous reviewers for their valuable comments.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [Yes] We only used public benchmark datasets.

Original: See Section 4.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [No] We did not use any data containing personally

Original: identifiable information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [No] We did not use crowdsourcing or conduct research with human
Revised: This work was fully supported by LG

Original: subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [No] We did not use crowdsourcing or conduct

Original: research with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation? [No] We did not use crowdsourcing or conduct [Table] research with human subjects.
Revised: AI Research.

Original: A Datasets Details
Revised: A Datasets Details

Original: We demonstrate the effectiveness of the BeamCLIP by using six downstream datasets.
Revised: We demonstrate the effectiveness of the BeamCLIP by using six downstream datasets.

Original: Table 5 showsthe details of the downstream datasets.
Revised: Table 8 shows the details of the downstream datasets.

Original: [Table]
Revised: [Table]

Original: B Method Details
Revised: B Method Details

Original: In this section, we provide some details of the BeamCLIP .
Revised: In this section [Table] we provide some details of the BeamCLIP .

Original: More specifically, we provide the detailsof two main contributions that are (1) cross-modal similarity matching (CSM) and (2) context-basedprompt augmentation (CPA).
Revised: More specifically, we provide the details of two main contributions that are (1) cross-modal similarity matching (CSM) and (2) context-based prompt augmentation (CPA).

Original: Also, we provide the other implementation details such as imageaugmentation, similarity smoothing, model hyperparameters, etc.
Revised: Also, we provide the other implementation details such as image augmentation, similarity smoothing, model hyperparameters, etc.

Original: B.1 Image augmentation details
Revised: B.1 Image augmentation details

Original: We use conventional image augmentation when performing representation transfer by using unlabeledimages in downstream datasets.
Revised: We use conventional image augmentation when performing representation transfer by using unlabeled images in downstream datasets.

Original: Table 6 provides a list of image augmentation used for unsupervisedrepresentation transfer on downstream datasets.
Revised: Table 9 provides a list of image augmentation used for unsupervised representation transfer on downstream datasets

Original: [Table]
Revised: [Table]

Original: size Normalize -
Revised: [Table]

Original: B.2 Cross-modal similarity matching details
Revised: B.2 Cross-modal similarity matching details

Revised: [Table]

Original: Cross-modal similarity matching (CSM) is the main method of the BeamCLIP .
Revised: [Table] similarity matching (CSM) is the main method of the BeamCLIP .

Original: To make the conceptof CSM clearer, we provide an illustration of CSM in Figure 3.
Revised: To make the concept of CSM clearer, we provide an illustration of CSM in Figure 5.

Original: B.3 Context-based prompt augmentation details
Revised: B.3 Context-based prompt augmentation details

Original: To prepare for better text anchor embeddings for unsupervised representation transfer, we introducecontext-based prompt augmentation (CPA).
Revised: To prepare for better text anchor embeddings for unsupervised representation transfer, we introduce context-based prompt augmentation (CPA).

Original: To make the concept of CPA clearer, we provide anillustration of CPA in Figure 4.
Revised: To make the concept of CPA clearer, we provide an illustration of CPA in Figure 6.

Original: Also, we provide an example of the hierarchical class labels in Table 7 and an example context-basedprompt augmentation for CIFAR100 in Table 8.
Revised: Also, we provide an example of the hierarchical class labels in Table 10 and an example context-based prompt augmentation for CIFAR100 in Table 11.

Original: B.4 Other implementation details
Revised: B.4 Other implementation details

Original: Self-supervised pre
Revised: Self-supervised pre

Original: -training of student.
Revised: -training of student.

Original: For self-supervised pre-training, we adopt SimCLR, sinceit is simple and effective.
Revised: For self-supervised pre-training, we adopt SimCLR, since it is simple and effective.

Original: SimCLR learns transferable visual representations by using InfoNCE loss
Revised: SimCLR learns transferable visual representations by using InfoNCE loss

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [30, 39] which encourages agreement between multiple views of the same image.
Revised: [30, 39] which encourages agreement between multiple views of the same image.

Original: More specifically,
Revised: More specifically,

Original: InfoNCE maximizes the similarity between multiple views of the same image ( i
Revised: InfoNCE maximizes the similarity between multiple views of the same image ( i

Original: [Table]
Revised: [Table]

Original: e., positive samples)and minimizes the similarity to multiple views of all other images in a training batch ( i
Revised: e., positive samples) and minimizes the similarity to multiple views of all other images in a training batch ( i

Original: [Table]
Revised: [Table]

Original: e., negativesamples).
Revised: e., negative samples).

Original: InfoNCE loss of SimCLR can be formulated as follows:
Revised: InfoNCE loss of SimCLR can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where h Si ∈ R 128 is a projection of a student representation q i ∈ R 512 , τ is a temperature hyperparameter that is set to 0.1, 

Revised: 10) where h Si ∈ R 128 is a projection of a student representation q i ∈ R 512 , τ is a temperature hyperparameter that is set to 0.1,

Original: [ k ̸
Revised: [ k ̸

Original: = i ] is an indicator function whose value is 1 if k ̸
Revised: = i ] is an indicator function whose value is 1 if k ̸

Original: = i , and B is a batchsize.
Revised: = i , and B is a batch size.

Original: Here, h i and h i ′ are projections of multiple views of the same input images x i .
Revised: Here, h i and h i ′ are projections of multiple views of the same input images x i .

Original: Similarity Smoothing.
Revised: Similarity Smoothing.

Original: To improve the effectiveness of distillation, we apply Label Smoothing (LS)[36] to the cross-modal similarity distillation loss.
Revised: To improve the effectiveness of distillation, we apply Label Smoothing (LS) [36] to the cross-modal similarity distillation loss.

Original: Recent works [27, 42] show that Label Smoothinghelps knowledge distillation.
Revised: Recent works [27, 42] show that Label Smoothing helps knowledge distillation.

Original: To apply Label Smoothing, we determine the most similar anchorrepresentation as follows:
Revised: To apply Label Smoothing, we determine the most similar anchor representation as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, we generate a modified cross-modal similarity distribution:
Revised: Then, we generate a modified cross-modal similarity distribution:

Original: [Equation]
Revised: [Equation]

Original: where 1 [ j = j ∗ ] is the indicator function whose value is 1 if j = j ∗ , M is the number of anchors, andα is the smoothing hyperparameter that is set to 0.2 in our experiments.
Revised: where 1 [ j = j ∗ ] is the indicator function whose value is 1 if j = j ∗ , M is the number of anchors, and α is the smoothing hyperparameter that is set to 0.2 in our experiments.

Original: B.5 Model hyperparameters
Revised: B.5 Model hyperparameters

Original: Table 9 provides the summary of model hyperparamters.
Revised: Table 12 provides the summary of model hyperparamters.

Original: We use the same hyperparameters on alldownstream datasets if not explicitly declared.
Revised: We use the same hyperparameters on all downstream datasets if not explicitly declared.

Original: [Table]
Revised: [Table]

Original: C Additional Experiment Results
Revised: C Additional Experiment Results

Original: [Figure]

Original: In this section, we provide additional experiment results.
Revised: In this section, we provide additional experiment results.

Original: First, we provide the learning curves that aregenerated while training the BeamCLIP .
Revised: First, we provide the learning curves that are generated while training the BeamCLIP .

Original: Second, we provide some experiment results on the effectsof random text prompts.
Revised: Second, we provide some experiment results on the effects of random text prompts.

Original: Third, we provide an example qualitative result that shows the advantage ofthe BeamCLIP .[Figure]
Revised: Third, we provide an example qualitative result that shows the advantage of the BeamCLIP .

Original: C.1 Learning curves of the BeamCLIP
Revised: C.1 Learning curves of the BeamCLIP

Original: [Figure]

Original: We provide the learning curve of the BeamCLIP for the experiment section.
Revised: We provide the learning curve of the BeamCLIP for the experiment section.

Original: Figure 5 shows thelearning curve for ImageNet-1K validation accuracy of BeamCLIP-RN50 representations trained

Original: [Figure]

Original:  with unlabeled ImageNet-1K. Figure 6 shows the learning curve for ImageNet-1K validation accuracyof BeamCLIP-RN18 representations trained with unlabeled ImageNet-1K. Figure 7 shows thelearning curve for ImageNet-1K zero-shot accuracy of BeamCLIP-RN50 representations trained with[Figure] unlabeled non-target data (ImageNet-21K).[Figure]
Revised: Figure 7 shows the learning curve for ImageNet-1K validation accuracy of BeamCLIP-RN50 representations trained with unlabeled ImageNet-1K. Figure 8 shows the learning curve for ImageNet-1K validation accuracy of BeamCLIP-RN18 representations trained with unlabeled ImageNet-1K. Figure 9 shows the learning curve for ImageNet-1K zero-shot accuracy of BeamCLIP-RN50 representations trained with unlabeled non-target data (ImageNet-21K).

Original: C.2 Effects of random text prompts
Revised: C.2 The effect of random text prompts

Original: In this section, we further analyze the effect of text prompts from the perspective of unsupervisedlearning.
Revised: In this section, we further analyze the effect of text prompts from the perspective of unsupervised learning.

Original: Before that, we briefly review the proposed method.
Revised: Before that, we briefly review the proposed method.

Original: In this paper, we propose the BeamCLIP, an unsupervised representation transfer method of a large pre-trained multimodal model such as
Revised: In this paper, we propose the BeamCLIP , an unsupervised representation transfer method of a large pre-trained multimodal model such as

Original: CLIP.
Revised: CLIP.

Original: The BeamCLIP can transfer the visual representations of CLIP by using unlabeled images on adownstream dataset.
Revised: The BeamCLIP can transfer the visual representations of CLIP by using unlabeled images on a downstream dataset.

Original: To achieve this, we propose cross-modal similarity matching (CSM).
Revised: To achieve this, we propose cross-modal similarity matching (CSM).

Original: In CSM,at first, given an unlabeled image, cross-modal similarity distribution is measured from multiple textprompt embeddings in the teacher’s embedding space.
Revised: In CSM, at first, given an unlabeled image, cross-modal similarity distribution is measured from multiple text prompt embeddings in the teacher’s embedding space.

Original: Then, a student model is encouraged to mimicthe cross-modal similarity distribution of the teacher model by matching these similarity distributions.
Revised: Then, a student model is encouraged to mimic the cross-modal similarity distribution of the teacher model by matching these similarity distributions.

Original: To achieve effective transfer, we use anchor text embeddings by encoding text prompts.
Revised: To achieve effective transfer, we use anchor text embeddings by encoding text prompts.

Original: For example,on CIFAR10, we use ten text prompts in the form of "a photo of {class name}" .
Revised: For example, on CIFAR10, we use ten text prompts in the form of "a photo of {class name}" .

Original: Note that thetext prompts are not paired with each image.
Revised: Note that the text prompts are not paired with each image.

Original: CIFAR10.
Revised: CIFAR10.

Original: To further decouple the dependency to the class names ofa dataset, we measure thetransfer learning accuracy by using random prompts. Figure 8 shows the transfer learning accuracyon CIFAR10 with regard to the number of random text prompt of CLIP. As shown in Figure 8 (a), the BeamCLIP is effective, while using only subset text prompts.
Revised: We measured how effective the BeamCLIP is in cases where the class names of the target dataset are not perfectly given.

Original: Note that the BeamCLIP (CE+EntMin)is more effective than BeamCLIP (KL), as we use more text prompts. As shown in Figure 9, the BeamCLIP is still effective compared to CLIP-RN50, even though random prompts from ImageNetare used.
Revised: Table 10 shows the effect of the randomly sampled text prompts on CIFAR10 [23].

Original: The values in Figure 8 are also presented in Table 10 and Table 12.
Revised: The values in Figure 10 are also presented in Table 13 and Table 14.

Original: CIFAR100.
Revised: CIFAR100.

Original: We perform the similar experiment on CIFAR100. Table 9 showsunsupervised transferlearning accuracy of CIFAR100 [23] with regard to the number of random text prompts.
Revised: Table 4 shows the effect of the randomly sampled text prompts on CIFAR100 [23].

Original: The valuesin Figure 9 are also presented in Table 12 and Table 13.
Revised: The values in Figure 4 are also presented in Table 15 and Table 16. [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: BeamCLIP (KL)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [32] (zero-shot) [Table] [32] (zero-shot)

Original: [Table]
Revised: [Table]

Original: Figure 10 shows qualitative results via top-5 text-image retrieval.
Revised: BeamCLIP (CE+EntMin)

Original: A red rectangle denotes an incorrectresult. CLIP-RN50 provides many incorrect results since it shows relatively low zero-shot accuracy. In contrast, BeamCLIP-RN50 provides much improved results, since it is distilled from CLIPViT/16 which shows very high zero-shot accuracy. More interestingly, BeamCLIP-RN50 providessurprisingly good text-image retrieval results, even though unseen text prompts such as "a photoof {pink rose}" or "a photo of {yellow rose}" are given.
Revised: [Table]