Original: Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching
Revised: Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Despite surprising performance on zero-shot transfer, pre-training a large-scalemultimodal model is often prohibitive as it requires a huge amount of data andcomputing resources.
Revised: Despite surprising performance on zero-shot transfer, pre-training a large-scalemultimodal model is often prohibitive as it requires a huge amount of data andcomputing resources.

Original: In this paper, we propose a method ( BeamCLIP ) that caneffectively transfer the representations of a large pre-trained multimodal model(CLIP ViT-B/16) into a small target model (ResNet-50).
Revised: In this paper, we propose a method ( BeamCLIP ) that caneffectively transfer the representations of a large pre-trained multimodal model(CLIP ViT-B/16) into a small target model (ResNet-50).

Original: For unsupervised transfer,we introduce cross-modal similarity matching (CSM) that enables a student modelto learn the representations of a teacher model by matching the relative similaritydistribution across text prompt embeddings.
Revised: For unsupervised transfer,we introduce cross-modal similarity matching (CSM) that enables a student modelto learn the representations of a teacher model by matching the relative similaritydistribution across text prompt embeddings.

Original: To better encode the text prompts,we design context-based prompt augmentation (CPA) that can alleviate the lexicalambiguity of input text prompts.
Revised: To better encode the text prompts,we design context-based prompt augmentation (CPA) that can alleviate the lexicalambiguity of input text prompts.

Original: Our experiments show that unsupervised representation transfer of a large pre-trained multimodal model can provide good ImageNetlinear probe accuracy (74.8%), while outperforming the existing self-supervisedlearning methods (SimCLR: 65.6%) and closing the gap with supervised learning(76.2%).
Revised: Our experiments show that unsupervised representation transfer of a large pre-trained multimodal model can provide good ImageNetlinear probe accuracy (74.8%), while outperforming the existing self-supervisedlearning methods (SimCLR: 65.6%) and closing the gap with supervised learning(76.2%).

Original:  Introduction
Revised:  Introduction

Original: Learning transferable representations is crucialfor successful downstream applications.
Revised: Learning transferable representations is crucialfor successful downstream applications.

Original: Contrastive learning such as SimCLR and MoCo-v2have shown notable success by forcing featuresof individual classes to be clustered and sufficiently scattered [38].
Revised: Contrastive learning such as SimCLR and MoCo-v2have shown notable success by forcing featuresof individual classes to be clustered and sufficiently scattered [41].

Original: But their downstreamperformances are still far behind the supervisedtraining as shown in Figure 1.
Revised: But their downstreamperformances are still far behind the supervisedtraining as shown in Figure 1.

Original: Recently, largescale vision and language pre-trained (VLP)models provide highly transferable visual representations via language supervision.
Revised: Recently, largescale vision and language pre-trained (VLP)models provide highly transferable visual representations via language supervision.

Original: However,learning VLP models from the scratch is prohibitive as it requires large amounts of training data and computing resource.
Revised: However,learning VLP models from the scratch is prohibitive as it requires large amounts of training data and computing resource.

Original: For example, training CLIP [31] requires 400M pairedimage-text data and several hundreds of GPUs.
Revised: For example, training CLIP [32] requires 400M pairedimage-text data and several hundreds of GPUs.

Original: ALIGN [21] further scales up to leverage alternative texts specified for descriptions of webimages.
Revised: ALIGN [22] further scales up to leverage alternative texts specified for descriptions of webimages.

Original: While these models are often based onlarge Transformers [37], small ConvNets such as ResNet-50 [17] and MobileNet [20] are still widelyused in practice [1] and even more crucial for low-resource environments.
Revised: While these models are often based onlarge Transformers [40], small ConvNets such as ResNet-50 [17] and MobileNet [20] are still widelyused in practice [1] and even more crucial for low-resource environments.

Original: We reformulate represen-
Revised: We reformulate represen-

Original: tation learning in terms of knowledge transfer from a large pre-trained model to a small practicalmodel.
Revised: tation learning in terms of knowledge transfer from a large pre-trained model to a small practicalmodel.

Original: Large pre-trained models exhibit strong alignments between different modalities.
Revised: Large pre-trained models exhibit strong alignments between different modalities.

Original: CLIP learns visualconcepts from natural language supervision, mapping image and text into the same vector space.
Revised: CLIP learns visualconcepts from natural language supervision, mapping image and text into the same vector space.

Original: Astheir training data is not only huge but inaccessible, however, conventional knowledge distillationbased on instance similarity is no longer a viable option.
Revised: Astheir training data is not only huge but inaccessible, however, conventional knowledge distillationbased on instance similarity is no longer a viable option.

Original: Instead, we propose cross-modal similaritymatching (CSM).
Revised: Instead, we propose cross-modal similaritymatching (CSM).

Original: Imagine your goal is to learn a high quality representation for an input dog imagein CIFAR-10 as in Figure 2.
Revised: Imagine your goal is to learn a high quality representation for an input dog imagein CIFAR-10 as in Figure 2.

Original: Since CLIP was trained on numerous image-caption pairs, angulardistances from the dog image embedding to the caption embeddings of other anchor prompt textssuch as “a photo of cat” or “a photo of horse” must comprehensively preserve their visual differences.
Revised: Since CLIP was trained on numerous image-caption pairs, angulardistances from the dog image embedding to the caption embeddings of other anchor prompt textssuch as “a photo of cat” or “a photo of horse” must comprehensively preserve their visual differences.

Original: By training the student to preserve angular relations witnessed from the teacher, our model achievesnear benchmark performance without accessing the original data for training CLIP.
Revised: By training the student to preserve angular relations witnessed from the teacher, our model achievesnear benchmark performance without accessing the original data for training CLIP.

Original: To measure the effects of different amounts of supervisions, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of the input text prompts.
Revised: To measure the effects of different amounts of supervisions, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of the input text prompts.

Original: We find that lexicalambiguity in prompt texts can lead to semantically incorrect text embeddings.
Revised: We find that lexicalambiguity in prompt texts can lead to semantically incorrect text embeddings.

Original: This may result inunexpected discrepancies of image-text alignment in the teacher’s embedding space.
Revised: This may result inunexpected discrepancies of image-text alignment in the teacher’s embedding space.

Original: Also, it is knownthat the zero-shot performance of CLIP can be improved by designing task-specific prompt texts.
Revised: Also, it is knownthat the zero-shot performance of CLIP can be improved by designing task-specific prompt texts.

Original: Inspired by this, we design CPA that extends the basic prompt of CLIP to better encode prototypicalanchor representations.
Revised: Inspired by this, we design CPA that extends the basic prompt of CLIP to better encode prototypicalanchor representations.

Original: Our experimental result shows that our BeamCLIP achieves the strongest and near benchmarkperformance on ImageNet-1K Top-1 linear probe accuracy when using most popular ResNet-50 asthe student network.
Revised: Our experimental result shows that our BeamCLIP achieves the strongest and near benchmarkperformance on ImageNet-1K Top-1 linear probe accuracy when using most popular ResNet-50 asthe student network.

Original: We also compare the effectiveness of BeamCLIP against zero-shot transferlearning.
Revised: We also compare the effectiveness of BeamCLIP against zero-shot transferlearning.

Original: Further, we provide ablation study results to show how much each component contribute tothe performance.
Revised: Further, we provide ablation study results to show how much each component contribute tothe performance.

Original: Contributions of this paper can be summarized as follows:
Revised: Contributions of this paper can be summarized as follows:

Original: • We propose a method ( BeamCLIP ) that can effectively transfer the representations of alarge pre-trained multimodal model (CLIP ViT-B/16) into a small target model (ResNet-50).
Revised: • We propose a method ( BeamCLIP ) that can effectively transfer the representations of alarge pre-trained multimodal model (CLIP ViT-B/16) into a small target model (ResNet-50).

Original: To achieve this, we introduce cross-modal similarity matching (CSM) and context-based
Revised: To achieve this, we introduce cross-modal similarity matching (CSM) and context-based

Original: prompt augmentation (CPA).
Revised: prompt augmentation (CPA).

Original: (Figure 2).
Revised: (Figure 2).

Original: • We empirically show that BeamCLIP can provide favorably good ImageNet linear probe ac-
Revised: • We empirically show that BeamCLIP can provide favorably good ImageNet linear probe ac-

Original: curacy (74.8%), while outperforming the state-of-the-art self-supervised methods (SimCLR:
Revised: curacy (74.8%), while outperforming the state-of-the-art self-supervised methods (SimCLR:

Original: 65.6%) and closing the gap with supervised learning (76.2%).
Revised: 65.6%) and closing the gap with supervised learning (76.2%).

Original: (Figure 1 and Table 2).
Revised: (Figure 1 and Table 2).

Original: • We also show that BeamCLIP (from CLIP-ViT to ResNet50) can significantly improve
Revised: • We also show that BeamCLIP (from CLIP-ViT to ResNet50) can significantly improve

Original: zero-shot transfer learning accuracy, especially when the zero-shot accuracy (CLIP-RN50)
Revised: zero-shot transfer learning accuracy, especially when the zero-shot accuracy (CLIP-RN50)

Original: is relatively week.
Revised: is relatively week.

Original: (Table 3 and Table 4)
Revised: (Table 4 and Table 5)

Original:  Related Work
Revised:  Related Work

Original: Vision and language pre-trainig.
Revised: Vision and language pre-trainig.

Original: Vision and language pre-training (VLP) aims to jointly learnvision and language representations that can be transferred to the downstream tasks such as visualquestion answering (VQA), image captioning, and vision and language navigation (VLN).
Revised: Vision and language pre-training (VLP) aims to jointly learnvision and language representations that can be transferred to the downstream tasks such as visualquestion answering (VQA), image captioning, and vision and language navigation (VLN).

Original: There are
Revised: There are

Original: BERT-based vision and language models such as VLBERT [32], ViLBERT [25], and UNITER [7].
Revised: BERT-based vision and language models such as VLBERT [35], ViLBERT [26], and UNITER [7].

Original: Also, there are contrastive learning-based models such as CLIP [31] and ALIGN [21].
Revised: Also, there are contrastive learning-based models such as CLIP [32] and ALIGN [22].

Original: These modelsuse contrastive loss [28] to learn aligned vision and language representations by performing a task ofmatching a large-scale image and text pairs.
Revised: These modelsuse contrastive loss [29] to learn aligned vision and language representations by performing a task ofmatching a large-scale image and text pairs.

Original: The BeamCLIP aims to transfer the rich representationsof large-scale vision and language pre-trained models such as CLIP and ALIGN to a small targetmodel.
Revised: The BeamCLIP aims to transfer the rich representationsof large-scale vision and language pre-trained models such as CLIP and ALIGN to a small targetmodel.

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: Self-supervised learning (SSL) aims to learn highly transferable representations by using unlabeled data.
Revised: Self-supervised learning (SSL) aims to learn highly transferable representations by using unlabeled data.

Original: In computer vision, at the early stage, task-specific self-supervisedmethods were introduced.
Revised: In computer vision, at the early stage, task-specific self-supervisedmethods were introduced.

Original: These include Context Prediction [11], Rotation Prediction [14], and Colorization [40].
Revised: These include Context Prediction [11], Rotation Prediction [14], and Colorization [43].

Original: More recently, contrastive learning-based methods were introduced as a task-agnosticapproach.
Revised: More recently, contrastive learning-based methods were introduced as a task-agnosticapproach.

Original: These include SimCLR [4] and MoCo-v2 [18].
Revised: These include SimCLR [4] and MoCo-v2 [18].

Original: However, since contrastive self-supervisedmethods require a large batch size, non-contrastive methods have been introduced.
Revised: However, since contrastive self-supervisedmethods require a large batch size, non-contrastive methods have been introduced.

Original: These include
Revised: These include

Original: SwAV [3], BYOL [16], and SimSiam [5].
Revised: SwAV [3], BYOL [16], and SimSiam [5].

Original: In this paper, we empirically show that the BeamCLIP can provide better visual representations than the state-of-the-art SSL methods by leveraging a large-scalepre-trained multimodal model.
Revised: In this paper, we empirically show that the BeamCLIP can provide better visual representations than the state-of-the-art SSL methods by leveraging a large-scalepre-trained multimodal model.

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [19] aims to transfer rich knowledge from astrong teacher model to a target student model.
Revised: [19] aims to transfer rich knowledge from astrong teacher model to a target student model.

Original: In a conventional setting, it encourages the studentmodel to mimic the task-specific prediction of the teacher model.
Revised: In a conventional setting, it encourages the studentmodel to mimic the task-specific prediction of the teacher model.

Original: As the student model is trainedto predict the same probability distribution over pre-defined classes as the teacher model’s, using
Revised: As the student model is trainedto predict the same probability distribution over pre-defined classes as the teacher model’s, using

Original: Kullback-Leibler (KL) divergence is a natural metric to measure the error between the two models.
Revised: Kullback-Leibler (KL) divergence is a natural metric to measure the error between the two models.

Original: For a classification task, the loss function can be formulated as follows:
Revised: For a classification task, the loss function can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: The first term indicates the supervised loss, where p i denotes the one-hot labels and H ( p, q ) denotescross-entropy.
Revised: The first term indicates the supervised loss, where p i denotes the one-hot labels and H ( p, q ) denotescross-entropy.

Original: The second term is the distillation loss, where p Ti and p Si are the softmax predictionsof the teacher and student models, respectively.
Revised: The second term is the distillation loss, where p Ti and p Si are the softmax predictionsof the teacher and student models, respectively.

Original: Similarity-based knowledge distillation.
Revised: Similarity-based knowledge distillation.

Original: Recently, similarity-based knowledge distillation suchas SEED [13], OSS [8], and ISD [34] was introduced in the context of self-supervised learning(SSL).
Revised: Recently, similarity-based knowledge distillation suchas SEED [13], OSS [8], and ISD [37] was introduced in the context of self-supervised learning(SSL).

Original: SEED [13] showed that the linear probe accuracy of a small student (ResNet-18) can beimproved by transferring the representations of a larger teacher (ResNet-50) pre-trained by SSLmethods such as MoCo-v2 [6].
Revised: SEED [13] showed that the linear probe accuracy of a small student (ResNet-18) can beimproved by transferring the representations of a larger teacher (ResNet-50) pre-trained by SSLmethods such as MoCo-v2 [6].

Original: Unlike this, OSS [8] aims to transfer representations of an evolvingteacher (ResNet-50) into a smaller student (ResNet-18) on the fly.
Revised: Unlike this, OSS [8] aims to transfer representations of an evolvingteacher (ResNet-50) into a smaller student (ResNet-18) on the fly.

Original: Unlike SEED and OSS, ISD [34]considered the same size student and teacher network (ResNet-18), and showed a student can learnvisual representations by iterativly distilling the similarity of teacher’s representations.
Revised: Unlike SEED and OSS, ISD [37]considered the same size student and teacher network (ResNet-18), and showed a student can learnvisual representations by iterativly distilling the similarity of teacher’s representations.

Original: These worksare closely related to our work.
Revised: These worksare closely related to our work.

Original: Unlike these works, the BeamCLIP aims to transfer rich vision andlanguage representations of large-scale pre-trained models such as CLIP-ViT/16 [31] into a smallernetwork such as ResNet-50.
Revised: Unlike these works, the BeamCLIP aims to transfer rich vision andlanguage representations of large-scale pre-trained models such as CLIP-ViT/16 [32] into a smallernetwork such as ResNet-50.

Original: Prompt engineering.
Revised: Prompt engineering.

Original: Recently, researchers showed that prompt engineering [2] is surprisinglyeffective at improving the performance of large-scale language models (LLMs) on downstreamtasks without fine-tuning.
Revised: Recently, researchers showed that prompt engineering [2] is surprisinglyeffective at improving the performance of large-scale language models (LLMs) on downstreamtasks without fine-tuning.

Original: Prompts are input texts of language models that usually consists of atask description or several examples.
Revised: Prompts are input texts of language models that usually consists of atask description or several examples.

Original: To further simplify prompt engineering, prompt tuning [23]proposed to add k learnable tokens to the input texts, while having language models frozen.
Revised: To further simplify prompt engineering, prompt tuning [24]proposed to add k learnable tokens to the input texts, while having language models frozen.

Original: Similarto GPT-3, it is known that the zero-shot performance of CLIP [31] can be improved by designingthe prompt texts to each task.
Revised: Similarto GPT-3, it is known that the zero-shot performance of CLIP [32] can be improved by designingthe prompt texts to each task.

Original: For example, on satellite image classification datasets, "A satellitephoto of a {label}" provides better performance than the default "A photo of a {label}" .
Revised: For example, on satellite image classification datasets, "A satellitephoto of a {label}" provides better performance than the default "A photo of a {label}" .

Original: Inspired by this, we propose context-based prompt augmentation that extends the basic prompt of
Revised: Inspired by this, we propose context-based prompt augmentation that extends the basic prompt of

Original: CLIP to better encode prototypical text anchor representations by alleviating the lexical ambiguity ofclass label texts.
Revised: CLIP to better encode prototypical text anchor representations by alleviating the lexical ambiguity ofclass label texts.

Original:  Method
Revised:  Method

Original: Problem formulation.
Revised: Problem formulation.

Original: Formally, our problem is to transfer aligned cross-modal representations ofa strong teacher model f Tϕ ( · ) into a target student model f Sθ ( · ) with unlabeled data D u
Revised: Formally, our problem is to transfer aligned cross-modal representations ofa strong teacher model f Tϕ ( · ) into a target student model f Sθ ( · ) with unlabeled data D u

Original: = { x i } Ni =1 .
Revised: = { x i } Ni =1 .

Original: Given each unlabeled image x i , we formulate representation transfer as a regression task that matchesteacher representations f
Revised: Given each unlabeled image x i , we formulate representation transfer as a regression task that matchesteacher representations f

Original: Tϕ ( x i ) to a student’s f
Revised: Tϕ ( x i ) to a student’s f

Original: Sθ ( x i ) .
Revised: Sθ ( x i ) .

Original: As the student network is parameterized by θ ,the learning objective is
Revised: As the student network is parameterized by θ ,the learning objective is

Original: [Equation]
Revised: [Equation]

Original: Normalizing the representations via l 2 -normalization ( i.e., q i =
Revised: Normalizing the representations via l 2 -normalization ( i.e., q i =

Original: f Sθ ( x i ) /
Revised: f Sθ ( x i ) /

Original: ∥
Revised: ∥

Original: f Sθ ( x i ) ∥ 2 and k i =
Revised: f Sθ ( x i ) ∥ 2 and k i =

Original: f Tϕ ( x i ) /
Revised: f Tϕ ( x i ) /

Original: ∥
Revised: ∥

Original: f Tϕ ( x i ) ∥ 2 ) leads to the following simplification:
Revised: f Tϕ ( x i ) ∥ 2 ) leads to the following simplification:

Original: [Equation]
Revised: [Equation]

Original: The problem now involves maximizing the cosine similarity between l 2 -normalized representationsfrom teacher and student models.
Revised: The problem now involves maximizing the cosine similarity between l 2 -normalized representationsfrom teacher and student models.

Original: The BeamCLIP first measures the normalized cross-modal similarity of the query image compared to anchor text representations in the teacher’s embedding space.
Revised: The BeamCLIP first measures the normalized cross-modal similarity of the query image compared to anchor text representations in the teacher’s embedding space.

Original: Then, it encourages the student to mimic the same cross-modal similarity in the student’s embedding space.
Revised: Then, it encourages the student to mimic the same cross-modal similarity in the student’s embedding space.

Original: To better align image representations, our method uses self-supervised pre-training of the student model.
Revised: To better align image representations, our method uses self-supervised pre-training of the student model.

Original: Finally, to avoid text ambiguity, we uses context-based prompt augmentation.
Revised: Finally, to avoid text ambiguity, we uses context-based prompt augmentation.

Original: Method overview.
Revised: Method overview.

Original: The overview of BeamCLIP is shown in Figure 2.
Revised: The overview of BeamCLIP is shown in Figure 2.

Original: The teacher model ofthe BeamCLIP consists of an image encoder f
Revised: The teacher model ofthe BeamCLIP consists of an image encoder f

Original: Tϕ ( · ) and a text encoder g Tψ ( · ) .
Revised: Tϕ ( · ) and a text encoder g Tψ ( · ) .

Original: These encoders arepre-trained under a simple task of matching images to texts with large-scale corpora.
Revised: These encoders arepre-trained under a simple task of matching images to texts with large-scale corpora.

Original: Image representations f
Revised: Image representations f

Original: Tϕ ( x i ) and text representations g Tψ ( t i ) are thus well-aligned within a cross-modal embeddingspace.
Revised: Tϕ ( x i ) and text representations g Tψ ( t i ) are thus well-aligned within a cross-modal embeddingspace.

Original: We provide the details of the BeamCLIP in the following sections.
Revised: We provide the details of the BeamCLIP in the following sections.

Original: More specifically, wedescribe how to extend the basic problem setting by leveraging the unique features of CLIP wherevision and language representations are precisely aligned.
Revised: More specifically, wedescribe how to extend the basic problem setting by leveraging the unique features of CLIP wherevision and language representations are precisely aligned.

Original: Throughout the paper, we use the notation
Revised: Throughout the paper, we use the notation

Original: CLIP-ViT/16 to denote the CLIP [31] model that uses Vision Transformer (ViT) [12] with the patchsize of 16x16 as the image encoder.
Revised: CLIP-ViT/16 to denote the CLIP [32] model that uses Vision Transformer (ViT) [12] with the patchsize of 16x16 as the image encoder.

Original: Similar to this, CLIP-RN50 denotes the CLIP model with
Revised: Similar to this, CLIP-RN50 denotes the CLIP model with

Original: ResNet-50 [17] as the image encoder.
Revised: ResNet-50 [17] as the image encoder.

Original: 3.1 Similarity-based cross-modal representation transfer
Revised: 3.1 Similarity-based cross-modal representation transfer

Original: To effectively distill cross-modal representations, we use similarity-based matching as describedabove.
Revised: To effectively distill cross-modal representations, we use similarity-based matching as describedabove.

Original: Our similarity-based representation transfer utilizes two carefully designed loss functions: (1)instance similarity matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.
Revised: Our similarity-based representation transfer utilizes two carefully designed loss functions: (1)instance similarity matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.

Original: Instance similarity matching.
Revised: Instance similarity matching.

Original: This objective is directly derived from Eq.
Revised: This objective is directly derived from Eq.

Original: Given a query imagex i , it encourages the student image encoder f Sθ ( · ) to regress the representation of the teacher imageencoder f Tϕ ( · ) .
Revised: Given a query imagex i , it encourages the student image encoder f Sθ ( · ) to regress the representation of the teacher imageencoder f T ϕ ( · ) .

Original: We apply conventional image augmentations (see Appendix B) on a query imagex i , and the same augmented image ˆ x i is fed to both the teacher and student image encoders.
Revised: We apply conventional image augmentations (see Appendix B) on a query imagex i , and the same augmented image ˆ x i is fed to both the teacher and student image encoders.

Original: Givenunlabeled query images D u
Revised: Givenunlabeled query images D u

Original: = { x i } Ni =1 , it is formulated as follows:
Revised: = { x i } Ni =1 , it is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: However, the similarity signal from a single instance is not enough to constraint the student representations.
Revised: However, the similarity signal from a single instance is not enough to constraint the student representations.

Original: For example, the topological ambiguity may occur in image encoding, since two symmetricrepresentations have the same cosine similarity compared to a single teacher representation.
Revised: For example, the topological ambiguity may occur in image encoding, since two symmetricrepresentations have the same cosine similarity compared to a single teacher representation.

Original: Weconjecture that this can be mitigated by incorporating multiple anchor points to the query points.
Revised: Weconjecture that this can be mitigated by incorporating multiple anchor points to the query points.

Original: Based on this idea, we introduce cross-modal similarity matching loss.
Revised: Based on this idea, we introduce cross-modal similarity matching loss.

Original: Cross-modal similarity matching.
Revised: Cross-modal similarity matching.

Original: To better align a student representation q i with the teacherrepresentation
Revised: To better align a student representation q i with the teacherrepresentation

Original: k i , we introduce cross-modal similarity matching (CSM) loss.
Revised: k i , we introduce cross-modal similarity matching (CSM) loss.

Original: We use multiple anchorpoints to cope with the ambiguity problem mentioned above.
Revised: We use multiple anchorpoints to cope with the ambiguity problem mentioned above.

Original: Further, we use text representations as
Revised: Further, we use text representations asanchor points, since we can easily generate prototypical anchor points by using text prompts and class

Original: [Table]
Revised: [Table]

Original: anchor points, since we can easily generate prototypical anchor points by using text prompts and classlabels.
Revised: labels.

Original: Since image and text representations are precisely aligned in CLIP, we can effectively applythis approach.
Revised: Since image and text representations are precisely aligned in CLIP, we can effectively applythis approach.

Original: More specifically, the BeamCLIP first measures the normalized image-text similarityof the query image compared to prototypical text points in the teacher’s embedding space.
Revised: More specifically, the BeamCLIP first measures the normalized image-text similarityof the query image compared to prototypical text points in the teacher’s embedding space.

Original: Then, itencourages the student to mimic the same image-text similarity in the student’s embedding space.
Revised: Then, itencourages the student to mimic the same image-text similarity in the student’s embedding space.

Original: More formally, we generate multiple anchor representations A
Revised: More formally, we generate multiple anchor representations A

Original: = { a j } Mj =1 by encoding class texts
Revised: = { a j } M j =1 by encoding class texts

Original: C = { c j } Mj =1 with the teacher text encoder g Tψ ( · ) (in other words, a j = g Tψ ( c j ) ).
Revised: C = { c j } Mj =1 with the teacher text encoder g Tψ ( · ) (in other words, a j = g Tψ ( c j ) ).

Original: To measure thesimilarity regarding to multiple anchor representations A , we define the normalized cross-modalsimilarity as follows:
Revised: To measure thesimilarity regarding to multiple anchor representations A , we define the normalized cross-modalsimilarity as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where τ is a temperature hyperparameter that is set to 0.01 in our experiments.
Revised: (5) where τ is a temperature hyperparameter that is set to 0.01 in our experiments.

Original: Then, we evaluate the cross-modal similarity distribution by using a set of normalized cross-modalsimilarities:
Revised: Then, we evaluate the cross-modal similarity distribution by using a set of normalized cross-modalsimilarities:

Original: [Equation]
Revised: [Equation]

Original: Then, the student model is optimized to mimic the normalized cross-modal similarity of the teacher’sembedding space by minimizing the cross entropy, i.e., H ( P ( k i | A ) , P ( q i | A )) .
Revised: Then, the student model is optimized to mimic the normalized cross-modal similarity of the teacher’sembedding space by minimizing the cross entropy, i.e., H ( P ( k i | A ) , P ( q i | A )) .

Original: We further minimize the entropy of normalized cross-modal similarities in the student embeddingspace i.e., H ( P ( q i | A )) .
Revised: We further minimize the entropy of normalized cross-modal similarities in the student embeddingspace i.e., H ( P ( q i | A )) .

Original: This minimization helps the student provide query representations q i that aremore attracted to anchor representations A
Revised: This minimization helps the student provide query representations q i that aremore attracted to anchor representations A

Original: = { a j } Mj =1 .
Revised: = { a j } Mj =1 .

Original: This entropy minimization is also known tobe effective in other domains such as semi-supervised learning [15].
Revised: This entropy minimization is also known tobe effective in other domains such as semi-supervised learning [15].

Original: Altogether, the CSM loss is formulated as follows:
Revised: Altogether, the CSM loss is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Final Loss.
Revised: Final Loss.

Original: The final loss of the BeamCLIP is formulated as follows:
Revised: The final loss of the BeamCLIP is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ISM is the scale hyperparameter that is set to 10 in our experiments.
Revised: where λ ISM is the scale hyperparameter that is set to 10 in our experiments.

Original: 3.2 Context-based prompt augmentation
Revised: 3.2 Context-based prompt augmentation

Original: We found that lexical ambiguity in prompt texts can lead to semantically incorrect text embeddings.
Revised: We found that lexical ambiguity in prompt texts can lead to semantically incorrect text embeddings.

Original: This may result in an unexpected discrepancy of image-text alignment in the teacher’s embeddingspace.
Revised: This may result in an unexpected discrepancy of image-text alignment in the teacher’s embeddingspace.

Original: For example, Flowers102 [27] dataset has some classes with unusual and ambiguous flowernames, such as “snapdragon”, “bird of paradise”, and “colt’s foot”.
Revised: For example, Flowers102 [28] dataset has some classes with unusual and ambiguous flowernames, such as “snapdragon”, “bird of paradise”, and “colt’s foot”.

Original: Therefore, incorrect prototypicalanchor points might be compared with a query image.
Revised: Therefore, incorrect prototypicalanchor points might be compared with a query image.

Original: To address this issue of semantic ambiguitiesin the text, we introduce context-based prompt augmentation (CPA), a data-driven approach thataugments basic prompts with contextual text such as Wikipedia descriptions or hierarchical labels.
Revised: To address this issue of semantic ambiguitiesin the text, we introduce context-based prompt augmentation (CPA), a data-driven approach thataugments basic prompts with contextual text such as Wikipedia descriptions or hierarchical labels.

Original: For prompt tuning with Wikipedia descriptions, we use the template "A photo of a {label}.{Wikipedia description} ".
Revised: For prompt tuning with Wikipedia descriptions, we use the template "A photo of a {label}.{Wikipedia description} ".

Original: We use this template for Flowers102 and Pets37 in our experiments.
Revised: We use this template for Flowers102 and Pets37 in our experiments.

Original: We provide some examples from the Flowers102 dataset in Table 1.
Revised: We provide some examples from the Flowers102 dataset in Table 1.

Original: For prompt tuning with hierarchical labels, we use the template "A photo of a {fine label}, categorized as {coarselabel} ".
Revised: For prompt tuning with hierarchical labels, we use the template "A photo of a {fine label}, categorized as {coarselabel} ".

Original: We use this template for CIFAR100 and ImageNet in our experiments.
Revised: We use this template for CIFAR100 and ImageNet in our experiments.

Original: Analogous examplesfrom CIFAR100 can be found in Table 8 of Appendix C.
Revised: Analogous examplesfrom CIFAR100 can be found in Table 8 of Appendix C.

Original: 3.3 Other details
Revised: 3.3 Other details

Original: Self-supervised pre
Revised: Self-supervised pre

Original: -training of student.
Revised: -training of student.

Original: To help the student mimic the teacher’s cross-modalembedding space better, we pre-train the student image encoder with a self-supervised method.
Revised: To help the student mimic the teacher’s cross-modalembedding space better, we pre-train the student image encoder with a self-supervised method.

Original: Sinceself-supervised pre
Revised: Sinceself-supervised pre

Original: -training such as SimCLR [4] and MoCo-V2 [18] provides a weakly clusteredembedding space based on similarities, it can be used as a better initial state for the student to mimicthe teacher’s embedding space.
Revised: -training such as SimCLR [4] and MoCo-V2 [18] provides a weakly clusteredembedding space based on similarities, it can be used as a better initial state for the student to mimicthe teacher’s embedding space.

Original: For self-supervised pre-training, we adopt SimCLR, since it is simple and effective.
Revised: For self-supervised pre-training, we adopt SimCLR, since it is simple and effective.

Original: SimCLR learnstransferable visual representations by using InfoNCE loss [29, 36] which encourages agreementbetween multiple views of the same image.
Revised: SimCLR learnstransferable visual representations by using InfoNCE loss [30, 39] which encourages agreementbetween multiple views of the same image.

Original: More specifically, InfoNCE maximizes the similaritybetween multiple views of the same image ( i.e., positive samples) and minimizes the similarity tomultiple views of all other images in a training batch ( i.e., negative samples).
Revised: More specifically, InfoNCE maximizes the similaritybetween multiple views of the same image ( i.e., positive samples) and minimizes the similarity tomultiple views of all other images in a training batch ( i.e., negative samples).

Original: InfoNCE loss of
Revised: InfoNCE loss of

Original: SimCLR can be formulated as follows:
Revised: SimCLR can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: (9) where h Si ∈ R 128 is a projection of a student representation q i ∈ R 512 , τ is a temperature hyperparameter that is set to 0.1, 

Revised: (9) where h Si ∈ R 128 is a projection of a student representation q i ∈ R 512 , τ is a temperature hyperparameter that is set to 0.1,

Original: [ k ̸
Revised: [ k ̸

Original: = i ] is an indicator function whose value is 1 if k ̸
Revised: = i ] is an indicator function whose value is 1 if k ̸

Original: = i , and B is a batchsize.
Revised: = i , and B is a batchsize.

Original: Here, h i and h i ′ are projections of multiple views of the same input images x i .
Revised: Here, h i and h i ′ are projections of multiple views of the same input images x i .

Original: Optimization.
Revised: Optimization.

Original: For optimization we use SGD with cosine annealing schedule (SGDR) [24].
Revised: For optimization we use SGD with cosine annealing schedule (SGDR) [25].

Original: Tostabilize training, we use a momentum encoder that updates its weights via exponential movingaverage (EMA) [18, 16].
Revised: Tostabilize training, we use a momentum encoder that updates its weights via exponential movingaverage (EMA) [18, 16].

Original: The momentum encoder of a student θ ˆ S is updated using the following rule:
Revised: The momentum encoder of a student θ ˆ S is updated using the following rule:

Original: [Equation]
Revised: [Equation]

Original: where θ S is the image encoder of a student model and m is a momentum hyperparameter that is set0.99 in our experiments.
Revised: where θ S is the image encoder of a student model and m is a momentum hyperparameter that is set0.99 in our experiments.

Original:  Experiments
Revised:  Experiments

Original: Downstream datasets.
Revised: Downstream datasets.

Original: We evaluate the BeamCLIP on six standard benchmark datasets: CIFAR10[22], CIFAR100 [22], STL10 [9], Flowers102 [27], Pets37 [30], and ImageNet-1K [10].
Revised: We evaluate the BeamCLIP on six standard benchmark datasets: CIFAR10[23], CIFAR100 [23], STL10 [9], Flowers102 [28], Pets37 [31], and ImageNet-1K [10].

Original: Followingconvention, we split the datasets into train, validation, and test sets.
Revised: Followingconvention, we split the datasets into train, validation, and test sets.

Original: Then, we use train set for transfer,and test set for evaluation.
Revised: Then, we use train set for transfer,and test set for evaluation.

Original: For ImageNet, we use the validation set as a test set, since its test set doesnot provide labels.
Revised: For ImageNet, we use the validation set as a test set, since its test set doesnot provide labels.

Original: More details on the datasets can be found in Appendix A.
Revised: More details on the datasets can be found in Appendix A.

Original: 4.1 Representation transfer
Revised: 4.1 Linear probe accuracy on ImageNet-1K

Original: ImageNet linear probe accuracy.
Revised: BeamCLIP-RN50.

Original: We compare the BeamCLIP with various self-supervised methods in terms of linear probe accuracy on ImageNet-1K. Following the conventional protocol, weuse ResNet-50 [17] as the base encoder and evaluate the learned representations by using logisticregression.
Revised: We compare the BeamCLIP with various self-supervised methods in terms oflinear probe accuracy on ImageNet-1K. Following the conventional protocol, we use ResNet-50[17] as the base encoder and evaluate the learned representations by using logistic regression.

Original: We use LBFGS algorithm [41] for logistic regression.
Revised: Weuse LBFGS algorithm [44] for logistic regression.

Original: Its hyperparameter C is determinedthrough coarse-grained hyperparameter search on the validation split.
Revised: Its hyperparameter C is determined throughcoarse-grained hyperparameter search on the validation split.

Original: And, the accuracy is evaluatedin the test split.
Revised: And, the accuracy is evaluated in thetest split.

Original: We found that it provides the best linear probe accuracy when C is set to 100.
Revised: We found that it provides the best linear probe accuracy when C is set to 100.

Original: Weperform our experiments on 8 NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training.
Revised: We performour experiments on 8 NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training.

Original: Table 2 shows the comparison of the BeamCLIP with vision-only self-supervised methods such as
Revised: Table 2 shows the comparison of the BeamCLIP with vision-only self-supervised methods such as

Original: SimCLR [4], MoCo-v2 [6], SwAV [3], BYOL [16], and SimSiam [5].
Revised: SimCLR [4], MoCo-v2 [6], SwAV [3], BYOL [16], and SimSiam [5].

Original: The BeamCLIP provides bettervisual representation by achieving 74.8% top-1 linear probe accuracy on ImageNet-1K [10].
Revised: The BeamCLIP provides bettervisual representation by achieving 74.8% top-1 linear probe accuracy on ImageNet-1K [10].

Original: Whileself-supervised methods take long training epochs to achieve comparable accuracy, BeamCLIP-RN

Revised: Whileself-supervised methods take long training epochs to achieve comparable accuracy, BeamCLIP-RN

Original: [Table]

Original: [Table]
Revised: [Table]

Original: achieves better accuracy with less training epochs.
Revised: achieves better accuracy with less training epochs.

Original: Also, note that BeamCLIP-RN50’s representationsprovide better accuracy than CLIP-RN50’s representations. 
Revised: Also, note that BeamCLIP-RN50’s representationsprovide better accuracy than CLIP-RN50’s representations. BeamCLIP-RN18.

Revised: We also provide the ImageNet-1K linear probe accuracy on BeamCLIP-RN18representations in Table 10 of Appendix C.

Revised: The BeamCLIP-RN18 achieves 62.7% linear probeaccuracy, while outperforming the SSL methods such as SimCLR (47.1%) and MoCo-v2 (48.6%).

Original: 4.2 Transfer learning
Revised: 4.2 Zero-shot accuracy on ImageNet-1K

Revised: To check if the BeamCLIP also inherits the powerful zero-shot capability of CLIP, we comparezero-shot accuracy of CLIP variants on ImageNet-1K. For zero-shot measure, we use CC-3M [34]and ImageNet-21K (12M samples) [33] that do not have overlap with Imagenet-1K. Table 3 shows thecomparison of zero-shot accuracy. The BeamCLIP-RN50 achieves about 53.6% zero-shot accuracythat is highly comparable with CLIP RN-50 (59.6%). 4.3 Transfer learning accuracy on various datasets

Original: Setting.
Revised: Setting.

Original: We evaluate how effectively the BeamCLIP transfers CLIP-ViT representations into astudent model by evaluating classification accuracy on various datasets (see Table 3).
Revised: We evaluate how effectively the BeamCLIP transfers CLIP-ViT representations into astudent model by evaluating classification accuracy on various datasets (see Table 4).

Original: We choose
Revised: We choose

Original: ResNet-50 [17] as the student model to compare the distilled target model with CLIP-RN50.
Revised: ResNet-50 [17] as the student model to compare the distilled target model with CLIP-RN50.

Original: Also,
Revised: Also,

Original: ResNet-50 is conventionally used in evaluating linear probe accuracy on ImageNet-1K.
Revised: ResNet-50 is conventionally used in evaluating linear probe accuracy on ImageNet-1K.

Original: However, the
Revised: However, the

Original: BeamCLIP can adopt any architecture, not just ResNet-50.
Revised: BeamCLIP can adopt any architecture, not just ResNet-50.

Original: As baselines, we choose two representative distillation methods among many methods: (1) conventional knowledge distillation (KD) [19] and (2) contrastive representation distillation (CRD) [35].
Revised: As baselines, we choose two representative distillation methods among many methods: (1) conventional knowledge distillation (KD) [19] and (2) contrastive representation distillation (CRD) [38].

Original: Since conventional KD aims to mimic the task-specific predictions of the teacher model unlike the
Revised: Since conventional KD aims to mimic the task-specific predictions of the teacher model unlike the

Original: BeamCLIP , we apply the KL divergence on minimizing the cross-modal similarity distribution ( i.e.,
Revised: BeamCLIP , we apply the KL divergence on minimizing the cross-modal similarity distribution ( i.e.,

Original: P ( q i | A ) and P ( k i | A ) ), instead of the Cross-entropy (CE).
Revised: P ( q i | A ) and P ( k i | A ) ), instead of the Cross-entropy (CE).

Original: CRD proposes a variant of InfoNCE lossfor representation distillation, which we apply on normalized representations ( i.e., q i and k i ).
Revised: CRD proposes a variant of InfoNCE lossfor representation distillation which we apply on normalized representations ( i e., q i and k i )

Original: Thedetails of each methodcan befound in the related work section 2.
Revised: Thedetails of each method be in the related work section achieve such a high zero-

Revised: , CLIP very large image-

Original: Results.
Revised: Instead the BeamCLIP can achieve the comparable zero- by effectively transferring representations [Table] -1K. We denoted with ∗ in cases our student model outperforms the teacher model. [Table]

Revised: r

Revised: [Table]

Original:  Table 3 shows a comparison of teacher and student accuracy on various datasets.
Revised: P e t s [Table] Results. Table 4 shows a comparison of teacher and student accuracy on various datasets.

Original: We empirically demonstrate that the BeamCLIP can effectively transfer vision and language representationsof a large teacher model (CLIP-ViT/16) into a small student model (ResNet-50), while achievingcomparable or better classification accuracy.
Revised: We empirically demonstrate that the BeamCLIP can effectively transfer vision and language representationsof a large teacher model (CLIP-ViT/16) into a small student model (ResNet-50) [Table] while achievingcomparable or better classification accuracy.

Original: We find that the KL divergence used in conventionalknowledge distillation (KD) is not effective in transferring CLIP-ViT representations.
Revised: We find that the KL divergence used in conventionalknowledge distillation (KD) is not effective in transferring CLIP-ViT representations.

Original: Also, thecontrastive learning-based approach is not effective.
Revised: Also, thecontrastive learning-based approach is not effective.

Original: Unlike this, the BeamCLIP can effectively transfer CLIP-ViT/16 representations into ResNet-50, achieving very high accuracy that is comparableor better than the teacher accuracy.
Revised: Unlike this, the BeamCLIP can effectively transfer CLIP-ViT/16 representations into ResNet-50, achieving very high accuracy that is comparableor better than the teacher accuracy.

Original: For example, for ImageNet-1K, when compared to zero-shot
Revised: For example, for ImageNet-1K, when compared to zero-shot

Original: CLIP-RN50, BeamCLIP-RN50 provides about 10.5% better accuracy on the same size model.
Revised: CLIP-RN50, BeamCLIP-RN50 provides about 10.5% better accuracy on the same size model.

Original: Whencompared to CLIP-ViT/16, BeamCLIP-RN50 provides about 1.8% better accuracy on the muchsmaller model.
Revised: Whencompared to CLIP-ViT/16, BeamCLIP-RN50 provides about 1.8% better accuracy on the muchsmaller model.

Original: KD simply minimizes the error between single instances.
Revised: KD simply minimizes the error between single instances.

Original: We conjecture that thecross-modal similarity to multiple anchor points introduced in the BeamCLIP helps the studentpreserve the topology of the teacher’s embedding space.
Revised: We conjecture that thecross-modal similarity to multiple anchor points introduced in the BeamCLIP helps the studentpreserve the topology of the teacher’s embedding space.

Original: Also, note that context-based prompt augmentation helps achieve better accuracy after distillation.
Revised: Also, note that context-based prompt augmentation helps achieve better accuracy after distillation.

Original: Since Flowers102 has many ambiguous labels, our experiment shows that text prompt augmentationsignificantly increases the student’s accuracy compared to the teacher’s accuracy. 
Revised: Since Flowers102 has many ambiguous labels, our experiment shows that text prompt augmentationsignificantly increases the student’s accuracy compared to the teacher’s accuracy. [Table]

Original: 4.3 Ablation study
Revised: 4.4 Ablation study

Original: Table 4 shows the ablation study results of the BeamCLIP .
Revised: Table 5 shows the ablation study results of the BeamCLIP .

Original: Our empirical findings are as follows:(1) instance similarity distillation is not enough by itself to preserve the topology of the teacher’sembedding space.
Revised: Our empirical findings are as follows:(1) instance similarity distillation is not enough by itself to preserve the topology of the teacher’sembedding space.

Original: (2) cross-modal similarity distillation compared to multiple anchor points helpsthe student mimic the teacher’s embedding space.
Revised: (2) cross-modal similarity distillation compared to multiple anchor points helpsthe student mimic the teacher’s embedding space.

Original: (3) self-supervised pre-training of the studentsignificantly helps the student mimic the teacher’s embedding space.
Revised: (3) self-supervised pre-training of the studentsignificantly helps the student mimic the teacher’s embedding space.

Original: (4) Entropy minimization(EntMin) helps to improve the accuracy.
Revised: (4) Entropy minimization(EntMin) helps to improve the accuracy.

Original: (5) Context-based prompt augmentation helps measure thesimilarity more precisely.
Revised: (5) Context-based prompt augmentation helps measure thesimilarity more precisely.

Original: Figure 3 shows qualitative results via top-5 text-image retrieval. A red rectangle denotes an incorrectresult.
Revised: As shown in the table, Flowers102 dataset is sensitive to self-supervisedpre-training of student.

Original: CLIP-RN50 provides many incorrect results since  it shows relatively low zero-shot accuracy.
Revised: We conjecture that since Flowers102 dataset has only 1020 training samplesfor the 102 classes, it is not enough to probe the teacher’s representation space.

Original:  In contrast, BeamCLIP-RN50 provides much improved results, since it is distilled from CLIPViT/16 which shows very high zero-shot accuracy. More interestingly, BeamCLIP-RN50 providessurprisingly good text-image retrieval results, even though unseen text prompts such as "a photoof {pink rose}" or "a photo of {yellow rose}" are given.
Revised:  Conclusion and Limitations

Original: Conclusion
Revised: Conclusion.

Original: Our BeamCLIP provides an effective and accessible framework that distills large-scale multimodalrepresentations of CLIP.
Revised: Our BeamCLIP provides an effective and accessible framework that distills large-scalemultimodal representations of CLIP.

Original: Combining our similarity-based distillation and context-based promptaugmentation with self-supervised training, we empirically demonstrate that the BeamCLIP caneffectively transfer the vision and language representations of a large teacher model (CLIP-ViT/16)into a smaller student model (ResNet-50) while achieving comparable or better classification accuracy. We also show that the BeamCLIP provides better visual representations by achieving 
Revised: Combining our similarity-based distillation and context-basedprompt augmentation with self-supervised training, we empirically demonstrate that the BeamCLIPcan effectively transfer the vision and language representations of a large teacher model (CLIPViT/16) into a smaller student model (ResNet-50) while achieving comparable or better classificationaccuracy. We also show that the BeamCLIP provides better visual representations by achieving 74.8%top-1 linear probe accuracy on ImageNet-1K when compared to vision-only self-supervised methods(SimCLR: 65.6%, supervised: 76.2%).

Original: [Table] [Table] [Figure] results denotes an incorrect result.
Revised: Limitations.

Original: CLIP provides many incorrect results, since it shows relatively low zero-shot accuracy.
Revised: With the help of rich representations of pre-trained CLIP, the BeamCLIP can learnbetter representations than SSL methods.

Original: In provides much improved results, since it is distilled very high While we mostly experiment with ResNet-50 for its popularity, we expect our the BeamCLIP isversatile enough to plug and play with various lightweight student models, which will be interestingfor future studies.
Revised: However, since SSL methods can increase the performanceat longer training epochs, the performance margin may be decreased in such a setting. Anothershortcoming is that context-based prompt augmentations may require additional engineering efforts.

Original: Broader impact
Revised: Broader impact

Original: This research aims to advance transfer learning by leveraging large-scale pre-trained models such as CLIP.
Revised: This research aims to provide an simple and effective way to leverage CLIP for representationlearning. With the help of CLIP, the BeamCLIP can learn better representations than self-supervisedlearning (SSL) methods.

Original: Since large-scale pre-training is expensive, it is highly required to effectively reuse representative pre-trained models.
Revised: Since training CLIP requires very large data and hundreds of GPUs, it isimportant to provide a way to effectively reuse the pre-trained CLIP rather than training from scratchon a target model.

Revised: We believe that BeamCLIP can help to save cost and time.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors a

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: ) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] See Section 1.(b)
Revised: [Yes] See Section 1. b

Original:  Did you describe the limitations of your work?
Revised: ) Did you describe the limitations of your work?

Original: [Yes] See Section 4 and Section 5.(c)
Revised: [Yes] See Section 4 and Section 5.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 5.(d)
Revised: Yes] SeeSection 5.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [No] Our work doesnot include theoretical results.(b)
Revised: [No] Our work doesnot include theoretical results.(b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [No] Our work does notinclude theoretical results.3.
Revised: [No] Our work does notinclude theoretical results.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide the code, data, and instructions in the supplemental material.
Revised: [Yes] We provide thecode, data, and instructions in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4 and Appendix B.(c)
Revised: [Yes] See Section 4 and Appendix B.

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 2, Section 3, and Section 4.
Revised: [Yes] See Section 2,Section 3, and Section 4.(b)

Original: (b) Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [No] We only used public benchmark datasets and open-sourced software.
Revised: [No] We only used public benchmarkdatasets and open-sourced software.

Original: See Section 4.
Revised: See Section 4.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: We did not use new assets.
Revised: We did not use new assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] We only used public benchmark datasets.
Revised: [Yes] We only used public benchmark datasets.

Original: See Section 4.
Revised: See Section 4.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No] We did not use any data containing personally
Revised: [No] We did not use any data containing personally

Original: identifiable information or offensive content.
Revised: identifiable information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots,
Revised: (a) Did you include the full text of instructions given to participants and screenshots,

Original: if applicable?
Revised: if applicable?

Original: [No] We did not use crowdsourcing or conduct research with human
Revised: [No] We did not use crowdsourcing or conduct research with human

Original: subjects.
Revised: subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [No] We did not use crowdsourcing or conduct
Revised: [No] We did not use crowdsourcing or conduct

Original: research with human subjects.
Revised: research with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [No] We did not use crowdsourcing or conduct
Revised: [No] We did not use crowdsourcing or conduct

Original: research with human subjects.
Revised: research with human subjects.

Original: A Datasets Details
Revised: A Datasets Details

Revised: We demonstrate the effectiveness of the BeamCLIP by using six downstream datasets.

Revised: Table 5 showsthe details of the downstream datasets.

Original: [Table]
Revised: [Table]

Original: B Method Details
Revised: B Method Details

Original: B.1 Image Augmentation
Revised: In this section

Original: [Table]
Revised: [Table]

Original: B.2 Context-based prompt augmentation 
Revised: we provide some details of the BeamCLIP . More specifically, we provide the detailsof two main contributions that are (1) cross-modal similarity matching (CSM) and (2) context-basedprompt augmentation (CPA). Also, we provide the other implementation details such as imageaugmentation, similarity smoothing, model hyperparameters, etc. B.1 Image augmentation details

Revised: We use conventional image augmentation when performing representation transfer by using unlabeledimages in downstream datasets.

Revised: Table 6 provides a list of image augmentation used for unsupervisedrepresentation transfer on downstream datasets.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: Normalize - B.2 Cross-modal similarity matching details Cross-modal similarity matching (CSM) is the main method of the BeamCLIP . To make the conceptof CSM clearer, we provide an illustration of CSM in Figure 3.

Original: Text Prompt baby "
Revised: B.3 Context-based prompt augmentation details

Original: A photo of a {baby}, categorized as {people}. " beaver "A photo of a {beaver}, categorized as {aquatic mammals}. " bee "A photo of a {bee}, categorized as {insect}."
Revised: To prepare for better text anchor embeddings for unsupervised representation transfer, we introducecontext-based prompt augmentation (CPA). To make the concept of CPA clearer, we provide anillustration of CPA in Figure 4.

Original: B.3 Other Implementation Details
Revised: B.4 Other implementation details

Original: Similarity Smoothing.
Revised: Similarity Smoothing.

Original: To improve the effectiveness of distillation, we apply Label Smoothing (LS)[33] to the cross-modal similarity distillation loss.
Revised: To improve the effectiveness of distillation, we apply Label Smoothing (LS)[36] to the cross-modal similarity distillation loss.

Original: Recent works [26, 39] show that Label Smoothinghelps knowledge distillation.
Revised: Recent works [27, 42] show that Label Smoothing helps knowledge distillation.

Original: To apply Label Smoothing, we determine the most similar anchorrepresentation as follows:
Revised: To apply Label Smoothing, we determine the most similar anchorrepresentation as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Then, we generate a modified cross-modal similarity distribution:
Revised: [Table] Then [Table] we generate a modified cross-modal similarity distribution:

Original: [Equation]
Revised: [Equation]

Original: (12) where 1 [ j = j ∗ ] is the indicator function whose value is 1 if j = j ∗ , M is the number of anchors, andα is the smoothing hyperparameter that is set to 0.2 in our experiments.
Revised: (12) where 1 [ j = j ∗ ] is the indicator function whose value is 1 if j = j ∗ , M is the number of anchors, andα is the smoothing hyperparameter that is set to 0.2 in our experiments [Table]

Original: B.4 Model Hyperparameters
Revised: B.5 Model hyperparameters

Revised: Table 9 provides the summary of model hyperparamters.

Revised: We use the same hyperparameters on alldownstream datasets if not explicitly declared

Original: [Table] 
Revised: [Table] [Table]

Revised: C Additional Experiment Results

Revised: In this section, we provide additional experiment results.

Revised: Especially, we provide some experiment results to discuss the characteristics of unsupervised representation transfer with cross-modal similaritymatching.

Revised: C.1 ImageNet-1K linear probe accuracy on ResNet-50

Revised: In the experiment section, we provide ImageNet-1K top-1 linear probe accuracy on ResNet-50to measure the effectiveness of the representations of the BeamCLIP .

Revised: We show that the visualrepresentations transferred from a large pre-trained multimodal model (i.e., CLIP) outperform thestate-of-the-art self-supervised learning methods such as SimCLR (65.6%) and MoCo-v2 (67.5%),while closing the gap with supervised learning (76.2%).

Revised: In this subsection, we additionally providethe linear probe accuracy by increasing the training epochs.

Revised: As shown in Figure 5, it slightly increases(from 74.8% to 75.0%), as the training epochs increases from 200 to 400.

Revised: Also, we provide the learning curve of transfer learning on ImageNet in Figure 6.

Revised: C.2 ImageNet-1K linear probe on ResNet-18

Revised: To check if the BeamCLIP can transfer CLIP representations into smaller models than ResNet-50(24M), we also measure ImageNet-1K Top-1 linear probe accuracy on ResNet-18 (11M).

Revised: ResNet-18is trained from scratch (not self-supervised pre-trained with SimCLR), while transferring CLIP

Revised: ViT-B/16 representations.

Revised: Table 10 shows ImageNet-1K Top-1 linear probe accuracy on ResNet-18representations.

Revised: As similar to ResNet-50, BeamCLIP learns better representations than SSL methodssuch as SimCLR [4], MoCo-v2 [6], BYOL [16], and SwAV [3].

Revised: More important, the BeamCLIPprovides better performance than OSS [8] that simultaneously learns and transfers representationsfrom ResNet-50.

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: Effects of random text prompts

Revised: In this section, we further analyze the effect of text prompts from the perspective of unsupervisedlearning.

Revised: Before that, we briefly review the proposed method.

Revised: In this paper, we propose the BeamCLIP, an unsupervised representation transfer method of a large pre-trained multimodal model such as

Revised: CLIP.

Revised: The BeamCLIP can transfer the visual representations of CLIP by using unlabeled images on adownstream dataset.

Revised: To achieve this, we propose cross-modal similarity matching (CSM).

Revised: In CSM,at first, given an unlabeled image, cross-modal similarity distribution is measured from multiple textprompt embeddings in the teacher’s embedding space.

Revised: Then, a student model is encouraged to mimicthe cross-modal similarity distribution of the teacher model by matching these similarity distributions.

Revised: To achieve effective transfer, we use anchor text embeddings by encoding text prompts.

Revised: For example,on CIFAR10, we use ten text prompts in the form of "a photo of {class name}" .

Revised: Note that thetext prompts are not paired with each image.

Revised: CIFAR10.

Revised: To further decouple the dependency to the class names of a dataset, we measure thetransfer learning accuracy by using random prompts.

Revised: Figure 8 shows the transfer learning accuracyon CIFAR10 with regard to the number of random text prompt of CLIP.

Revised: As shown in Figure 8 (a), the

Revised: BeamCLIP is effective, while using only subset text prompts.

Revised: Note that the BeamCLIP (CE+EntMin)is more effective than BeamCLIP (KL), as we use more text prompts.

Revised: As shown in Figure 9, the

Revised: BeamCLIP is still effective compared to CLIP-RN50, even though random prompts from ImageNetare used.

Revised: The values in Figure 8 are also presented in Table 11 and Table 13.

Revised: CIFAR100.

Revised: We perform the similar experiment on CIFAR100.

Revised: Table 9 shows unsupervised transferlearning accuracy of CIFAR100 [23] with regard to the number of random text prompts.

Revised: The valuesin Figure 9 are also presented in Table 13 and Table 14.

Revised: [Figure]

Revised: [Table]

Revised: CLIP [32] (zero-shot)

Revised: [Table]

Revised: BeamCLIP (CE+EntMin)

Revised: [Table]

Revised: BeamCLIP

Revised: [Table]

Revised: KL)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: shot)

Revised: [Table]

Revised: [32] (zero

Revised: [Table]

Revised: shot)

Revised: [Table]

Revised: BeamCLIP

Revised: [Table]

Revised: [Table]

Revised: BeamCLIP

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Method

Revised: [Table]

Revised: CLIP [32] (zero-shot)

Revised: [Table]

Revised: CLIP [32] (zero-shot)

Revised: [Table]

Revised: BeamCLIP (CE+EntMin)

Revised: [Table]

Revised: BeamCLIP (KL)

Revised: [Table]

Revised: [Table]

Revised: CLIP [32] (zero-shot)

Revised: [Table]

Revised: CLIP [32] (zero-shot)

Revised: [Table]

Revised: BeamCLIP (CE+EntMin)

Revised: [Table]

Revised: BeamCLIP (KL)

Revised: S

Revised: [Table]

Revised: C.4 Qualitative results

Revised: Figure 10 shows qualitative results via top-5 text-image retrieval.

Revised: A red rectangle denotes an incorrectresult.

Revised: CLIP-RN50 provides many incorrect results since it shows relatively low zero-shot accuracy.

Revised: In contrast, BeamCLIP-RN50 provides much improved results, since it is distilled from CLIPViT/16 which shows very high zero-shot accuracy.

Revised: More interestingly, BeamCLIP-RN50 providessurprisingly good text-image retrieval results, even though unseen text prompts such as "a photoof {pink rose}" or "a photo of {yellow rose}" are given.