Paper: Rethinking Semantic Few-Shot Image Classification (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this work is limited as the off-the-shelf text encoder has been proposed previously using a similar contrastive loss for few-shot learning. The idea has overlapping to the idea proposed in VS-alignment (Afham et al., 2022) with a marginal extension in the meta-learning technique with MAML. Some discussion (head-to-head) on the proposed method and Afham et al. would be beneficial for the readers to spot the difference and novelty of the work.

Author response:
**A1.** Thank you for your valuable feedback. While both VS-alignment and our method utilize contrastive learning to align visual and textual features, they differ in several key aspects.

Firstly, VS-alignment directly aligns the visual and textual features in a manner similar to CLIP. However, for few-shot tasks, **direct alignment can result in very poor performance**, as illustrated in Figure 1 of our paper. This is because the meta-test task and meta-training task may have a large distribution gap in few-shot learning, indicating that alignment may be no longer satisfied for the meta-test task. In our work, **we address this limitation by introducing a task-specific image-text metric module**, which can adaptively align the visual and textual features. As a result, the experimental results in Table 1 of our paper demonstrate **substantial performance improvement**, rising from 65.89% to 74.48% over VS-alignment.

Secondly, our contribution extends beyond the design of a metric module for aligning text embeddings and image embeddings. More importantly, **we introduce a novel meta-learning framework for learning the parameters of our method**. In particular, we carefully design the metric module as a task-specific component, and make other modules such as the feature extractor updating in the outer loop. 

Thirdly,  **our work demonstrates the feasibility of aligning fixed text embeddings extracted from pre-trained language models with visual features**. As shown in Table 1 of our paper, it is noteworthy that our FITE-GloVe and FITE-RoBERTa exhibit a remarkable ability to achieve performances closely aligned with that of FITE-CLIP.

These distinctions emphasize the uniqueness and advancements introduced in our approach compared to the VS-alignment method, providing a clearer understanding of the novelty in our work.

Reviewer comment (Novelty):
The citation to Meta learning paradigm in Page 3 is not precise. Vinyals et al., 2016 do not discuss about meta-learning but the work is more related to learn in a few data regime. MAML paper would be a more relevant citation in this part.

Author response:
**A2.** Thank you for your feedback. The term "meta-learning paradigm" refers to a training strategy that updates the model based on a series of meta-tasks, encompassing various approaches, including but not limited to "model-agnostic" meta-learning methods such as MAML (Finn et al., 2017). Notably, MatchingNet (Vinyals et al., 2016) is recognized as one of the pioneering efforts that introduced the meta-learning paradigm into few-shot learning.

However, we acknowledge your suggestion and have incorporated the MAML paper in the citation to the Meta-learning paradigm on Page 3. We believe this adjustment enhances the precision of our references and clarifies the connection between our work and relevant meta-learning methodologies.

Reviewer comment (Theory):
The manuscript is not well written. Equation (3?) in Page 6 is not precisely correct as the gradient descent should be performed w.r.t. I, T, and M. Please check the expression after $\nabla$. Also, the equation in Page 6 has no number. Please fix this in the revised version.

Author response:
**A3.** Thank you for your feedback. We would like to clarify that only the metric module (M) is designed to be updated in the inner loop, while the image and text encoders (I and T) remain unchanged in the inner loop. Therefore, the gradient descent should not be performed with respect to I and T. As such, the equation in Page 6 is accurate as stated.

Regarding the equation numbering, we follow Occam's Rule (http://www.ams.org/notices/199508/hwang.pdf): Number only those equations that are referred back to. We appreciate your understanding and will ensure that the necessary corrections are made in the revised version.

Reviewer comment (Experiments):
Regarding this sentence in Page 2: “Secondly, in contrast to vision-language pre-trained models where both visual and textual encoders are learnable to align embeddings, we utilize frozen public textual encoders. This leads to totally different structures of textual embedding spaces and thus makes the alignment between visual and textual features difficult,” one concern is that why we cannot use the image encoder from a pretrained image encoder (e.g., CLIP image encoder)? Are there any settings that do not allow this condition? If this was feasible, would this work still consider MAML with several gradient steps to obtain optimal performance?

Author response:
**A4.** Thank you for your question. We would like to clarify that our work follows the standard semantic few-shot classification setting as established in [r1][r2][r3], wherein only pre-trained text encoders are available, and pre-trained image encoders (e.g., CLIP image encoder) are not accessible. Notably, our method can achieve state-of-the-art (SOTA) performance in this standard setting.

If a pre-trained image encoder is allowed to access, the performance will surpass that of the aforementioned setting, making it unfair for direct comparison with previous works. However, if such access were feasible, employing MAML would be still considered to achieve better performance. This is because pre-trained CLIP focuses on aligning image and text on the training dataset, in contrast, our few-shot learning method aims at quickly re-aligning image and text on unseen datasets with unseen categories, which may have totally different distributions from the training dataset. MAML is specifically designed to address such scenarios, making it a valuable consideration for improving performance in this context.

[r1] AM3: Adaptive cross-modal few-shot learning.

[r2] LPE: Semantic guided latent parts embedding for few-shot learning.

[r3] SP: Semantic prompt for few-shot image recognition.

Reviewer comment (Novelty):
This work must consider one important work [1] in this direction. This work is the pioneer in using the off-the-shelf pretrained language model for few-shot learning but there is no discussion and citation to this work.

Author response:
**A5.** Thank you for reminding us of this related paper. While both Frozen [1] and our method use the frozen pre-trained language model and train the visual encoder from scratch for the few-shot learning task, there are several key differences between them.

Firstly, Frozen encodes images into the word embedding space of a large pre-trained language model and lets the language model generate captions for images, **which solves discriminative problems in a generative manner**. In contrast, **our method introduces the concept of alignment, which is more suitable for classification tasks**, leading to superior performance in few-shot classification tasks.

Secondly, Frozen focuses on training a large multimodal model that can be generalized to various multimodal tasks in both zero-shot and few-shot settings. However, **it does not thoroughly investigate about leveraging support examples to enable the model to quickly adapt to unseen datasets**. In contrast, **our method aims at quickly re-aligning image and text on unseen datasets with unseen categories**. Specifically, we introduce a task-specific metric module and design a meta-learning framework, achieving strong generalization performance on meta-test datasets. 

The specific performance comparisons related to the above two points are discussed in **A6**. We have cited this paper and added the discussion about this work in the revised paper.

Reviewer comment (Experiments):
The experiments are quite limited as most of the comparisons only involve image data without text information. It would be better to provide some other datasets e.g., COCO. Some other experiments as in [1] might be considered to show the efficacy of the proposed method and comparison with [1] on different modes (e.g., frozen and not frozen).

Author response:
**A6.** Thanks for your suggestion.

Firstly, we have also conducted experiments on several other datasets with semantic category names in the meta-dataset [r4] (including COCO), and the results are presented in the table below.
|  | Textures | VGG Flower | Quick Draw | Fungi | MSCOCO |
| --- | --- | --- | --- | --- | --- |
| ProtoNet | 66.56 | 85.27 | 48.96 | 39.71 | 41.00 |
| RelationNet | 52.97 | 68.76 | 43.30 | 30.55 | 29.15 |
| fo-Proto-MAML | 66.49 | 87.15 | 51.52 | 39.66 | 43.74 |
| **FITE-CLIP** | **71.06±0.79** | **88.73±0.64** | **68.89±0.79** | **60.27±0.86** | **60.57±0.87** |

Although with a weaker backbone (ResNet-12) than that used in meta-dataset (ResNet-18), our method outperforms all the baselines reported in meta-dataset. Besides, **the performance on the meta-dataset also proves the strong cross-domain ability of our method** as the model is only trained on the meta-training dataset of miniImageNet. We have added this experiment to the revised version of our paper.

Secondly, our method exhibits stronger performance in few-shot classification scenarios. On the one hand, although Frozen uses a stronger backbone (e.g., NF-ResNet-50) and trains it on a large-scale multimodal dataset (e.g., Conceptual Captions), its performance is not significantly above chance (20.2% for 1-shot and 21.3% for 5-shot) on miniImagenet in the 5-way setting. In contrast, our FITE-CLIP achieves accuracies of 74.48% and 84.45% on miniImagenet in 5-way 1-shot and 5-way 5-shot settings respectively. On the other hand, as the number of examples increases, the performance improvement of our method (an increase of 9.97% from 1-shot to 5-shot) is more significant than Frozen (an increase of 1.1% from 1-shot to 5-shot). From the performance comparison, it can be observed that our method is more suitable for few-shot classification scenarios.

[r4] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Reviewer comment (Novelty):
Overclaim the technical novelty. The framework of learning image and text alignment with a bilinear function has been extensively studied in zero-shot learning. Some zero-shot learning methods (e.g., Xu et al.) even show that the framework generalizes well to few-shot learning setting. Although this paper adopts a different few-shot learning setting with episodic evaluation protocol, in principle, the core method is rather similar. In my view, this work seems to be a trivial combination of MAML and previous zero-shot learning methods, which can not be counted as a significant technical contribution. This paper seems to ignore this point and fails to discuss how it improves the zero-shot learning methods. [A] Xu et al., Attribute Prototype Network for Any-Shot Learning. IJCV 2022.

Author response:
Thanks for your comments. 

First of all, we would like to clarify that the metric module of our method is designed as a general learnable function for measuring the alignment between image and text, which is **not limited to the bilinear function**. Our method is suitable for other function forms beyond the bilinear function.

Second, zero-shot learning methods such as CLIP align image and text on the training dataset, in contrast, our few-shot learning method aims at **quickly re-aligning image and text on few-shot datasets which may have totally different distributions from the meta-training dataset**. Therefore, **few-shot learning methods have distinct goals from zero-shot learning methods**, and the performance of zero-shot learning methods may not translate well to few-shot learning scenarios, as they involve different core ideas.

Thirdly, even when considering the combination of MAML and zero-shot learning methods, there exist various possible solutions, and our approach is not a trivial combination. Among these possible solutions, **we carefully design the metric module as a task-specific module and make other modules such as the feature extractor updated in the outer loop**. Intuitively, feature extraction should be general for different downstream tasks. However, each concrete downstream task may have different focuses on feature channels. Therefore, the image-text alignment conducted by our metric module should be task-specific and updated in the inner loop. 

We have cited this paper and added the discussion about zero-shot learning methods in the revised paper.

Reviewer comment (Evaluation):
Lack of insights. Although the proposed method achieves SOTA on some dataset, it mainly relies on the CLIP text embeddings, which is somewhat expected because it is known that CLIP embeddings can achieve impressive image classification results. The results using word embeddings and language embeddings are actually worse than some baseline methods. There are not much insights except the comparison between three different text embeddings.

Author response:
Firstly, it is expected that CLIP text embeddings achieve better performance than word embedding and language embeddings, as CLIP text embeddings are trained with contrastive learning on large-scale image-text pairs data. However, we would like to clarify that **our work provides an insight that textual embeddings trained without multimodal data (e.g., GloVe, RoBERTa) can also be aligned with visual features**. In particular, how to align the text embeddings extracted from the language model pre-trained on text corpora and image embeddings is a challenging problem. As shown in Figure 1 of our paper, direct alignment results in bad performance. To address the problem, we introduce a metric module and design a meta-learning framework. By doing so, our FITE-GloVe and FITE-RoBERTa can achieve performance close to FITE-CLIP.

Secondly, while CLIP embeddings can indeed achieve impressive image classification results, we would like to highlight that **our FITE-CLIP achieves SOTA performance on various datasets compared to other baselines that also adopt pre-trained CLIP embeddings** (e.g., SP-CLIP). This underscores the effectiveness of our approach where the metric module and MAML framework are introduced.

Thirdly, it's important to note that **semantic few-shot learning SOTA methods typically focus on feature fusion between visual and textual embeddings** (e.g., SP-CLIP). In contrast, **our method introduces the concept of alignment**, leading to superior performance. This suggests that alignment offers a more promising approach compared to feature fusion in semantic few-shot learning.

In conclusion, our work not only leverages the expected advantages of CLIP embeddings but also offers novel insights into the alignment between visual and text embeddings in the few-shot learning problem.

Reviewer comment (Reproducibility):
In Sec. 4.1, it says "Besides, we make the prompt templates learnable to avoid time-consuming prompt engineering following." But I did not find any technical detail about this part.

Author response:
Sorry for the confusion. Specifically, the word embedding of each token in the prompt template is replaced by a learnable vector of the same dimension. These learnable vectors are initialized with original word embeddings and updated in the outer loop together with other parameters. We have added this detail to the revised version of our paper.

Reviewer comment (Experiments):
It is usual to study the dependence on the visual backbone. Only Resnets-12 is used in the paper but other backbones such as visual transformers backbone or recent foundation models could be included.

Author response:
**Q1.** First, it is usual to study the dependence on the visual backbone. Only Resnets-12 is used in the paper but other backbones such as visual transformers backbone or recent foundation models could be included.

**A1.** Thanks for your suggestion. We only use ResNet-12 in our paper as it is the standard backbone used in the few-shot classification setting. For example, many highly-cited few-shot papers also only use ResNet-12 in their experiments, such as 
- AM3: Adaptive Cross-Modal Few-shot Learning
- TRAML: Boosting Few-Shot Learning With Adaptive Margin Loss 
- RENet: Relational Embedding for Few-Shot Classification 
- DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover’s Distance and Structured Classifiers

Moreover, in response to your suggestion, we also conduct experiments with ViT-Small and Swin-Tiny, and the results are presented in the table below.
| Backbone | miniImageNet |  | tieredImageNet |  |
| --- | --- | --- | --- | --- |
|  | 1 shot | 5 shot | 1 shot | 5 shot |
| ResNet-12 | 74.48±0.55 | 84.45±0.40 | 79.20±0.43 |  88.85±0.13 |
| ViT-Small | 74.82±0.76 | 86.19±0.54 | 81.38±0.88 | 89.53±0.61 |
| Swin-Tiny | **76.49±0.75** | **87.33±0.54** | **81.98±0.80** | **90.48±0.54** |

From the table, we observe that **the replacement of the backbone from ResNet-12 with more powerful ViT-Small and Swin-Tiny yields significant performance improvements**. The results also suggest that **our approach can adapt well to different types of backbones without the need for adjustments**. We have incorporated this ablation study into the revised version of our paper.

Reviewer comment (Experiments):
The influence of the prompting strategy could be experimented with in more detail. The appendix provides an analysis of the learnable prompt template but it could have been interesting to correlate this prompting strategy to a more formalized definition of the type of semantic information that should be carried.

Author response:
**Q2.** The influence of the prompting strategy could also be experimented with in more detail. The appendix provides an analysis of the learnable prompt template but it could have been interesting to correlate this prompting strategy to a more formalized definition of the type of semantic information that should be carried. 

**A2.** Sorry for the confusion. In our paper, "semantic information" and "textual embedding" are used interchangeably without differentiation, following the terminology used in previous semantic few-shot learning works (e.g. SP [r1]). In particular, we design the prompting strategy and use a learnable prompt template, obtaining textual embeddings that are better for alignment in specific tasks.

[r1] Semantic prompt for few-shot image recognition

Reviewer comment (Reproducibility):
Some technical details are missing. In particular, it is well known that contrastive learning is highly dependent on the negative sampling strategy but also on the size of the batch. This information is missing in the paper.

Author response:
**Q3.** Some technical details are missing. In particular, it is well known that contrastive learning is highly dependent on the negative sampling strategy but also on the size of the batch. This information is missing in the paper.

**A3.** Thank you for your feedback. We would like to clarify that **some concepts in contrastive learning will have slight variations when applied to the few-shot setting**. Specifically, the few-shot learning problem is formulated as N-way K-shot classification (see the problem definition section). Taking 5-way 1-shot as an example, a fixed number of support samples (5) are loaded in each episode. The contrastive loss should be calculated among all these samples rather than selecting some hard negative samples. Consequently, the negative sample size is fixed and equal to 5-1=4. Therefore, due to the few-shot learning setting, we can not change the negative sampling strategy or the negative sample size.

Reviewer comment (Experiments):
Some benchmarks have been provided in the Few Shot community to better take into account the semantics. See for instance [meta-dataset](https://github.com/google-research/meta-dataset) or the work presented [here](https://arxiv.org/abs/2205.05155). What is the behavior of the approach on these benchmarks?

Author response:
**Q4.** Some benchmarks have been provided in the Few Shot community to better take into account the semantics. See for instance meta-dataset or the work presented here. What is the behavior of the approach on these benchmarks?

**A4.** We choose widely used few-shot datasets (e.g., miniImageNet, tieredImageNet, and CUB) as benchmarks following the baselines in our paper. Following your suggestion, we have also conducted experiments on several other datasets with semantic category names in the meta-dataset. The results are presented in the table below.
|  | Textures | VGG Flower | Quick Draw | Fungi | MSCOCO |
| --- | --- | --- | --- | --- | --- |
| ProtoNet | 66.56 | 85.27 | 48.96 | 39.71 | 41.00 |
| RelationNet | 52.97 | 68.76 | 43.30 | 30.55 | 29.15 |
| fo-Proto-MAML | 66.49 | 87.15 | 51.52 | 39.66 | 43.74 |
| **FITE-CLIP** | **71.06±0.79** | **88.73±0.64** | **68.89±0.79** | **60.27±0.86** | **60.57±0.87** |

Although with a weaker backbone (ResNet-12) than that used in meta-dataset (ResNet-18), our method outperforms all the baselines reported in meta-dataset. Besides, **the performance on the meta-dataset also proves the strong cross-domain ability of our method** as the model is only trained on the meta-training dataset of miniImageNet. We have added this experiment to the revised version of our paper.

Reviewer comment (Novelty):
Contrastive learning has been studied in the context of multimodal data. The positioning to these works could be interesting. See for instance [this paper](https://openaccess.thecvf.com/content/CVPR2021/papers/Yuan_Multimodal_Contrastive_Training_for_Visual_Representation_Learning_CVPR_2021_paper.pdf).

Author response:
**Q5.** Contrastive learning has been studied in the context of multimodal data. The positioning to these works could be interesting. See for instance this paper.

**A5.** Different from previous studies (e.g., [r2]) about contrastive learning on multimodal data, our paper focuses on applying multimodal contrastive learning to few-shot scenarios. Previous studies (e.g., [r2]) aim at training the model on large-scale multimodal datasets (e.g., COCO and Stock) to obtain high-quality multimodal features for downstream tasks. In contrast, **only small-scale few-shot datasets and fixed pre-trained language models are available in the semantic few-shot setting, which makes it more difficult to align visual and textual features, especially with language models pre-trained without multimodal data**. To address the problem, we introduce a metric module and design a meta-learning framework, achieving alignment quickly on meta-test datasets with three types of language models. We have added the discussion about this series of works in the revised version of our paper.

[r2] Multimodal Contrastive Training for Visual Representation Learning

Reviewer comment (Novelty):
Another concern is about the novelty of the proposed approach compared to (Chen et al, 2023). Compared to this work, the authors propose to add the metric module but this latter is also not new. I would appreciate it if the authors argued more on the novelty of this part, maybe in relation to the few shot settings and the way to tackle support and query sets.

Author response:
**Q6.** Another concern is about the novelty of the proposed approach compared to (Chen et al, 2023). Compared to this work, the authors propose to add the metric module but this latter is also not new. I would appreciate it if the authors argued more on the novelty of this part, maybe in relation to the few shot settings and the way to tackle support and query sets.

**A6.** Thank you for your valuable feedback. While both SP (Chen et al, 2023) and our method utilize semantic information to improve few-shot classification, they differ in several key aspects.

Firstly, we would like to clarify that **SP and our method perform semantic few-shot classification from two different lines of thought**. Semantic few-shot learning SOTA methods (including SP) typically focus on **feature fusion** between visual and textual embeddings. In contrast, our method introduces the concept of **alignment**, leading to superior performance. This suggests that alignment offers a more promising approach compared to feature fusion in semantic few-shot learning.

Secondly, our contribution is not only the design of a metric module to align the text embeddings and image embeddings, but more importantly, **the design of a meta-learning framework for learning the parameters of the metric module in the few-shot scenario**. Our task-specific metric module leverages the support set, enhancing the alignment of text embeddings and image embeddings on the query set and leading to better performance on meta-test tasks.

Reviewer comment (Experiments):
Recent works have shown the prevalence of transductive few-shot learning in image classification with this transductive setting that outperforms inductive approaches. How is the proposed approach compared to transductive sota approaches? How adapting the proposed approach to this scheme?

Author response:
**Q7.** Recent works have shown the prevalence of transductive few-shot learning in image classification with this transductive setting that outperforms inductive approaches. How is the proposed approach compared to transductive sota approaches? How adapting the proposed approach to this scheme?

**A7.** Thanks for your question. We would like to clarify that **inductive few-shot learning (FSL) and transductive FSL are two different branches of few-shot learning**. The former predicts query samples one by one while the latter predicts query samples as a whole. The transductive setting leverages the statistics of the unlabeled query set and thus generally outperforms inductive inference. Consequently, inductive methods and transductive methods are typically compared independently in the literature. Therefore, it is unfair to compare our approach for inductive FSL with transductive FSL methods. As the methods of these two branches employ quite different techniques, inductive approaches cannot be directly adapted to the transductive FSL scheme. However, we observe that MAML can be adapted to the transductive FSL scheme [r3]. Thus, the extension of our work to transductive FSL is considered a potential direction for future work, and we acknowledge this as an avenue for further exploration.

[r3] Leveraging the Feature Distribution in Transfer-based Few-Shot Learning

Reviewer comment (Theory):
Is it possible to better formalize the notion of semantic information? What kind of information should be added?

Author response:
**Q8.** It is possible to better formalize the notion of semantic information? What kind of information should be added?

**A8.** As this question is similar to **Q2**, please see **A2** for the explanation of the semantic information.

Reviewer comment (Experiments):
There is a lack of comparison with other state-of-the-art approaches, such as TRIDENT [1], BAVARDAGE [2], PEMnE-BMS [3], etc, though these methods may not rely on pretrained vision-language models. Besides, the authors miss some references about referring to category names for few-shot learning [4].

Author response:
**A1.** Thanks for your feedback. 

Firstly, we would like to clarify that **inductive few-shot learning (FSL) and transductive FSL are two different branches of few-shot learning**. The former predicts query samples one by one while the latter predicts query samples as a whole. The transductive setting leverages the statistics of the unlabeled query set and thus generally outperforms inductive inference. Consequently, inductive methods and transductive methods are typically compared independently in the literature. Therefore, **it is unfair to compare our approach for inductive FSL with transductive FSL methods** such as [1][2][3]. We have added these transductive methods to the related work section.

Secondly, both [4] and our paper utilize semantic information extracted by pre-trained text encoder to improve the performance of few-shot classification. The differences and advantages of our approach will be detailedly discussed in A3 below.

Thank you for reminding us of these related papers. We have cited and discussed them in the revised paper.

Reviewer comment (Experiments):
In the bi-level optimization, the metric module is updated in the inner loop and then all parameters are updated later in the outer loop. Could the authors provide any reasons or motivation for this training strategy?

Author response:
**A2.** Thanks for your question. We design this training strategy to make the metric module task-specific. **Intuitively, feature extraction is general for downstream classification tasks. However, image-text alignment, which is conducted by the metric module, is task-specific as each concrete classification task focuses on different parts of features.** Therefore, we make the metric module task-specific by updating it in the inner loop, and updating other modules, such as the feature extractor, in the outer loop. We have added the above discussion to the revised paper.

Reviewer comment (Experiments):
This paper aims to minimize the gap between the visual and textual modalities by introducing the metric module. It adds degrees of freedom so that there are some learnable parameters for bridging the gap. However, vision-language models are capable of zero-shot inference, therefore the way initializing the "bridge" between two modalities is crucial to the performance. Have the authors considered leveraging category name embedding for initialization [4]? By doing so, two modalities will be aligned automatically.

Author response:
**A3.** Thanks for your suggestion.

Firstly, we would like to clarify that **the initialization of our method is equivalent to using the category name embedding for initialization**. Specifically, in our implementation, the metric module is represented as a bilinear function $M(z, t) = z^TMt$, where $z$ is the visual feature, $t$ is the textual feature, and $M$ is a square matrix with learnable parameters for bridging the gap. We set $M$ to an identity matrix at the initial state in our implementation. In this scenario, it is equivalent to leveraging the category name embedding for initialization, as $M\cdot t=I\cdot t=t$ and $M(z, t) = z^Tt$. We have added this implementation detail in the revised paper.

Secondly, we would like to highlight that direct alignment leads to poor performance, as described in Figure 1 of our paper. Therefore, we design the bi-level optimization to update the metric module $M$ and make it task-specific.

Thirdly, [4] chooses the pre-trained textual encoder of CoCa, which is a strong vision language model as the language model. However, for language models pre-trained without image-text pairs, the alignment between image and text is more difficult. Our experiments with GloVe and RoBERTa show that our method is also effective with this kind of language model.