CLIP Models are Few-Shot Learners: Empirical Studies on VQA and Visual Entailment (2022)

Abstract: CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP’s zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few-shot results on the visual question answering and visual entailment tasks without introducing any additional pre-training procedure.

## introduction vision-language understanding (vlu) tasks, such as visual question answering @xcite and visual entailment @xcite , test a system's ability to comprehensively understand the semantics of both visual world and natural language. to capture the alignment between vision and language, various efforts have been made to build the vision-language pre-trained models @xcite @xcite @xcite . despite their superior performances, these methods have extensively utilized human-annotated training data that are expensive or require expert knowledge, such as object detection datasets @xcite and aligned image-text pairs @xcite . collecting such datasets requires heavy work on data gathering and figure 1 : examples of the two vision-language understanding tasks. for vqa, language prompts are used. for visual entailment, caption and hypothesis, i.e., texttext pairs, are used in training, while image and hypothesis, i.e., image-text pairs, are used at inference. human annotation, and thus their scales are only in the realm of tens of millions, which are much smaller than the internet text corpora for nlp pretraining @xcite . recently, clip @xcite has been proposed to learn visual concepts with natural language supervision, where its 400 million imagetext pairs are crawled from the internet. clip consists of a visual encoder and a text encoder, and it learns visual representations by aligning images and texts through contrastive loss. in this way, clip achieves strong zero-shot performances on vision benchmarks such as imagenet. besides, @xcite prove that clip could be leveraged as a strong visual encoder to benefit downstream vision-language tasks. however, there are two major differences between clip and previous visual encoders: 1) it is trained on much larger yet noisy web data, and 2) it has a shallow interaction between vision and language. the first feature promises the generalization ability of clip, and the second one equips alignment ability across modalities. could the strong zero-shot ability of clip be transferred to vision-language understanding tasks? to answer the above question, in this work, we empirically study how to transfer clip's zero-shot ability into vlu tasks and further turn clip into a few-shot learner. we carried out experiments on two vlu tasks: 1) visual question answering, where the model needs to give an answer according to the details of an image and a natural sentence question, and 2) visual entailment, where the model needs to determine the entailment relation between an image and a natural sentence. figure 1 demonstrates the basic forms of the two studied tasks. for the zero-shot visual question answering task, the key to a successful zero-shot capability transfer is to mitigate the gap between the pre-training task of clip and the task form of question answering. inspired by the recent advancements of few-shot learning in nlp @xcite , we address this issue by introducing a two-step prompt generation strategy, including automatic conversions from question to statement to get masked templates, and a span-infilling with generative pre-trained t5 model @xcite to get candidate answers. we explore a zero-shot cross-modality (language and vision) transfer capability through the visual entailment task. specifically, we replace the image with its captions during training and only update a small classification layer. then at inference, as usual, we still use image-text pairs for testing. this allows us to investigate how well the language and vision representations are aligned in clip models. we further leverage few-shot learning to improve clip's visual question answering performance based on the zero-shot transferring methods. we find that optimizing only bias and normalization (binor) parameters would make better use of limited examples and yield better results than the latest few-shot model frozen @xcite . experiments confirm that clip models can be good vision-language few-shot learners. our contributions are summarized as follows: ## preliminaries t(text) • v(image), which is used as an alignment score between the input image and text. it is pretrained to distinguish aligned image-text pairs from randomly combined ones by a contrastive loss. instead of training on vision benchmarks, clip leverages abundant language supervisions from 400 million web-crawled image-text pairs and can conduct a variety of image classification tasks without specific optimizing. however, directly applying clip as a vision-language understanding model is still difficult @xcite . ## zero-shot cross-modality transfer recent pre-trained multilingual language models @xcite @xcite have been shown to be successful in transferring representations across different languages. for example, they can be only fine-tuned on a source language and evaluated on various target languages without specific training, yet still achieving good performance. on the other hand, the clip models achieve strong zero-shot performances on both image-to-text and text-to-image retrieval tasks @xcite only through a dot product between vision and language representations, which gives us an intuition that the two modalities are well aligned in the clip models. is there a cross-modality capability between language and vision in the clip models, just like the multilingual ones across languages? to answer the above question, we utilize the visual entailment task ( § 2.2) to explore the zero-shot cross-modality performance. figure 4 briefs the key idea. specifically, we train an mlp classifier over the fused representations of premise and hy-pothesis, and the fusion function is: in this section, we aim to investigate whether the clip models could benefit from few-shot learning, where we work on the visual question answering task to study it. ## experiments 6.1 experimental settings datasets. for visual question answering and visual entailment, we carry out experiments on the vqav2 @xcite and the snli-ve @xcite datasets, respectively. we report the statistics of the two datasets in appendix a. for the vqa task's evaluation, we follow the frozen model @xcite to calculate the vqa scores on the vqav2 validation set. for visual entailment, we calculate the accuracy on both validation and test sets through the sklearn toolkit. ## related work vision-language few-shot learning. leveraging aligned caption data, vision-language models pre-trained by an image-text discriminative loss have recently enabled strong zero-shot generalization on image classification and cross-modality retrieval tasks @xcite . different from the discriminative manner, tsimpoukelli et al. ( 2021 ) prompt a large frozen language model with vision prefix in a generative way, which is the first vision-language few-shot model. language model prompting. this work is also inspired by the line of research in language model prompting @xcite . initialized by the gpt series @xcite @xcite , prompting has become a popular manner to mining knowledge from pre-trained language models @xcite in a zero-shot or few-shot way @xcite @xcite . besides mining knowledge from the language model, pet work @xcite presents a semi-supervised prompting method for improving few-shot language understanding performance. ## conclusions in this work, we empirically studied how to transfer clip models into vision-language understanding tasks. we first explored the clip models' zero-shot vqa capability by leveraging language prompts and further proposed a parameter-efficient finetuning method to boost the few-shot performance. we also demonstrate a zero-shot cross-modality transfer capability of clip models on the visual entailment task. experiments and analyses on vqav2 and snli-ve confirm that the clip models can be good vl few-shot learners.