A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022)

Abstract: Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners. For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM).Furthermore, we analyze the effect of diverse prompts for few-shot tasks. Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2% point and achieves comparable results to a 246x larger model, PICa.In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at https://github.com/woojeongjin/FewVLM

## introduction fine-tuning large pre-trained language models (plms) have led to strong results in various domains including vision-language tasks @xcite @xcite . such large plms can learn a new task with a few examples or generalize to a new task without fine-tuning on any training examples, i.e., few-shot and zero-shot learnin our setup, we convert the tasks into generative tasks in which models need to generate target text given input text and an image. transformer encoder transformer decoder what position is this man playing? <@xmath0: [q] answer: <@xmath1: what position is this man playing? answer: <@xmath2. in such a few-shot setting, task-specific prompts or task descriptions are important and have shown effectiveness in few-shot nlp tasks @xcite @xcite . to extend the success to vl tasks, we aim to answer the following questions for prompt-based low-resource vl learning. q1) how does prompt design affect zero/few-shot learning on new tasks? q2) does prompt design still matter given larger training? q3) how do different pre-training objectives affect zero/few-shot learning? to answer these questions, we explore various prompt formats including hand-crafted and noisy prompts on zero/few-shot vl learning datasets. in addition, we study pre-training objectives on few-shot tasks inspired by @xcite : prefix language modeling (prefixlm) inspired by @xcite and masked language modeling (maskedlm). to this end, we investigate the model's performance on few-shot vl tasks including visual question answering @xcite @xcite , captioning @xcite (fig. 1 ), and mini-imagenet @xcite . in our empirical analysis, our fewvlm with prompt-based learning outperforms frozen @xcite which is 31× larger than fewvlm by 18.2% point on zero-shot vqav2 and achieves comparable results to a 246× larger model, pica @xcite . furthermore, we observe that (1) prompts significantly affect zero-shot performance but marginally affect fewshot performance on new tasks ( §6.2 and §6.3), (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data ( §6.5), and (3) maskedlm helps few-shot vqa tasks while prefixlm boosts captioning performance ( §6.6). ## related work vision-language few-shot learning. recently, several few-shot learners on vision-language tasks were proposed including gpt @xcite , frozen @xcite , pica @xcite , and simvlm @xcite . frozen @xcite is a large language model based on gpt-2 @xcite , and is transformed into a multimodal few-shot learner by extending the soft prompting to incorporate a set of images and text. their approach shows the fewshot capability on visual question answering and image classification tasks. similarly, pica @xcite uses @xcite to solve vqa tasks in a few-shot manner by providing a few in-context vqa examples. it converts images into textual descriptions so that gpt-3 can understand the images. simvlm @xcite is trained with prefix language modeling on weakly-supervised datasets. it demonstrates its effectiveness on a zero-shot captioning task. while these models achieve improvement on few-shot tasks, they are impractical to use in real-world applications due to their model sizes. language model prompting. providing prompts or task descriptions play an vital role in improving pre-trained language models in many tasks @xcite @xcite . among them, gpt models @xcite achieved great success in prompting we pretrain fewvlm with masked language modeling (maskedlm) and prefix language modeling (pre-fixlm). or task demonstrations in nlp tasks. in light of this direction, prompt-based approaches improve small pre-trained models in few-shot text classification tasks @xcite . clip @xcite also explores prompt templates for image classification which affect zero-shot performance. we follow these core ideas so we aim to improve zero-shot and few-shot performance using prompts in visionlanguage tasks. ## analysis setup in this work, we study the zero-shot and few-shot performance of vision-language models l. we introduce our analysis setup: problem formulation, analysis questions, downstream tasks and datasets, evaluation metrics, and baselines. ## method before diving into the analysis, we introduce our model, fewvlm, to do zero/few-shot learning on vl tasks and answer the analysis questions we raised. we introduce fewvlm architecture and pre-training objectives. ## low-resource adaptation in downstream tasks, we train our model with few-shot examples. fig. 2 shows an illustration of fewvlm in inference time. given a prompt template p, we first get input text and target text using the template x, @xmath3, label). then we train model parameters by minimizing the negative log-likelihood in eq. ( 1 ). in inference, we use the same prompt and the model generates the label text. here we obtain the final label by removing the target prompt template. ## results and discussion in this section, we first discuss our main results on zero-shot and few-shot tasks and then answer the questions we raised: does prompt design matter in zero/few-shot learning? ## conclusion in this work, we present fewvlm, a few-shot prompt-based learner on vision-language tasks. on diverse datasets, fewvlm outperforms baselines and shows comparable results to pica which is 246× larger than ours. we observe that prompts are vital in zero-shot and few-shot tasks and each pre-training objective helps different few-shot tasks. also, we find out that models with larger training data are not significantly affected by noisy prompts. future work includes exploring automatic prompt generation and diverse formats of few-shot tasks such as multiple-choice vqa. finding optimal prompts require exhaustive engineering to achieve the best performance and leads to impressive results. we leave the exploration of these directions to future investigations.