CPL : Counterfactual Prompt Learning for Vision and Language Models (2022)

Abstract: Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts.Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel Counterfactual Prompt Learning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework.Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09% and 25.08% relative improvements across three few-shot scenarios on unseen test sets respectively.

## introduction pre-trained vision and language foundation models @xcite have shown encouraging results toward open-domain visual-concept matching. benefiting from prompt engineering @xcite , where free-form text prompts are designed for specific task goals, those foundation models can be easily transferred to a wide array of tasks under 1 our code is released at https://github.com/ eric-ai-lab/cpl . a: a large long train on a steel track b: a large long train on a steel track near a barn what if we add a barn to image a (or remove the barn from image b)? will the prompt be changed ? figure 1 : a conceptual overview of counterfactual prompt learning. cpl constructs counterfactuals by identifying non-spurious feature change that causally causes the prompt change. in this case, the "barn" feature is the essential cause between prompt a and b. prompt tuning @xcite , on the other hand, liberates us from manual prompt engineering and automates this process. prompt tuning methods @xcite @xcite are proposed to effectively transfer clip to image recognition tasks after tuning a learnable prompt with a few examples of the classes. however, those methods purely conduct empirical risk minimization (erm) and optimize for predictive accuracy, which often produces spurious, inefficient, or entangled representations @xcite . therefore, the generalization ability of existing prompt tuning methods for vision and language models is limited, and they often fail to transfer well to unseen classes or concepts. for example, the image classification performance of the sota method cocoop @xcite is similar or even degrades on unseen classes when compared with zero-shot clip. learning non-spurious representation for better generalization requires disentangling features that causally determine the prompts. one solution is counterfactual reasoning. counterfactual ("counter to the facts") is a concept that describes the human capacity to learn from limited prior experiences by imagining the outcome of an alternative action that could have been taken. so we can do counterfactual intervention by asking "what if ..." questions in prompt learning. for example, as shown in figure 1 , a change in the visual feature of the barn would cause the label to change (if we view the two prompts as two labels). therefore, we introduce a new causality-based approach, counterfactual prompt learning (cpl), for non-spurious and efficient prompt learning. first, we introduce a text-based negative sampling strategy to discover the most semantically-similar negative sample based on text similarity. then we generate a counterfactual example by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causally causes prompt change. finally, we adopt contrastive learning in the joint optimization framework (with counterfactual construction) to tune the learnable prompts using both factual and counterfactual examples. the causally fine-tuned prompts will eventually guide vision-and-language foundation models to distinguish images from unseen concepts, thereby improving the generalization ability of prompt learning. we extensively evaluate cpl using seven standard datasets for image classification, two for image-text-retrieval, and one for visual question answering (vqa). we show that cpl outperforms the baseline on all three tasks: on image classification, our method achieves 3.55% average relative improvement on unseen classes across the seven datasets in terms of accuracy; on image-text retrieval, our method improves the most (4.09% relative improvement in terms of recall@1) when using 0.5% of total training instances on mscoco @xcite and @xcite ; on vqa, we gain up to 25.08% relative improvement on the vqav2 @xcite ) dataset. our main contributions are summarized below: prompt tuning. many works focus on learning from discrete natural language prompts, e.g., auto-prompt @xcite elicits knowledge from language models with automatically generated discrete prompts. lately, many other works @xcite directly tune prompts in continuous vector forms. @xcite introduces q-learning to optimize the soft prompt. p-tuning v2 @xcite shows that continuous prompt tuning achieves the same performance as fine-tuning in various settings. prompt tuning also receives great interest in the computer vision domain. for example, coop proposes a continuous prompt optimization strategy to avoid prompt design. cocoop @xcite extends coop by further learning an instance-conditional network to generate an input-conditional token for each image. however, these methods trained with empirical risk minimization (erm) may learn to rely on correlations between class labels and spurious attributes by minimizing average training error @xcite . they usually learn spurious, inefficient, and entangled representation, lacking generalization ability to unseen scenarios. ## conclusion in this paper, we propose a counterfactual prompt learning (cpl) framework to avoid timeconsuming prompt engineering and learn more generalizable prompt representation for vision and language models. we conduct abundant experiments on seven widely used image classification datasets, two image-text retrieval datasets, and one visual question answering dataset. our proposed cpl method outperforms the previous prompt tuning baseline and the zero-shot clip across the three tasks. in the future, we plan to develop more sophisticated methods based on cpl and extend cpl to other vision and language tasks.