I nt C o O p: Interpretability-Aware Vision-Language Prompt Tuning (2024)

Abstract: Image-text contrastive models such as CLIP learn transferable and robust representations for zero-shot transfer to a variety of downstream tasks. However, to obtain strong downstream performances, prompts need to be carefully curated, which can be a tedious engineering task. To address the issue of manual prompt engineering, prompt-tuning is used where a set of contextual vectors are learned by leveraging information from the training data. Despite their effectiveness, existing prompt-tuning frameworks often lack interpretability, thus limiting their ability to understand the compositional nature of images. In this work, we first identify that incorporating compositional attributes (e.g., a “green” tree frog) in the design of manual prompts can significantly enhance image-text alignment scores. Building upon this observation, we propose a novel and interpretable prompt-tuning method named IntCoOp, which learns to jointly align attribute-level inductive biases and class embeddings during prompt-tuning. To assess the effectiveness of our approach, we evaluate IntCoOp across two representative tasks in a few-shot learning setup: generalization to novel classes, and unseen domain shifts. Through extensive experiments across 10 downstream datasets on CLIP, we find that introducing attribute-level inductive biases leads to superior performance against state-of-art prompt tuning frameworks. Notably, in a 16-shot setup, IntCoOp improves CoOp by 7.35% in average performance across 10 diverse datasets.

## introduction recently, significant advancements have been achieved in the field of vision-language models, with notable examples like clip @xcite , flamingo @xcite , align @xcite , and coca @xcite . these models have excelled in acquiring transferable and robust image representations, a feat accomplished through a combination of two fundamental components: (i) large-scale paired image-text datasets ranging from 400m to 2b image-text pairs; (ii) a contrastive objective that aligns the image and text embeddings into a common subspace. leveraging these ingredients, vision-language models have obtained strong performances in zero-shot classification, image-text retrieval, and robustness to distribution shifts. for all these tasks, contrastive models such as clip enable zero-shot inference: given an image i and a set of text prompts {t i } n @xmath0, the most relevant text-prompt @xmath1. adapting image-text contrastive models for downstream tasks is a complex undertaking. achieving optimal performance with image-text contrastive models necessitates the manual creation of domain-specific prompts, a process that demands extensive domain knowledge and is exceptionally challenging and time-consuming. even with considerable prompt engineering, there is limited assurance that the designed prompt is truly optimal. to address this issue, recent research @xcite @xcite has turned to prompt-tuning techniques, borrowing concepts from the field of nlp and applying them to vision-language models like clip to achieve good image recognition performance on downstream tasks. however these frameworks often lack interpretability and as a result the model struggles to understand the composition of the images. in this study, we address this challenge by learning a method to extract and embed attribute-level information into the prompt-tuning framework during training. we define an attribute as an interpretable concept that is relevant to the image and encapsulates its semantic essence. although manwe present the overarching architecture for generating attribute labels a for a given training image using blip-2 vqa model. based on these findings, we present an interpretable prompt-tuning approach known as intcoop, which incorporates attribute information into the prompt-tuning procedure thereby generating more interpretable prompts. while one might initially consider leveraging off-the-shelf image captioning models to generate attribute labels, this approach becomes infeasible during inference when class labels are unavailable. consequently, generating attribute descriptions for images emerges as a non-trivial task. to mitigate this challenge, we train a compact hypernetwork responsible for predicting embeddings corresponding to attribute descriptors. we test our prompt-tuning method intcoop on a range of diverse downstream datasets to test for generalization to novel classes, and robustness to distribution shifts. in section 5, we show that our method intcoop has improved robustness to distribution shifts, domain generalization, and fewshot learning. notably, in domain generalization setup, intcoop outperforms plot @xcite by 19.32% in average performance across 4 diverse domains. in summary, our research provides compelling empirical support for the substan-tial advantages of integrating attribute-level inductive biases into the prompt-tuning process. overall, our paper makes the following key contributions: to mitigate this issue, recent research has turned to prompt-tuning techniques to automatically learn domain specific prompts. prompt tuning. given a set of text-instructions and an image, existing vision-language models make their decisions by selecting the textinstruction which has the maximum similarity between the image and text-embeddings. recent advances in this field, such as methods like coop @xcite , cocoop @xcite , proda @xcite , vpt @xcite , maple @xcite , kg-coop @xcite , prograd @xcite , lasp @xcite , rpo @xcite , dapt @xcite , plot @xcite , and lfa @xcite have shifted from manually designed prompts to automatically learning prompts through fine-tuning learnable vectors with image-text pairs from the target domain. coop fine-tunes clip to optimize a set of learnable tokens in the input layer of the text-encoder. cocoop enhances coop by incorporating conditional image information in the promptlearning process. vpt learns tokens in each layer of a given encoder through a fine-tuning objective. kgcoop introduces a regularizer to constrain the prompt tuning process such that the representations of the learned prompts do not deviate significantly from the manually crafted prompts. pro-grad leverages the prompt gradients to fine-tune the learnable tokens such that the prior knowledge in the vision-language model is preserved. plot applies optimal transport to match the vision and text modalities for generating the discriminative and visual-aligned local textual prompt tokens. refer @xcite for a comprehensive survey on prompt-tuning frameworks. overall, none of the existing works aim to understand if augmenting certain inductive biases in the prompt-tuning process is beneficial. our work intcoop specifically addresses this issue and shows that incorporating compositional attributes in the prompt-tuning pro-cess can indeed be beneficial for downstream tasks. ## preliminaries contrastive language-image pre-training (clip) @xcite is a visionlanguage model trained on a large dataset of 400 million image-text caption pairs using a contrastive loss. clip primarily consists of two major components: to obtain the final image representation, the class token x l of the last transformer layer (v l ) is projected to a common image-text latent embedding space via a linear projection layer. for zero-shot inference on a downstream dataset consisting of c classes with class names {[cls] c } c @xmath2, clip uses hand-crafted prompts to generate the textual class embeddings. specifically, given a hand-crafted prompt template "a photo of a [cls]", let s c represent the sequence embedding for the prompt "a photo of a [cls] c " corresponding to the c-th class. given an input image i, the output probability is given by: context optimization (coop) @xcite . designing hand-crafted prompts in clip for every downstream data set is a tedious and time-consuming task. to mitigate this issue of prompt engineering, coop @xcite proposed to learn the prompts directly from the data by replacing the hand-crafted prompt with a context vector comprising of m tunable vectors. let the context vector be represented as @xmath3, u 2 , • • • , u m }, where u i represents a 512-dimensional vector 2 . unlike the hand-crafted prompt template, the tunable prompts are now designed as to allow the exchange of information learned from the data, the context vector u is common across all the class categories. finally, the context vector u is learned by minimizing the cross-entropy loss 2 the vector ui is of same dimension as the wordembedding of class names [cls]c. in this study, we primarily use clip-vitb/16 model where text embeddings are projected in a 512-dimensional space. between the ground-truth and predicted label as follows: ## intcoop: interpretability-aware prompt tuning in this section, we provide a detailed overview of our proposed prompt-tuning approach intcoop. in section 4.1, we detail the process of extracting attribute information from a given image. next, in section 4.2, we delve deeper to understand the process of generating image-conditioned prompts. finally, we outline our entire training framework in section 4.4, demonstrating the integration of all components into the training pipeline. similar to past context optimization approaches @xcite , intcoop can also be easily applied to a broad family of clip-like vision-language models. ## discussion for additional discussion, we refer the reader to appendix e. intcoop learns interpretable prompts. in this section, we delve deeper into understanding the quality of the attributes generated by intcoop during inference. given a test image i with true label y, we first extract its corresponding learned attribute embedding a(v(i)). to evaluate the quality of this embedding, we utilize the blip-2 model to produce an attribute label a i . we evaluate two setups: (1) firstly, to validate the quality of the attributes generated by intcoop, in figure 3 , we visualize the cosine similarity of the learned attribute embedding a(v(i)) and the blip-2 generated label a i . across all datasets, we observe a high similarity between the generated attribute embedding and the blip-2-generated label. this confirms that intcoop effectively learns contextually relevant and correct attribute information. (2) secondly, as illustrated in figure 4 (appendix), we observe that the prompts crafted using the learned attribute embedding a(v(i)) closely align with the original prompt format "a photo of [a] [cls]", as evidenced by high cosine similarity. on the other side, prompts lacking the attribute information exhibit reduced similarity. this analysis highlights that during inference, intcoop generates prompts with interpretable compositional information, thereby explaining the improved performance. ## ablations on design choice in this section, we delve into a comprehensive exploration of the design choices made in our proposed framework. ablations on visual prompting. as illustrated in section 4.1, to enhance image representations intcoop effectively utilizes the deep visual prompting approach. to substantiate our design rationale, we conduct ablation experiments as outlined in ablations on instance conditioning. to condition the prompts on the input image, prior studies @xcite have proposed the direct addition of the image embedding to the context vector. however, as elaborated in section 4.2, we employ a multi-head attention module for generating image-conditioned prompts in the training of intcoop. in ## conclusion in our paper, we initially observe that incorporating relevant attributes into prompts significantly improves image-text alignment in clip. to achieve this enhancement, we present a novel technique called intcoop, which integrates these attributes into learned prompts. this integration is made possible by leveraging a blip-2 @xcite model to annotate attributes in few-shot datasets. with the image as a conditioning factor, we devise a hypernetwork responsible for predicting embeddings corresponding to attribute descriptors. simultaneously, we optimize the other context vectors using clip's contrastive objective. our comprehensive testing across diverse datasets underscores the significant improvement in zero-shot performance achieved by intcoop.