Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models (2024)

Abstract: Vision-language models (VLMs) like CLIP have demonstrated remarkable applicability across a variety of downstream tasks, including zero-shot image classification. Recently, the use of prompts or adapters for efficient transfer learning (ETL) has gained significant attention for effectively adapting to downstream tasks. However, previous studies have overlooked the challenge of varying transfer difficulty of downstream tasks. In this paper, we empirically analyze how each ETL method behaves with respect to transfer difficulty. Our observations indicate that utilizing vision prompts and text adapters is crucial for adaptability and generalizability in domains with high difficulty. Also, by applying an adaptive ensemble approach that integrates task-adapted VLMs with pre-trained VLMs and strategically leverages more general knowledge in low-difficulty and less in high-difficulty domains, we consistently enhance performance across both types of domains. Based on these observations, we propose an adaptive ensemble method that combines visual prompts and text adapters with pre-trained VLMs, tailored by transfer difficulty, to achieve optimal performance for any target domain. Upon experimenting with extensive benchmarks, our method consistently outperforms all baselines, particularly on unseen tasks, demonstrating its effectiveness.

## introduction vision-language models (vlms), such as clip @xcite and align @xcite , have demonstrated remarkable applicability across various downstream tasks such as image classification. a distinctive feature of these vlms for image classification is their ability to classify unseen classes that have not been encountered during pre-training through zero-shot inference, which is not possible to traditional vision models. ## backgrounds here, we provide a brief overview of the background related to our method. for a detailed explanation with more related works is in appendix e. zero-shot clip. clip @xcite is designed for creating visual features based on natural language guidance. the clip model can perform zero-shot inference, classifying an image into one of c possible classes without additional training. this is achieved by calculating the cosine similarity between an visual feature z, derived from the visual encoder, and the text features of each class {t i } c @xmath0, which are obtained from the text encoder. for processing the image, let us define the visual encoder as v, which comprises l v layers, denoted as {v i } l v @xmath1. the encoder takes patch embeddings e 0 ∈ r m ×dv as input, which are obtained by dividing the image i into m fixed-size patches. patch embeddings e i is then fed into the (i + 1) th transformer block (v i+1 ) along with a learnable class ([cls]) tokens c i . this process is sequentially carried out through all l v transformer blocks, formulated as follows: here, [•, •] denotes the concatenation operation. we can obtain the text features in a similar way with word embeddings w 0 = [w 1 0 , . . . , w n 0 ] ∈ r n ×d l and text encoder t which is consist of l t layers {t i } l t @xmath2, as follows: the predicted probability for class i is as: also, we can replace eq. ( 3 ) to belows by introducing text prompt for the fisrt j t layers of text encoder: here, we train the visual and text prompt for the first j v and j t layers of corresponding encoders. adapter-style tuning for clip. to enable adapter-style tuning, we replace eq. ( 2 ) and eq. ( 4 ) by introducing imgadapt and txtadapt which are shallow stacking networks upon the frozen clip model @xcite @xcite . ## motivating observations here, we analyze the behavior of visual and text encoders depending on different tuning methods and transfer difficulty of target domains within the framework of etl. to accomplish this, we begin by categorizing domains based on their relative transfer difficulty (rtd), which is a metric first defined by @xcite . definition 1 (relative transfer difficulty @xcite ). let f (•) and g(•) be random classifiers where the precision of each equals 1/c, and zeroshot clip, respectively. also, prec f and prec g denote the precision of classifiers f and g. then, rtd is formulated as follows: under this metric, we identify eurosat, dtd, and fgvc aircraft as the three most challenging domains, while imagenet, sun397, and stanford cars are recognized as the three easiest domains. we will primarily focus on these six domains to clearly demonstrate the impact of rtd on vlms' behavior. to assess adaptability and generalizability, we train the clip-b/16 utilizing each prompt 5 10 20 30 70 75 80 cars 5 10 20 30 30 35 40 45 aircraft 5 10 20 30 70 80 90 eurosat vpt (base) vpt (novel) tpt (base) tpt (novel) vpt+tpt (base) vpt+tpt (novel) figure 3: comparison of the accuracy (%) of base and novel categories using tpt, vpt, and their combination (vpt+tpt) on three transfer learning datasets over various training epochs. eurosat sun397 eurosat 63.6 64.05 figure 4: t-sne (van der maaten and hinton, 2008) plots of visual features for novel category with their corresponding labels (left), zero-shot clip prediction (middle), and prediction with tpt (right). 50 samples are randomly selected from each class in eurosat and sun397, using all 5 classes in eurosat and 5 randomly chosen classes from sun397. dotted lines within the t-sne plot represent the decision boundaries corresponding to each class, indicated by the same color. observation 1. vpt offers better generalizability than tpt. while tpt has greater adaptability to seen classes in low-difficulty domains, it is not effective for high-difficulty domains and shows overfitting to the base classes. we commence with an analysis of the separate behavior of visual and text prompts during the tuning process. fig. 2 illustrates the performance discrepancy between the two categories for each method. across all domains, vpt consistently shows the smallest performance gap for every shot number, indicating reduced overfitting to base classes. this observation is especially prominent in domains with high rtd though the trend is not as pronounced in domains with low rtd. we also observe that combining vpt and tpt does not consistently mitigate the overfitting of tpt, as evidenced by the larger performance gap in fgvc aircraft and eurosat compared to tpt alone. fig. 3 displays the comparative performance of base and novel categories over different epochs. while all prompt tuning methods show an improvement in base category performance at the expense of generalization, vpt consistently exhibits a lesser decline in novel category performance. notably, for challenging domains like fgvc aircraft and eurosat, vpt exceeds the novel performance of tpt and their combination regardless of epoch. observation 2. low class separability of visual features is the primary reason for the overfitting of tpt on high rtd. class separability is a critical factor in determining the transferability of a source model to a target domain @xcite . to determine the class separability of visual features, we use the ratio of intra-to inter-class cosine similarities @xcite . fig. 5 demonstrates that the ratio is higher in domains with lower rtd, which are considered easier, and lower in more challenging datasets with higher rtd. these findings suggest that the class separability highly correlates with transfer difficulty, strongly influencing the overfitting risk of tpt on high rtd domains. to see how class separability affects tpt, we further explore the visual features and predictions image net (0.14) sun (0.38) cars (0.77) caltech (1.08) food (1.15) ucf (1.42) flowers (1.52) pets (3.01) fgvc aircraft (4.07) dtd (4.95) euro sat (18.37) datasets (rtd 10 2 ) 1.0 1.2 1.4 1.6 intra/inter ratio cosine similarity ratio for intra/inter-class base novel h.m 65 70 75 80 acc. (%) easy base novel h.m 40 50 60 70 challenge vpt vpt+ta vpt+tpt ta tpt tpt+va figure 6: comparison of the combined effectiveness of prompt tuning and adapter-style tuning. "easy" refers to three domains with low rtd, and "challenge" refers to three domains with high rtd. observation 3. ta effectively enhances adaptability with a low risk of overfitting when employed with vpt, especially on higher rtd datasets. fig. 6 shows that while ta and vpt each exhibit less adaptability than tpt alone, together they outperform across all categories, signifying both high adaptability and generalizability. this advantageous combination is particularly significant for higher rtd, while the performance improvement in novel categories with lower rtd is marginal. this synergy occurs because vpt enhances the class separability in visual features, allowing the linear transformation of classifier weights to suffice for adaptation, as depicted in fig. 7 . ta simply modifies the features of the pre-trained text encoder, preventing overconfidence in the decision boundary, especially for domains with high rtd and low class separability. in addition, we conduct experiments using a combination of tpt and a visual adapter (va). however, this combination proves less effective than integrating vpt and ta, further emphasizing the importance of visual feature separability. observation 4. by modulating the influence of ta through an ensemble of pre-adapter and postadapter features, each with a domain-specific coefficient, we can significantly improve generalization in low rtd domains while maintaining high performance in high rtd domains. while combining vpt and ta has great synergy in high rtd domains, utilizing ta can result in the loss of some general knowledge from the original clip, which is crucial for domains with low rtd. this is evident in tab. 1, as naïvely using vpt and ta together may lead to a degradation in performance on novel classes in domains with low rtd. this is because for low rtd, a lot of tasks within the domain need to lie in the region of general knowledge, as illustrated in fig. 1(b ). but the training of a ta creates a task-specific boundary which may not be optimal for other tasks within the same domain. in domains with high rtd, task-specific knowledge gained from adapters can also enhance performance on unseen tasks, as the general knowledge is often insufficient for these domains. this degradation in domains with low rtd can be mitigated by diminishing the influence of ta. inspired by the residual connection in adapter-style tuning methods @xcite , we use an ensemble of pre-adapter and postadapter features for the text encoder. this ensemble, defined with coefficient α, can be expressed as: as tab. 1 illustrates, the ensemble method improves performance in domains with low rtd. however, using pre-adapter features can yield suboptimal outcomes in more challenging domains. for instance, performance on eurosat drops from 77.73% to 75.87% when α is set as a fixed coefficient, as domains with high rtd demand more from ta. by optimally setting α for each domain, we consistently outperform zero-shot clip across all domains by effectively combining general and task-specific knowledge tailored to each domain's needs. observing this optimal coefficient, we note that that more challenging domains typically require a higher coefficient. these findings highlight the necessity of a method to calculate an adaptive coefficient of ensemble, which would modulate ta activation according to domain and its rtd. ## method based on our observations, we propose a new method, apex, which is a difficulty-agnostic approach that utilizes an adaptive ensemble with tuning methods including vpt and ta. ## experiments we describe our experimental setup and results for verifying superiority of our method. additional experimental results are described in appendix c. ## conclusion we propose apex to address the challenges of conventional prompt and adapter-style etl methods for vlms. our approach incorporates two key components based on our observations: (1) using vpt and ta for exploiting the property of each modality and (2) adaptive ensemble coefficient in the inference stage. we empirically demonstrate the superior performance of apex, consistently achieving a better performance than the previous methods.