AP o LL o : Unified Adapter and Prompt Learning for Vision Language Models (2023)

Abstract: The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.

## introduction recent years have witnessed tremendous success of largescale pre-trained language models @xcite @xcite @xcite @xcite as well as visual models @xcite @xcite leading to a surge in pre-trained vision-language models @xcite @xcite @xcite for multi-modal downstream tasks. despite being largely successful in terms of generalization capabilities, these vlp models such as clip @xcite are difficult to fine-tune for few-shot learning-based downstream tasks @xcite . this is mainly because of the massive scale of these models coupled with the deficiency of training data figure 1 : apollo sets a new state-of-the-art across several datasets for the base-to-novel class generalization task by learning a vision-language representation using a unified adapter and prompt tuning strategy under a few-shot setting. each axis denotes an accuracy value, either on base or novel class (specified accordingly) for the corresponding dataset (refer to the effectiveness of such fine-tuning-based methods notwithstanding recent investigations reveal @xcite the caveats of these approaches -such as neglecting the useful knowledge gained from the preceding pre-training step and overfitting with the downstream tasks. although we find a considerable wealth of studies that involve text-based prompt learning @xcite @xcite , the same for the visual pipeline still remains an under-explored area. main results: we present apollo , a novel unified adapter and prompt learning approach for vlp models to tackle the generalizability problems in few-shot scenarios. we enforce consistency between intra-modal encoders using consistency-guided contrastive loss @xcite . this is done to teach the association between the query and the semantically similar negative in-batch keys. to further enhance cross-modal alignment, we employ cross-attention in the modality-specific adapter layers. this leads to better awareness of the multimodal features. experimental results over a varied range of recognition datasets demonstrate the efficacy of our approach. some novel aspects of our work include: ## conclusions and future work we present apollo , a unified multi-modal adapter and prompt tuning method to facilitate fewshot learning without compromising zero-shot generalizability. to this end, we leverage contrastiveconsistency loss and ensure cross-modal synergy which results in an improved performance on three carefully chosen downstream tasks: base-to-novel generalization, cross-dataset evaluation, and domain generalization on 10 benchmark datasets as observed from the experimental results. moreover, the ablation analysis outlines the salient contributions of each of the sub-modules. future work can include assessing the performance of such frameworks for fine-grained tasks which remains an open problem. moreover, the adapter layers may lead to over-parameterization which can be further optimized in subsequent works.