Advancing Vision-Language Models with Adapter Ensemble Strategies (2024)

Abstract: CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. However, the sheer size of these pretrained models makes full-model finetuning exceedingly costly. One common solution is the “adapter”, which finetunes a few additional parameters while freezing the backbone. It harnesses the heavy-duty backbone while offering a light finetuning for small downstream tasks. This synergy prompts us to explore the potential of augmenting large-scale backbones with traditional machine learning techniques. Often employed in traditional fields and overlooked in the large-scale era, these techniques could provide valuable enhancements. Herein, we delve into the “adapter ensembles” in the realm of large-scale pretrained vision-language models. We begin with a proof-of-concept study to establish the efficacy of combining multiple adapters. We then present extensive evidence showing these ensembles excel in a variety of settings, particularly when employing a Multi-Scale Attention (MSA) approach thoughtfully integrated into the ensemble framework. We further incorporate the LoRA to mitigate the additional parameter burden. We focus on vision-language retrieval, using different backbones under constraints of minimal data, parameters, and finetuning budgets. This research paves the way for a synergistic blend of traditional, yet effective, strategies with modern large-scale networks.

## introduction large-scale pretraining leverages massive data, robust architectures with strategic training to push performance boundaries @xcite @xcite . such pretraining strategy notably advances vision-language capabilities, pioneered by figure 1 : clip vit-b/16 ensemble ablation on selfattention and feedforward (sec. 2). y-axis/x-axis are the retrieval accuracy and the unit amount of learnable parameters in each layer. baselines (on-top, rb, mlp) and our ens are finetuned/evaluated on yfcc. sharing the same amount of learnable parameters, ensemble outperforms baselines and derives improvement when the number of ensemble parameters increases. clip @xcite , using contrastive learning on a massive image-text corpus, seamlessly integrates visual and linguistic modalities. various studies further advance vision-language pretraining by integrating auxiliary supervision (e.g., self-supervision/captioning loss) or extra information (e.g., tags/bounding boxes) @xcite @xcite @xcite @xcite . however, the necessity for extensive datasets and complex training pipelines for pretraining remain a challenge, particularly affecting finetuning efficiency. adapter @xcite is a favored technique for efficient finetuning, initially for language models such as bert @xcite and recently adapted for the visual domain @xcite . along with its variants such as lora @xcite and @xcite , adapter offers the solution by updating a few additional parameters with limited data while fixing the pretrained backbone. these approaches combine large-scale pretraining with small-sized efficient adapters, proposing a unified modeling pipeline. this fusion compels us to consider if we can borrow certain traditional machine learning techniques, which work well on previous smallsized scenarios but are easily ignored in the current large-scale era, to benefit the popular pretrained models. informed by this, our study delves the classic ensemble on adapter for large-scale visionlanguage pretrained models and assesses its impact on cross-modal retrieval. ensemble has long been a cornerstone in traditional machine learning field @xcite @xcite , combining diverse base learners to harness collective intelligence, thereby enhancing model performance and robustness @xcite @xcite . in past decades, early methods provided weak yet cheap base learners using limited data, the ensemble compensated by pooling their strengths. recently neural networks, with more data and complex models, present base learners of greater individual capability. yet, the ensemble continues to offer performance boosts @xcite , albeit at a cost, given the non-negligible resources to entirely train each deep network as a base learner. nowadays, the focus shifts towards leveraging single, robustly pretrained models, leaving ensembles less tapped for these larger models due to their prohibitive computational demands. however, our curiosity lies in applying ensemble to efficiently finetune large-scale pretrained models using adapters, which act as a nexus for integrating large-scale backbone and small-sized techniques. this study marks the initial exploration into the use of the adapter-based ensembles in large-scale pretrained models. we infuse the pretrained model with parallel learnable parameters in an ensemble fashion while fixing original weights. our proofof-concept study (sec. 2) shows substantial performance gain of ensemble over baselines (fig. 1 ). we further extensively validate its effectiveness with a well-designed multi-scale attention (msa) in an ensemble framework (sec. 3). finally, we enhance our strategy by incorporating lora @xcite technique, managing the extra parameter overhead to maintain efficiency with competitive performance even when scaling to ensemble applications. we summarize contributions of our study as below: ## ensemble proof-of-concept study ensemble is often interpreted as a weighting strategy @xcite , where data or feature fusion can be regarded as an ensemble process to some extent. for instance, residual connection @xcite is an ensemble process fusing identity mapping and learned residual information. in this section, we conduct an instructive experimental analysis as a proof-of-concept study to show the effectiveness of using an ensemble strategy on adapter. we finetune (using limited 0.1m data) and test on yfcc @xcite to compare our ensemble (att-ens/ffn-ens) with three baselines (on-top, att-rb/ffn-rb, att-mlp/ffn-mlp) on clip backbone. ## adapter ensemble we show the effectiveness of involving an adapter ensemble into a pretrained model in sec. 2. next, we introduce a bottleneck adapter baseline, a pyramid ensemble, and a well-designed multi-scale attention (msa) ensemble for our comprehensive validation on multiple settings. furthermore, we easily adopt lora @xcite into our ensemble design to ease the parameter burden caused by ensemble operation. ## related works vision-language retrieval multi-modal learning integrates information from different modalities @xcite for various tasks @xcite @xcite @xcite @xcite . recently, vision-language understanding @xcite becomes popular in this field mainly evaluated by retrieval task @xcite @xcite . such cross-domain retrieval is pioneered by vse++ @xcite , using hard-negative mining. scan @xcite designs cross-modal encoding for finegrained features. vsrn @xcite uses graph and recurrent networks to reason visual semantics. large-scale pretraining boosts the performance using massive web data @xcite . recent works @xcite @xcite explore different strategies for pretraining such as coca @xcite jointly using retrieval and captioning loss and blip @xcite utilizing cross-modal encoding. they significantly improves retrieval performance yet requires much more resource. herein, we explore an efficient ensemble, combined with adapter, to further enhance the pretrained visionlanguage backbones for retrieval tasks. ## conclusion our curiosity lies in exploring how traditional machine learning techniques, typically used for small-sized models, can be leveraged to benefit recent large-scale pretrained vision-language models. we identify adapter ensemble as an ideal fusion point, effectively finetuning large-scale models while seamlessly integrating small-sized methodologies. through a proof-of-concept study, we validate the ensemble adapter efficacy. we then demonstrate its effectiveness for vision-language retrieval on different settings. specifically, a multiscale attention (msa) is designed to benefit ensemble operation. furthermore, to address the potential increase in parameter requirements brought by the ensemble, we integrate the lora for msa, significantly reducing the parameter overhead. our empirical results showcase the ensemble capacity to improve the performance of large-scale pretrained models, achieving efficiency for data, parameter, and finetuning budgets.