D ue T : Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (2023)

Abstract: This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning. In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters. Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting. We report that DueT outperformed simple fine-tuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.

## introduction pre-training of vision and language for learning the semantic correspondence between the two modalities has achieved impressive performance gains in a wide variety of downstream tasks. in particular, contrastive learning methods for image-text dual encoder models such as clip @xcite and align @xcite have attracted much attention. these models learn visual concepts directly from raw text about images by jointly training image and text encoders. a crucial aspect of the high performance of such models is that they are trained on a massive amount of image-text data. for example, clip was trained from scratch on a dataset of 400 million pairs collected from the web without human annotations, and align was trained on a dataset of over one billion pairs. thus, to build a model for a new non-english language or a domain not covered by the existing model, it necessitates collecting ample image-text pair data for the target language or domain prior to training. another aspect is that the models are trained with substantial computational resources because it is important to form very large mini-batches for contrastive learning. therefore, building on top of powerful pre-trained models with more data-and compute-efficient strategies is highly demanding. however, this issue has not yet been thoroughly investigated; most prior studies @xcite @xcite focused on training models from scratch using largescale corpora. in an effort to address this challenge, lit @xcite , a parameter-efficient approach for developing vision and language models, has been proposed. it involves fixing the parameters of pre-trained uni-modal image encoders and exclusively training the text encoders. however, a comprehensive evaluation of the model's performance remains to be conducted. in this study, we propose duet (dual-adapter tuning), a method to learn the vision and language models. duet uses a single-modal pre-trained model as fixed values for the parameters of each encoder and adds adapters equipped with a gating mechanism to both encoders (figure 1 , right). in experiments in the english and japanese domains, duet outperformed simple transfer learning and lit @xcite . the main contributions of this study are as follows: ## methodology the goal of this method is to efficiently construct vision and language models for diverse languages and domains, through knowledge transfer from unimodal encoders such as vit and bert. to achieve this goal, we propose to train only gated adapter units (gaus), which are designed for image-text contrastive learning, that are added to each of the frozen pre-trained dual encoders. this method can transfer and extend the excellent representations possessed by the pre-trained uni-modal encoders to the more challenging task of cross-modal understanding without requiring substantial fine-tuning of the original models. the gaus learn to modulate and adapt the internal features of both the image and text encoders, effectively bridging the gap between the two modalities and enabling them to interact in a more meaningful way. ## conclusion we propose duet, which performs transfer learning by using adapter tuning @xcite 0 5 10 15 20 25 score(%) sst2 stl10 flowers102 mnist oxfordiiitpet caltech101 sun397 country211 dtd stanfordcars cifar10 fgvcaircraft food100 cifar100 gtsrb +0.88 +1.05 +1.10 +1.40 +3.04 +3.85 +4.32 +4.59 +4.89 +6.01 +7.88 +8.31 +9.59 +15.52 +21.08 +0.27 -0.46 +0.42 +1.68 -0.52 +0.09 +0.98 +3.42 +3.99 +4.10 +4.13 +4.06 +6.99 +8.00 +24.43 duet finetuning figure 5: linear probe on uni-modal vit vs. linear probe on ft-clip and duet. bold: scores within the 99.5% clopper-pearson confidence interval showing significant difference from uni-modal vit. red: scores where duet significantly outperformed ft, determined by the 99.5% clopper-pearson confidence interval. blue: scores where duet significantly underperformed ft, determined by the 99.5% clopper-pearson confidence interval. in the pre-training of clip @xcite , a visual and language model constructed by contrastive learning. a single-modal pre-trained model is used as fixed values for the parameters of each encoder, and adapter gaus equipped with a gating mechanism extended from the one described in @xcite are added to both encoders for training. transfer learning of uni-modal pretrained models is an important topic in regard to clip, which requires a large amount of training data. the fact that we were able to devise a method that is superior in performance and parameter efficiency to fine-tuning, lit @xcite , and lora @xcite is a major contribution to research on vision and language models. the results of the experimental evaluations also gave us insight into the number of additional parameters, number of training data, and necessity of gates for transfer learning in the construction of japanese clip. the results of this research will contribute to the development of services such as dialogue, navigation, and content generation and retrieval that require integrated visual and linguistic understanding.