T ransfer CVLM : Transferring Cross-Modal Knowledge for Vision-Language Modeling (2024)

Abstract: Recent large vision-language multimodal models pre-trained with huge amount of image-text pairs show remarkable performances in downstream tasks. However, the multimodal pre-training has limitations in terms of resources and training time when it comes to obtaining new models that surpass existing models. To overcome these issues, we propose TransferCVLM, a method of efficient knowledge transfer that integrates pre-trained uni-modal models (and cross-modal fusion-encoder) into a combined vision-language model (CVLM), without pre-training the CVLM with large amount of multimodal data, and then for each task application, fine-tunes the CVLM and transfers the multimodal knowledge of a teacher vision-language model to the CVLM by using knowledge distillation techniques. We demonstrate that 1) the fine-tuned CVLM performs comparable to other vision-language models of similar size, that 2) the multimodal knowledge transfer consistently enhances the CVLM, and the knowledge-transferred CVLM composed of large-size unimodal models outperforms the teacher multimodal model in most of downstream tasks, and that 3) TransferCVLM can also be used for model compression when using small-size unimodal models. We estimate that the training of TransferCVLM takes only 6% of pre-training of other vision-language models. Our code is available at https://github.com/DMCB-GIST/TransferCVLM.

## introduction ever since neural networks and deep learning proved successful in addressing visual and textual comprehension problems, numerous tasks have surfaced with the goal of comprehending and processing both images and text simultaneously, such as visual question answering, image-text retrieval, * hyunju lee is the corresponding author. and visual entailment @xcite @xcite . developing models proficient in these tasks is vital given that many real-world problems involve multiple modalities. a trend in deep learning is self-supervised learning, and models pre-trained in self-supervised manner are predominant in most domains. there have been several approaches to pre-training multimodal models for the multimodal tasks, either pre-training a randomly initialized model from scratch @xcite , or combining two unimodal pre-trained models and further pre-train the combined model @xcite @xcite . both approaches require to pre-train the models with unlabeled multimodal data, typically pairs of text and image, to learn cross-modal dependencies, but the pre-training with a large amount of unlabeled multimodal data is hugely costly. moreover, for the latter approach, when there are new better unimodal pre-trained models (e.g. gpt-3 < gpt-3.5 < gpt-4) @xcite , it should repeat the costly pre-training. to reduce the repetitive cost of large-scale multimodal pre-training of the second approach, we propose a method to transfer cross-modal dependencies from an existing multimodal pre-trained model to a combination of two new unimodal pretrained models for a given task. with the proposed knowledge transfer, when there are better unimodal pre-trained models, we do not have to redo pretraining the models with a large amount of unlabeled multimodal data. we present transfercvlm, a method of efficient knowledge transfer from an existing visionlanguage pre-trained model (vlm; e.g. flava @xcite ) to a combination of two unimodal pre-trained models for vision and language (e.g. bert @xcite , vit @xcite ). for this paper, we consider the two popular settings that the vision pre-trained model is encoder-only, and the language pre-trained model is either encoder-only or encoder-decoder. we first build a new multimodal model (cvlm) by concatenating the encoders of the two unimodal pretrained models and connecting the two encoders with, if any, the fusion encoder (fe) of the existing multimodal model (vlm). we then fine-tune both multimodal models (vlm, cvlm) with labeled multimodal data of a given task and finally transfer the cross-modal dependencies of the existing model (vlm; teacher) to the new model (cvlm; student) via a knowledge distillation technique. our contributions can be summarized as follows: ## methodology we introduce the implementation of transfer-cvlm first to build an encoder-only model in sections 2.1 (cvlm) and 2.2 (knowledge transfer) and then an encoder-decoder model in section 2.3. we also explain details of experiments about adapting transfercvlm to build a small-size model in section 2.4. figure 1 depicts the overview of transfercvlm for encoder-only model. ## conclusion we present a novel knowledge transfer method transfercvlm for quickly building enhanced multimodal models of cvlm by using latest unimodal pre-trained models and by distilling crossmodal knowledge from a teacher of multimodal pretrained model, without any pre-training with large amount of multimodal data. the proposed method consistently improves performance of cvlm across downstream tasks and across model architectures, even outperforming the teacher model in many cases when using large-size unimodal pretrained models. when using small-size unimodal pre-trained models, transfercvlm can be used as model compression method without large-scale pre-training.