E fficient VLM : Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning (2023)

Abstract: Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and latency constraints. In this work, we introduce a distilling then pruning framework to compress large vision-language models into smaller, faster, and more accurate ones. We first shrink the size ofa pre-trained large VLM and apply knowledge distillation in the vision-language pre-training stage to obtain a task-agnostic compact VLM. Then we propose a modal-adaptive pruning algorithm to automatically infer the importance of vision and language modalities for different downstream tasks and adaptively remove redundant structures and neurons in different encoders with controllable target sparsity. We apply our framework to train EfficientVLM, a fast and accurate vision-language model consisting of 6 vision layers, 3 text layers, and 3 cross-modal fusion layers, accounting for only 93 million parameters in total, which is 44.3% of the teacher model. EfficientVLM retains 98.4% performance of the teacher model and accelerates its inference speed by 2.2×. EfficientVLM achieves a large absolute improvement over previous SoTA efficient VLMs of similar sizes by a large margin on various vision-language tasks, including VQAv2 (+4.9%), NLVR2 (+5.6%), ITR (R@1 on TR +17.2%, on IR + 15.6% ) and COCO caption generation (CIDEr +6.5), demonstrating a large potential on training lightweight VLMs.

## introduction inspired by the success of large pre-trained language models @xcite in the field of natural language processing (nlp), recent studies @xcite @xcite @xcite in vision-language pretraining (vlp) have advanced the state-of-the-art on various vision-language tasks such as image captioning, visual question answering, and image-text retrieval. however, in both nlp and vision-language domains, large transformer-based pre-trained models often consist of hundreds of millions, if not billions, of parameters, bringing various practical challenges for deployment. as summarized in @xcite and @xcite , large pre-trained models require large amounts of space (in terms of gpu memory and disk storage) and heavy computing for fine-tuning and inference, which is both costly and may lead to negative environmental impact. furthermore, large models inevitably lead to low latency, which poses a challenge for the production environment. recent literature revealed that bert @xcite , a popular transformer-based pretrained language model, can be effectively compressed and accelerated via knowledge distillation @xcite @xcite . however, only a few prior works investigated building efficient vlms. for instance, @xcite introduced minivlm which combines a lighter object detector with minilm @xcite . @xcite further proposed distilvlm, which uses knowledge distillation to pre-train a compact vlm with the guidance of a large pretrained vlm. however, their approach is limited to object-feature-based vlms. as such, the vision feature extractor cannot be distilled together with the transformer model in an end-to-end manner, which limits the potential of knowledge distillation. as a result, existing compact vlms are generally falling short compared to regular-size vlms. in this work, we investigate strategies for vlm compression and introduce a distilling then pruning framework for compressing fully transformerbased vlms. specifically, in the first stage, we use knowledge distillation for task-agnostic compression of a pre-trained vlm by aligning the logits, attention distribution, and hidden representations between the student model and the teacher model. this results in a task-agnostic compact vlm that achieves competitive results on many downstream vision-language tasks by simply fine-tuning. the general distillation stage reduces the size of all modules (i.e., vision encoder, text encoder, crossmodal encoder) equally so that the compressed model can be versatile to different downstream tasks. however, our preliminary study, which is described in detail in section 3.3, shows that not all modules are created equal in a vlm and their importance drastically varies on different downstream vision-language tasks requiring different levels of understanding on either vision and text modalities. this indicates that compressing a vlm requires modal-and task-specific designs. therefore, in the second stage, we propose to prune the compact vlm when fine-tuning on different downstream tasks to flexibly adjust the model size/latency according to modal importance. concretely, we propose a modal-adaptive pruning strategy that regularizes the model with a differentiable approximation to the l 0 -norm regularization @xcite to automatically infer the importance of vision and language modalities with controllable target sparsity. in this way, our method can adaptively prune different modules in the vlm in the fine-tuning stage according to the relative importance of visionlanguage modalities on different downstream tasks. we apply our framework to compress x-vlm @xcite , a recent transformerbased vlm and train efficientvlm, a fast and accurate vision-language model. efficientvlm consists of 6 vision layers, 3 text layers, and 3 crossmodal fusion layers, accounting for only 93 million parameters in total, which is 44.3% of the x-vlm model. efficientvlm recovers 98.4% performance of x-vlm and accelerates its inference speed by 2.2×. experimental results show that despite being trained with fewer image-text pairs, efficientvlm achieves a large absolute improvement over distil-vlm, the previous best-performing efficient vlm with similar size and inference speed, on various vision-language tasks, including vqav2 @xcite (+6.7%), nlvr2 @xcite (+7.8%), itr-coco @xcite ) (r@1 on tr +19.9%, r@1 on ir + 15.6% ) and coco caption generation @xcite (cider +6.5), demonstrating a large potential on training lightweight vlms. to the best of our knowledge, our work is the first attempt to (1) compress a fully transformerbased vision-language model, and (2) combine knowledge distillation with (modal-adaptive) pruning for vision-language model compression. ## related work vision-language pre-training the existing work on vision language pre-training typically falls into two categories. most methods rely on object detection @xcite @xcite @xcite @xcite @xcite @xcite @xcite , where an image is represented by dozens of object-centric features. however, the object detection process requires high-resolution images as model input and is very time-consuming. moreover, most works under this category utilize pre-trained object detectors @xcite , and do not optimize the model in an end-to-end manner, yielding sub-optimal performance. therefore, recent works turn to encoding images by convolutional network @xcite @xcite or vision transformer @xcite , largely improving the inference speed. nevertheless, some recent work @xcite @xcite shows that understanding fine-grained vision language alignments (e.g. object-level) is critical for some downstream tasks such as visual reasoning and visual grounding. ## efficientvlm in this section, we present efficientvlm, a fast and accurate vision-language model trained with our distilling then pruning framework. we choose x-vlm @xcite , one of the state-of-the-art vision-language models, as the teacher model. foot_0 ## conclusion we introduce efficientvlm, a fast and accurate vision-language model trained with a distilling then pruning framework. empirical results show that efficientvlm retains 98.4% performance of the base-size teacher model while only preserving 44.3% parameters and achieving a speed-up ratio of 2.2×. efficientvlm also achieves a large absolute improvement over previous efficient vlms, demonstrating a large potential towards lightweight vlms.