Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022)

Abstract: The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7\times fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

## introduction recent large-scale dual-stream vision-language pre-training (vlp) models like clip @xcite and align @xcite , have shown remarkable performance on various downstream multimodal alignment tasks, e.g., imagetext retrieval and image classification. these models are pre-trained using cross-modal contrastive learning on tremendous image-text pairs and learn strong multimodal representations. despite their success, as mentioned by @xcite , their text encoder is relatively weak by only having a discriminative multimodal pre-training objective, ... ? answer: a napkin. ## proposed method we propose to distill multimodal knowledge from clip to bart for generative multimodal tasks, which takes the strengths from both sides (powerful multimodal representations of clip and text generation ability of bart). to this end, we propose three objectives (section 3.2). the overall architecture is illustrated in figure 2 . ## experiments to demonstrate the effectiveness of vlkd, we evaluate it on generative multimodal tasks for both zero-shot and finetuning. specifically, we test the image captioning task, and also the vqa task under the open-ended scenario. furthermore, we also run the model on nlu and nlg tasks to investigate the influence of vlkd on the text processing ability of the original pre-trained bart. ## ablation study knowledge distillation objectives. ## conclusion recent dual-stream vlp models (e.g., clip) are powerful in various multimodal classification and retrieval tasks. however, their ability of multimodal generation or pure nlp tasks is highly restricted. in this paper, we propose a novel knowledge distillation method to efficiently align clip's multimodal encoders and bart's textual encoder to the same mutlimodal space, as well as a crossmodal lm loss to consort bart encoder and decoder. this enables multimodal generation under zero-shot and also fully-finetuned settings without losing the original bart's nlp ability. empirical results show that our model achieves new stateof-the-art zero-shot performance on vqa and excellent performance on both nlp and multimodal tasks when finetuned, demonstrating the effectiveness of our proposed method.