MCAD : Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval (2024)

Abstract: Due to the success of large-scale visual-language pretraining (VLP) models and the widespread use of image-text retrieval in industry areas, it is now critically necessary to reduce the model size and streamline their mobile-device deployment. Single- and dual-stream model structures are commonly used in image-text retrieval with the goal of closing the semantic gap between textual and visual modalities. While single-stream models use deep feature fusion to achieve more accurate cross-model alignment, dual-stream models are better at offline indexing and fast inference. We propose a Multi-teacher Cross-modality Alignment Distillation (MCAD) technique to integrate the advantages of single- and dual-stream models. By incorporating the fused single-stream features into the image and text features of the dual-stream model, we formulate new modified teacher similarity distributions and features. Then, we conduct both distribution and feature distillation to boost the capability of the student dual-stream model, achieving high retrieval performance without increasing inference complexity. Extensive experiments demonstrate the remarkable performance and high efficiency of MCAD on image-text retrieval tasks. Furthermore, we implement a lightweight CLIP model on Snapdragon/Dimensity chips with only ~100M running memory and ~8.0ms search latency, achieving the mobile-device application of VLP models.

## introduction image-text mutual retrieval is a fundamental problem of multimodal learning, whose primary objective is to bridge the semantic gap between visual and textual modalities, enabling accurate match of image (text) based on the given text (image). however, aligning and matching visual and textual information is non-trivial due to the differences in their representations and structures. in recent years, the rapid growth of large-scale paired visionlanguage datasets @xcite has paved the way for the development of powerful models that can bridge the gap between visual and textual information. these models, known as vision-language pretraining (vlp) models, have shown remarkable capabilities in understanding both vision and language @xcite @xcite . typically, the dual-stream architecture, e.g., clip @xcite , align @xcite , facilitates autonomous processing of individual modalities through segregated streams, exhibiting inferior retrieval performance due to the lack of effective cross-modal feature fusion. nevertheless, the disentanglement of image and text encoder enable fast retrieval speed. on the contrary, singlestream models integrate information from multiple modalities during encoding through a deep interaction module, e.g., transformer block @xcite , commonly leading to superior retrieval performance but sacrificing flexibility and resulting in extremely low retrieval speed. thus, despite their size hindering deployment in lightweight scenarios like mobile devices, dual-stream models remain the preferred choice in industrial applications. in recent years, several works endeavor to transfer knowledge of large models into small models through distillation technology @xcite @xcite @xcite @xcite @xcite . but they just consider soft-label, feature, or attention map distillation from one teacher or homogeneous teachers. the strategy of homogeneous, multi-teacher distillation has not yet been explored. among these works, a critical question is how to distill the knowledge of the single-stream models into efficient dual-stream models. although dide @xcite proposes to employ cross-modal attention distillation to transfer the knowledge of the vilt @xcite teacher to a clip student, this is not a universal method since other single-stream structures, e.g., albef @xcite cannot discriminate explicit image and text features after cross-attention fusion so that the image-text attention maps are unavailable. another work loopitr @xcite considers only employ the single-stream output scores of top k hard examples chosen by the dual-stream model to enhance the dual-stream model itself, which cannot excite the whole ability of the single-stream model. so the integration of single-stream and dual-stream teachers is a non-trivial challenge. in this paper, we are motivated to propose a multi-teacher crossmodal alignment distillation (mcad) method to make full use of the information fusion ability of the single-stream model and the large-scale parallel training advantage of the dual-stream model. specifically, after extracting features through the frozen single-and dual-stream teacher models, we apply different learnable projection layers to align image or text features from different latent spaces, as shown in fig. 2 . finally, we employ similarity distribution and feature distillation based on the newly-formulated fused features to boost the performance of the dual-stream student model, as shown in fig. 1 . in summary, our main contributions are as follows: