C oca CLIP : Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval (2023)

Abstract: Large-scale pre-trained text-image models with dual-encoder architectures (such as CLIP) are typically adopted for various vision-language applications, including text-image retrieval. However, these models are still less practical on edge devices or for real-time situations, due to the substantial indexing and inference time and the large consumption of computational resources. Although knowledge distillation techniques have been widely utilized for uni-modal model compression, how to expand them to the situation when the numbers of modalities and teachers/students are doubled has been rarely studied. In this paper, we conduct comprehensive experiments on this topic and propose the fully-Connected knowledge interaction graph (Coca) technique for cross-modal pre-training distillation. Based on our findings, the resulting CocaCLIP achieves SOTA performances on the widely-used Flickr30K and MSCOCO benchmarks under the lightweight setting. An industry application of our method on an e-commercial platform further demonstrates the significant effectiveness of CocaCLIP.

## related work cross-encoder @xcite @xcite @xcite refers to multiple layers of dense crossmodal interactions, e.g., cross-attention @xcite , are typically employed to image and text representations for more fine-grained merge and alignment. although it often achieves superior retrieval accuracy thanks to the patch/token-level integration, the high memory cost and computation inefficiency make it impractical under time-critical real-world settings. oppositely, for dual-encoder @xcite @xcite , image and text features are encoded into a joint embedding space separately, and the modality interaction is only handled by a simple cosine similarity of the final image and text feature vectors. such approaches can be regarded as scalable and indexable: the specific choices of encoder architectures can be independent and dynamic, and the late-interaction scheme allows for efficient largescale searching. pre-training distillation for lightweight dualencoder architecture has been rarely studied. vanilla knowledge distillation @xcite can be referred to as the knowledge transfer from a teacher to a student in the same modality based on soft targets. however, it is a general procedure without awareness and pertinence for crossmodal learning. motis @xcite separately compresses text or image encoder with an intra-modal contrastive objective that aligns the output embeddings of the student and teacher of each modality, which can be seen as an alternative form of knowledge distillation. nevertheless, these methods ignore or do not find an appropriate approach to leverage the cross-modal distillation process. further than them, our method is dedicated to exploring the fully-connected knowledge interaction graph for dual-encoder distillation, which is a natural and effective extension. ## methodology in this section, we first give the preliminary knowledge, then propose our pre-training distillation framework with cona. finally, we introduce various supervision strategies. ## industry application we apply the proposed technique to end-to-end cross-modal retrieval in an e-commerce platform, where we vectorize the search queries and the products and then perform product retrieval and ranking with nearest-neighbor search @xcite @xcite , as shown in fig. 2 . we first collect massive data of text-image pairs from e-commerce products in our platform, where the titles of products can act as text information. we utilize most of the data to pre-train an e-commerce version of the clip model (denoted as ec-clip) with vit-b/32 as the image encoder, which is overly large for online deployment. for the remaining data, we utilize 3m pairs for distilling the lightweight ec-conaclip. to evaluate its effectiveness, we hold out a separate set of 100k pairs for fine-tuning and 5k/5k pairs used in validating/testing. in this set of experiments, we train ec-conaclip for 20 epochs in pretraining distillation, and fine-tune both ec-clip and ec-conaclip for 5 epochs. the remaining settings are the same as in section 4.1. in apart to the r@k metric, we also report the disk space (mb) and the acceleration rate of query per second (qps i for image and qps t for text) to evaluate model's memory footprints and inference speed. in tab. 4, we report the averaged results where the inference speed is tested on an nvidia tesla v100 (16g) gpu. as seen, the compressed ec-conaclip-6l only takes 44% disk space (254mb) of ec-clip meanwhile being 1.44×/1.92× faster with image/text queries. it also performs on par with ec-clip. our ec-conaclip-2l can further achieve up to 4.86× inference speed-up with text queries, and 64% size reduction (from 578mb to 206mb). we provide some case studies in a.4. ## conclusion in this paper, we propose cona for pre-training distillation with dual-encoder architecture. it gathers every type of knowledge interaction learning with appropriate supervision choice to benefit the cross-modal distillation. the resulting conaclip achieves superior performances on both general benchmarks and industry applications. for future work, we will explore more variants of visual encoders, and continue to tap the potential of dual-encoder distillation. from these cases, we can find that our model can better capture conceptual and fine-grained fashion information during cross-modal text-image retrieval, and maintain the cross-modal alignment effect of text-image samples after the lightweight distillation. for example, in case 1, our model more accurately captures the cartoon subject in the target commodity as "unicorn". in case 2, our model pays more attention to fine-grained information "2 layers double handle", while maintaining the correct perception of other information such as "stainless steel", "steamers pot" and "with lid". in case 3, our ec-conaclip better captures the color clue of "hot red". although the retrieval result of clip also conforms to the information of "sports tights high waist yoga pants", its color is more like "dark red". based on our distillation technique, the resulting model can sufficiently learn the perception ability of the teacher model about commodity fashion concepts and reduce matching errors.