Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022)

Abstract: Current text-image approaches (e.g., CLIP) typically adopt dual-encoder architecture using pre-trained vision-language representation. However, these models still pose non-trivial memory requirements and substantial incremental indexing time, which makes them less practical on mobile devices. In this paper, we present an effective two-stage framework to compress large pre-trained dual-encoder for lightweight text-image retrieval. The resulting model is smaller (39% of the original), faster (1.6x/2.9x for processing image/text respectively), yet performs on par with or better than the original full model on Flickr30K and MSCOCO benchmarks. We also open-source an accompanying realistic mobile image search application.

## introduction text-image retrieval is the task aiming at retrieving a list of relevant images from a large set of images given a textual query specified by the user. recently, large-scale vision-language pretraining (vlp) has spawned models @xcite @xcite that established state-of-the-art results in various vision-language tasks @xcite , including text-image retrieval. existing vlp models for text-image retrieval can be divided into two categories: cross-encoder architecture and dual-encoder architecture. cross-encoder models show better retrieval accuracy by allowing fine-grained cross-modal attention among image and text. however, they are prohibitively slow to apply to the entire image pool because each image has to go through the deep transformer again whenever a new text query comes in. moreover, most cross-encoder models rely on external object detection models @xcite to extract visual features, which further increase memory consumption. on the other hand, dual-encoder models are more scalable in that they allow pre-computing image representations as reusable vectors independent of the text queries. these image vectors can be indexed and efficiently retrieved at runtime using approximate nearest neighbor (ann) search @xcite . as long as the image pool remains unchanged, the image encoder is not required. however, a more practical scenario calls for dynamic indexing of new images into the pool (e.g., private photo collections on mobile devices), which requires both the image encoder and the text encoder to be resident in memory. this makes the above approach less practical on mobile devices with limited memory and processing power. unfortunately, little attention has been paid to fulfill this need. in this paper, we show that a large dualencoder model can be compressed into a much smaller and faster counterpart while retaining its retrieval accuracy using a novel two-stage compression framework. in the first stage, we make use of abundant non-paired texts/images to separately compress text or image encoder with an effective intra-modal contrastive knowledge distillation scheme. in the second stage, we sequentially fine-tune the distilled image or text encoder on paired text-image data with comprehensive learning objectives. using clip @xcite as the target model, our compressed models deliver comparable performance on mscoco and flickr30k while being just 39% of the original size and 1.6x/2.9x times faster for processing image/text. detailed ablation study shows the effectiveness of each component in the compression framework and their synergistic effects. our contributions are three-folds: 1) an effective compression framework tailored for lightweight text-image retrieval; 2) a leaner and faster model with competitive accuracy; 3) open-sourced models and text-to-image search mobile applications on both ios and android at https://github. com/drsy/motis . dual-encoder. in contrast to cross-encoder, dual-encoder architecture @xcite trains two seperate encoders for vision and language modalities. the exact choices of encoder architecture may be different. for example, clip utilizes transformers for both visual and text encoders, while align @xcite uses pre-trained bert as text encoder and effi-cientnet as visual encoder. in dual encoder, interactions between different modalities take place only at the final encoder layer, resulting in slightly worse performance compared to cross-encoders. nevertheless, this late-interaction scheme of dualencoder allows for efficient similarity computation, thus rendering it suitable for prividing real-time searching. ## conclusion in this paper, we present a two-stage framework for lightweight text-image retrieval. experiments on two benchmarks show the effectiveness of each component in the framework and the best performance is achieved when combining them together. it holds the merit of reducing model size and accelerating inference time, making memory/responsesensitive applications more practical.