L ightning DOT : Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021)

Abstract: Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computational cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by extracting pre-cached feature indexes offline, and employing instant dot-product matching online, which significantly speeds up retrieval process. In fact, our LightningDOT achieves superior performance across mainstream ITR benchmarks such as Flickr30k and COCO datasets, outperforming existing pre-trained models that consume 1000 times magnitude of computational hours using the same features.

## introduction image-text retrieval (itr) has been widely studied as a staple benchmark task in both nlp and computer vision communities. traditional itr search engines typically deploy ranking-based models built upon visual-semantic embedding matching @xcite or deep cross-modal fusion with attention mechanism @xcite . earliest works @xcite @xcite 4. propose later studies @xcite @xcite improve this paradigm by employing advanced region-level visual encoder (e.g., faster-rcnn) and applying cross-attention between word features and region features for multimodal fusion (figure 1(b) ). with the advent of transformer @xcite and bert @xcite , crossmodal retrieval tasks are more recently dominated by vision-and-language (v+l) pre-trained models, such as vilbert @xcite , uniter @xcite , oscar @xcite , and villa @xcite . large-scale pre-trained models learned from massive corpus of image-text pairs can power heterogeneous downstream tasks that take diverse modalities as inputs (e.g., text, image, video, audio). these models benefit from the self-attention mechanism in transformer architecture, learning joint image+text embeddings through pre-training objectives such as masked language modeling (mlm) and masked region modeling (mrm) (figure 1(c) ). however, the very ingredient that engenders the success of these pre-trained models, crossmodal attention between two modalities (through self-attention), also destines the inevitable latency and huge computation cost in training and deploying such massive-scale models. for example, uniter @xcite builds upon 12/24 transformer layers, and trains over 10 million image+text pairs. the inference time of such large models with 110 million parameters is 48 seconds on average for text query from coco dataset @xcite , not scalable in real-life applications serving millions of queries per second. to make real-time itr possible with low latency, we ask a bold question: can we go back to the beginning, reverting to simple dot product for efficient cross-modal retrieval? to make this retro experiment feasible, we rely on transformer to pre-train high-quality image and text encoders, but use efficient dot product for multimodal fusion instead of computationally heavy self-attention. to still facilitate effective cross-modal embedding learning, we use a special [cls] token on both encoders, which transfers the learned embedding from the other modality (figure 1(d) ). we name this new paradigm lightningdot, for its lightening speed benefiting from dot product computation. by removing the time-consuming cross-attention between modalities, the model can learn visualsemantic embeddings without extensive matching between each image-text pair during inference, as used in existing pre-trained models @xcite @xcite . further, by eliminating the dependency on real-time computation over image-text pairs, we can compute all image and text embeddings independently offline just for once, and reuse these embeddings as cached indexes for new queries on the fly (figure 2 ). for model training, we propose three learning objectives to jointly train two transformer blocks: image encoder and language encoder. specifically, visual-embedding fused mlm (namely vmlm) and semantic-embedding fused mrm (namely smrm) ensure cross-modal information is harnessed even without cross-modality self-attention. a cross-modal retrieval objective (namely cmr) encourages the model to learn multimodal fusion through pre-training. to maintain competitive model performance, we further introduce a reranking mechanism to bring back the benefit of cross-attention methods. in summary, lightningdot is designed with late fusion to learn visual-semantic embeddings. experiments on popular itr benchmarks show that lightningdot is 600/1900 times faster than existing pre-trained models on flickr30k/coco, while achieving new state-of-the-art results. when retrieving from larger candidate pool (>120k images), lightningdot is 23,000 times faster. to the best of our knowledge, this is the first known effort on improving v+l model efficiency. image-text retrieval early cross-modal embedding works @xcite @xcite focus on using a twostream model to learn a unified visual-semantic embedding, with progressive improvement on two popular benchmarks: flickr30k @xcite and coco @xcite . later methods with cross-attention @xcite @xcite become more popular, with significant performance gain. pre-trained v+l models also fall into this category. by exploiting large-scale image-text datasets, pretrained v+l models further push the performance on flickr30k and coco. although achieving high recall, cross-attention requires excessive computation cost during inference that cannot be overlooked. 2 in this work, inspired by dense retrieval in text retrieval domain @xcite @xcite @xcite , we propose a more efficient attention-less framework. with pre-training, our model achieves better performance while being significantly faster than cross-modal attention methods. note that the proposed approach is orthogonal to model compression techniques that reduce the number of layers/parameters @xcite , since we do not reduce the number of parameters from the uniter baseline. these two approaches can be combined to further boost the speed, which is an interesting future work direction. ## lightningdot framework in this section, we present the proposed light-ningdot framework, which consists of two deep transformers as image and language encoders. we first introduce three tasks designed to pre-train the model, then present our inference pipeline from offline feature extraction to online instant retrieval. ## experiments this section discusses our experiments on pretraining and evaluating lightningdot on downstream itr benchmarks. ## conclusion in this paper, we propose a pre-training framework that learns joint visual-semantic embedding without any cross-attention between modalities. light-ningdot outperforms previous state of the art, while significantly speeding up inference time by 600-2000× on flickr30k and coco image-text retrieval benchmarks. future work includes extending the efficient training framework to other v+l tasks. that capture at least three keywords (figure 6 ). we also present image retrieval results where the text query is sampled from coco dataset. we randomly sample 3 queries and present the results as below (ground truth on the top, retrieved top-10 images at the bottom). clearly, our model retrieves related images from the full dataset.