Inflate and Shrink:Enriching and Reducing Interactions for Fast Text-Image Retrieval (2021)

Abstract: By exploiting the cross-modal attention, cross-BERT methods have achieved state-of-the-art accuracy in cross-modal retrieval. Nevertheless, the heavy text-image interactions in the cross-BERT model are prohibitively slow for large-scale retrieval. Late-interaction methods trade off retrieval accuracy and efficiency by exploiting cross-modal interaction only in the late stage, attaining a satisfactory retrieval speed. In this work, we propose an inflating and shrinking approach to further boost the efficiency and accuracy of late-interaction methods. The inflating operation plugs several codes in the input of the encoder to exploit the text-image interactions more thoroughly for higher retrieval accuracy. Then the shrinking operation gradually reduces the text-image interactions through knowledge distilling for higher efficiency. Through an inflating operation followed by a shrinking operation, both efficiency and accuracy of a late-interaction model are boosted. Systematic experiments on public benchmarks demonstrate the effectiveness of our inflating and shrinking approach.

## introduction efficiency and accuracy are two key factors of a retrieval system. in many cases, designing a retrieval system is striving to balance efficiency and accuracy. embedding-based methods @xcite @xcite are early works for tackling the cross-modal retrieval. they encode each image or text into a global embedding. then the text-image similarity is measured by the distance between their embeddings in the learned feature space. since there are no interactions between text and image, embedding-based methods only need o(n +m ) computational complexity to encode n images and m texts. the linear computational complexity of embedding-based methods makes them scalable to large-scale crossmodal retrieval. they hence have been widely deployed in real-world cross-modal retrieval tasks. recently, inspired by the great success of transformer @xcite and bert @xcite in natural language processing, some methods @xcite @xcite @xcite investigate cross-bert model to exploit the cross-modal attention and devise several pre-training tasks. benefiting from cross-modal attention and pre-training, they have achieved significantly higher retrieval accuracy than their embedding-based counterparts. nevertheless, the heavy text-image interaction from utilizing cross-modal attention leads to an o(n m ) computational complexity in encoding when calculating the similarities between n images and m texts. the quadratic computational complexity of cross-bert methods makes them not suitable for large-scale cross-modal retrieval applications. several methods attempt to gain satisfactory efficiency and maintain high accuracy through trading off efficiency and accuracy. these methods can be coarsely grouped into two categories: twostage methods @xcite @xcite and late-interaction methods @xcite @xcite . the two-stage methods apply a retrieve-and-rerank strategy. given a query, in the first stage, they conduct a coarse-level retrieval through an embedding-based method to obtain an initial top-t list of potentially relevant items. then in the second stage, the items in the top-t list are re-ranked through a powerful cross-bert model. since the heavy interactions are only deployed in the second stage, and t is smaller than the number of total items in the corpus, the efficiency is boosted. nevertheless, to obtain a satisfying retrieval accuracy, t should be large enough. in consequence, two-stage methods cannot achieve high efficiency as embedding-based methods. in parallel, the late-interaction methods improve the retrieval accuracy of the embedding-based methods through lightweight text-image interactions. to be specific, scan @xcite and visualsparta @xcite only conduct word-region interactions in the late stage when the word/region features have already been extracted by the image encoder and the text encoder. therefore, the text-image interactions in late-interaction methods are cheap. empirically, due to only using light-weight interactions, late-interaction methods normally attain faster inference but lower accuracy than their two-stage counterparts @xcite @xcite . in fact, late-interaction methods can be deployed into the first stage of a two-stage method as the alternative to the embedding-based method to further improve the performance of the two-stage methods. in this work, we propose an inflating and shrinking approach to enhance the effectiveness and efficiency of the existing late-interaction methods. we observe that the representing capability of the lateinteraction methods is limited by the text length and the region count. for instance, given a sentence of n words and an image with m regions, scan and visualsparta only have access to nm times regionword interactions between n words and m regions. to thoroughly exploit region-word interactions, we propose an inflating operation. we plug additional k codes in the input of the text/image encoder besides the word/region features. it generates m + k image vectors in the output of the image encoder and n + k text vectors in that of the text encoder. the number of region-word interactions increases from mn to (n + k)(m + k). nevertheless, incorporating additional codes inevitably brings more computational cost and makes the retrieval slower. to boost efficiency, we propose a shrinking operation based on distilling to reduce interactions. through inflating followed by shrinking, we obtain two models, the base model and the fast model. our base model obtains a considerably higher retrieval accuracy than visualsparta. meanwhile, our fast model achieves a comparable retrieval accuracy as visualsparta but takes much less latency. we visualize the efficiency and accuracy comparisons with visualsparta and cross-bert model including unicoder-vl @xcite and 12-in-1 @xcite in figure1. systematic experiments on public benchmarks, including mscoco1k and flickr30k, demonstrate the effectiveness of the proposed inflating and shrinking approach. attention-based methods. by paying attention to key visual cues in the image and key words in the text, the attention-based methods @xcite @xcite @xcite @xcite achieve considerably better performance than the embedding-based methods in cross-modal retrieval. sm-lstm @xcite takes a multi-modal context-modulated attention scheme to selectively attend to a pair of instances of image and sentence. scan @xcite computes the similarities between regions and words, and only counts the region-word pairs of high relevance. vsrn @xcite adopts graph convolution to attend the region features based on the textual context. mmca @xcite incorporates self-attention, which is originally used in transformer, to enhance the region and word features. drawn inspiration from the success achieved by bert through pre-training, several cross-bert methods @xcite @xcite @xcite are proposed for cross-modal retrieval. they stack a few transformer blocks and devise several pre-training tasks for facilitating multi-modal understanding such as masked language modeling, masked region modeling and text-image matching. after being pre-trained on large-scale datasets, they have achieved state-of-the-art performance in cross-modal retrieval. nevertheless, cross-bert methods need quadratic computational complexity, and they are slow and not scalable. trade-off methods. to alleviate the heavy computational burden while maintaining the high retrieval accuracy, several trade-off methods are proposed. they can be coarsely grouped into two-stage methods and late-interaction methods. two-stage methods @xcite @xcite take a retrieve and re-rank strategy. in the first stage, they adopt an embedding-based method to conduct coarse-level retrieval and obtain a top-t list. after that, in the second stage, they re-rank the top-t list using a cross-bert method. since the number of items for re-ranking, t, is smaller than the total number of reference items, two-stage methods achieve higher efficiency than cross-bert methods. in parallel, the lateinteraction trade-off methods @xcite apply a light-weight interaction in the late stage after feature encoding, and also achieve higher efficiency than cross-bert methods with heavy interaction in encoding. ## background embedding-based methods @xcite and cross-bert methods @xcite @xcite are two mainstream approaches for measuring the similarity between an image i and a text t . embeddingbased methods encode each image as well as each text into a global embedding. the text-image similarity is determined by the similarity between their global embeddings. consequently, given n images and m texts, the embedding-based methods only take o(n + m ) complexity in encoding. in contrast, cross-bert methods take each textimage pair as input. making use of self-attention operation, cross-bert methods achieve significantly higher retrieval accuracy than embeddingbased methods. but self-attention brings significantly more computational cost. to give an example, given n images and m texts, cross-bert methods take o(n m ) complexity to encode n m text-pairs. thus, cross-bert methods are prohibitively slow in large-scale retrieval. to trade off the efficiency and accuracy, researchers proposed late-interaction methods @xcite . similar to embeddingbased methods, they only need o(n + m ) complexity in encoding. but they exploit light-weight attention in the scoring phase based on extracted local embeddings. taking advantage of attention, they obtain higher accuracy than embedding-based methods and they are efficient since the attention is lightweight. below we introduce embedding-based methods and late-interaction methods in detail. ## experiments datasets. mscoco consists of 123, 287 images, and each image contains 5 ground-truth captions. we adopt karpathy split @xcite with 113, 287 images for training and 1, 000 images for testing. flickr30k contains 31, 783 images, and each one has 5 annotated textual descriptions. following @xcite , we use 1000 images for testing. ## conclusion in this paper, we propose an inflating and shrinking approach to boost the accuracy and efficiency of cross-modal retrieval. the inflating operation plugs multiple codes in the input of the image encoder and the text encoder. it enriches the textimage interactions and improves the retrieval accuracy. the shrinking operation gradually reduces the text-image interactions through knowledge distilling to improve the retrieval speed. systematic experiments on two widely-used public benchmarks demonstrate the effectiveness and efficiency of the proposed inflating and shrinking approach.