V isual S parta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words (2021)

Abstract: Text-to-image retrieval is an essential task in cross-modal information retrieval, i.e., retrieving relevant images from a large and unlabelled dataset given textual queries. In this paper, we propose VisualSparta, a novel (Visual-text Sparse Transformer Matching) model that shows significant improvement in terms of both accuracy and efficiency. VisualSparta is capable of outperforming previous state-of-the-art scalable methods in MSCOCO and Flickr30K. We also show that it achieves substantial retrieving speed advantages, i.e., for a 1 million image index, VisualSparta using CPU gets ~391X speedup compared to CPU vector search and ~5.4X speedup compared to vector search with GPU acceleration. Experiments show that this speed advantage even gets bigger for larger datasets because VisualSparta can be efficiently implemented as an inverted index. To the best of our knowledge, VisualSparta is the first transformer-based text-to-image retrieval model that can achieve real-time searching for large-scale datasets, with significant accuracy improvement compared to previous state-of-the-art methods.

## introduction text-to-image retrieval is the task of retrieving a list of relevant images from a corpus given text queries. this task is challenging because in order to find the most relevant images given text query, the model needs to not only have good representations for both textual and visual modalities, but also capture the fine-grained interaction between them. existing text-to-image retrieval models can be broadly divided into two categories: query-agnostic and query-dependent models. the dual-encoder architecture is a common query-agnostic model, which uses two encoders to encode the query * this work was partially done during an internship at soco figure 1 : inference time vs. model accuracy. each dot represents recall@1 for different models on mscoco 1k split. by setting top n-terms to 500, our model significantly outperforms the previous best query-agnostic retrieval models, with ∼2.8x speedup. see section 5.1 for details. and images separately and then compute the relevancy via inner product @xcite @xcite . the transformer architecture is a well-known querydependent model @xcite . in this case, each pair of text and image is encoded by concatenating and passing into one single network, instead of being encoded by two separate encoders @xcite . this method borrows the knowledge from large pretrained transformer models and shows much better accuracy compared to dual-encoder methods @xcite . besides improving the accuracy, retrieval speed has also been a long-existing subject of study in the information retrieval (ir) community @xcite . query-dependent models are prohibitively slow to apply to the entire image corpus because it needs to recompute for every dif-ferent query. on the other hand, query-agnostic model is able to scale by pre-computing an image data index. for dual-encoder systems, further speed improvement can be obtained via approximate nearest neighbors (ann) search and gpu acceleration @xcite . in this work, we propose visualsparta, a simple yet effective text-to-image retrieval model that outperforms all existing query-agnostic retrieval models in both accuracy and speed. by modeling fine-grained interaction between visual regions with query text tokens, our model is able to harness the power of large pre-trained visual-text models and scale to very large datasets with real-time response. to our best knowledge, this is the first model that integrates the power of transformer models with real-time searching, showing that large pre-trained models can be used in a way with significantly less amount of memory and computing time. lastly, our method is embarrassingly simple because its image representation is essentially a weighted bag-of-words, and can be indexed in a standard inverted index for fast retrieval. comparing to other sophisticated models with distributed vector representations, our method does not depend on ann or gpu acceleration to scale up to very large datasets. contributions of this paper can be concluded as the following: (1) a novel retrieval model that achieves new state-of-the-art results on two benchmark datasets, i.e., mscoco and flickr 30k. (2) weighted bag-of-words is shown to be an effective representation for cross-modal retrieval that can be efficiently indexed in an inverted index for fast retrieval. (3) detailed analysis and ablation study that show advantages of the proposed method and interesting properties that shine light for future research directions. ## related work large amounts of work have been done on learning a joint representation between texts and images @xcite @xcite @xcite . in this section, we revisit dual-encoder based retrieval model and transformer-based retrieval model. ## visualsparta retriever in this section, we present visualsparta retriever, a fragment-level transformer-based model for efficient text-image matching. the focus of our proposed model is two-fold: ## conclusion in conclusion, this paper presents visualsparta, an accurate and efficient text-to-image retrieval model that shows the state-of-the-art scalable performance in both mscoco and flickr30k. its main novelty lies in the combination of powerful pre-trained image encoder with fragment-level scoring. detailed analysis also demonstrates that our approach has substantial scalability advantages compared to previous best methods when indexing large image datasets for real-time searching, making it suitable for real-world deployment.