Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2022)

Abstract: Current state-of-the-art approaches to cross- modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image. While offering unmatched retrieval performance, such models: 1) are typically pretrained from scratch and thus less scalable, 2) suffer from huge retrieval latency and inefficiency issues, which makes them impractical in realistic applications. To address these crucial gaps towards both improved and efficient cross- modal retrieval, we propose a novel fine-tuning framework that turns any pretrained text-image multi-modal model into an efficient retrieval model. The framework is based on a cooperative retrieve-and-rerank approach that combines: 1) twin networks (i.e., a bi-encoder) to separately encode all items of a corpus, enabling efficient initial retrieval, and 2) a cross-encoder component for a more nuanced (i.e., smarter) ranking of the retrieved small set of items. We also propose to jointly fine- tune the two components with shared weights, yielding a more parameter-efficient model. Our experiments on a series of standard cross-modal retrieval benchmarks in monolingual, multilingual, and zero-shot setups, demonstrate improved accuracy and huge efficiency benefits over the state-of-the-art cross- encoders.1

## introduction information-rich and efficient methods for dealing with large unstructured data in both computer vision and nlp are required to process and understand huge amounts of user-created content and beyond. in multi-modal contexts, such methods enable fundamental applications such as image retrieval. a typical efficient bi-encoder 2 approach encodes images and text separately and then induces a shared high-dimensional multi-modal feature space. this enables cross-modal retrieval, where standard distance metrics identify the most similar examples for each query in the data collection via nearest-neighbor search @xcite @xcite @xcite . these bi-encoder approaches have already been shown to achieve reasonable performance in search and retrieval applications, both monolingually for english @xcite @xcite @xcite and in multilingual contexts @xcite @xcite @xcite . however, they cannot match performance of more recent attention-based methods. here, a typical modus operandi is to apply a cross-attention mechanism between examples from the two modalities to compute their similarity score, relying on transformer-based neural architectures @xcite . such so-called multi-modal cross-encoders (ce) @xcite @xcite @xcite @xcite pass each text-image pair through the multi-modal encoder to compute their similarity, see figure 1a . while the results accomplished by the ce methods look impressive @xcite @xcite , this comes at a prohibitive cost. in particular, they have extremely high search latency: processing a single text query with an image collection of 1m items may take up to 36 minutes using a single nvidia v100 gpu (see in unimodal text-only setups, transformerbased architectures have recently been integrated with bi-encoder (be) methods @xcite @xcite @xcite , yielding computationally more efficient sentence encoders. instead of jointly encoding sentence pairs with cross-attention, a pretrained transformer model (e.g., bert @xcite ) is fine-tuned within a twin network with shared transformer weights, as illustrated in figure 1b . in a nutshell, each sentence is passed through the encoder separately, and a loss function is defined on top of the two respective separately computed encodings. however, despite their strong performance on sentence retrieval and similarity tasks @xcite , these encoders cannot match the task performance of cross-encoders @xcite . motivated by these insights, in this work we aim to leverage the best of both worlds towards improved and more efficient cross-modal search and retrieval: 1) efficiency and simplicity of be approaches based on twin networks, as well as 2) expressiveness and cutting-edge performance of ce methods. we first provide a systematic comparative analysis on the effectiveness and efficiency of transformer-based multi-modal be and ce methods across a range of image search evaluation benchmarks. we then propose two novel models that aim to blend the main strengths of ce and be. the idea behind the first model variant, termed cooperative (sep+coop), is to retrieve and rerank with two separate, independently trained retrieval models: 1) an initial top-k list of potentially relevant items (i.e., texts or images) is retrieved by the more efficient be model, and then 2) this top-k list is reranked ''smartly'' by the more accurate ce model, as illustrated in figure 1c . our second, joint (joint+coop) model variant also operates in the same retrieve-and-rerank setup, but it now trains a multi-modal cross-encoder and a multi-modal be model jointly with tied weights, as illustrated in figure 1d . the retrieve step, where efficiency is paramount, is again executed by the be sub-model, and the precision-oriented rerank step is conducted via the ce sub-model. we propose a general framework for crossmodal search and retrieval, where joint+coop and sep+coop models are independent of the chosen pretrained vision-language representation architectures. the experiments are thus based on a stateof-the-art vision-language architecture oscar @xcite (experiments in english) and m3p @xcite (multilingual), and we demonstrate consistent improvements over the original oscar model on the standard benchmarks mscoco and flick30k and improvements over the original m3p in multiple languages on the multi30k dataset. we empirically validate huge efficiency benefits of the proposed framework. contributions. 1) we construct and systematically evaluate twin-networks combined with multi-modal transformers (be); they outperform all previous bi-encoder approaches, but lag behind their ce counterparts. 2) we evaluate be and ce approaches within a cooperative retrieve-andrerank approach; their combination outperforms the individual models, while offering substantial efficiency boosts compared to ce methods. 3) we propose a novel joint ce-be model (joint+ coop), which is trained to simultaneously crossencode and embed multi-modal input; it achieves the highest scores overall while maintaining retrieval efficiency. 4) finally, we propose a more realistic evaluation benchmark; we demonstrate harsh drops in overall cross-modal retrieval performance of all models in this more difficult scenario, calling for improved evaluation benchmarks and protocols in future work. ## related work efficient approaches to cross-modal image-text retrieval relied on the induction of shared multimodal visual-semantic embedding spaces (vses) @xcite @xcite . in a multilingual setup, all languages share the same embedding space along with the visual data @xcite @xcite . more recently, attention-based cross-encoder models, typically based on transformer architectures @xcite have considerably outperformed the vse-based approaches. however, this comes at a severe cost of decreased retrieval efficiency and increased latency @xcite . the current state-of-the-art multi-modal models jointly encode and crossattend over text tokens and image features @xcite @xcite @xcite @xcite . these ce methods leverage image captioning datasets such as mscoco @xcite and flick30k @xcite and train a classification head that learns to identify whether or not an (image, caption) input pair constitutes an aligned pair. each image-text combination must be passed through the network, which scales quadratically with the number of examples. to handle this quadratic increase, we use a cooperative retrieve-and-rerank approach. although to the best of our knowledge this has not been proposed for cross-modal settings, it has a long history in nlp, where @xcite date it back to the 1960s @xcite . until recently, bag-of-words methods (bow; e.g., bm25) were commonly used for the first retrieval step. for the second step, pretrained language models (lms) were fine-tuned to either rerank candidates @xcite or-for question-answering tasks-directly generated the answer span @xcite . more recent work on text-based retrieval and qa tasks has moved away from bow methods towards learned (neural) models for the first retrieval step @xcite @xcite . our work is inspired by recent be-based approaches in unimodal text-only setups. here, lms are fine-tuned via twin-network architectures on auxiliary tasks such as semantic textual similarity @xcite , paraphrasing @xcite , response retrieval @xcite @xcite , or translation ranking @xcite . this effectively turns the lms into universal sentence encoders which can then be used off-the-shelf for efficient text-based monolingual and cross-lingual retrieval @xcite . in this work, we first extend this idea to multi-modal setups, and then show that our cooperative and joint approaches yields improved cross-modal retrieval models, maintaining retrieval efficiency. joint approaches like our joint+coop model, which aim to align the retriever and reranker can be found in different forms: @xcite ''mark'' exact matches from the bag-ofwords retrieval for the reranker; @xcite share the parameters between a passage expander (which adds more relevant terms for a bag-ofwords retriever) and the reranker; hofstätter et al. (2020) distill knowledge from the reranker into the retriever model with soft labels generated by the teacher. specifically for question-answeringwhere a two stage retriever-reader setup similar to the retrieve-and-rerank approach is commonresearch aims to synchronize the models through knowledge distillation from the reader to the retriever @xcite or by directly training both models end-toend @xcite . the challenge here is that the reader and the retriever are coupled-the reader requires candidates from the retriever that contain the solution. our proposed reranker side-steps this problem as it uses no candidates from the retriever during training and only learns if a given input pair is (dis)similar. this way, we can train both components, the retriever and the reranker, side-by-side and align them by sharing their weights. the work most closely related to ours includes contemporaneous models: albef @xcite , clip @xcite , align @xcite , and visualsparta @xcite . albef includes contrastive learning as one of its pretraining tasks but then uses a ce approach for downstream retrieval. clip and align use similar contrastive learning strategies as we do, but are cast as full-fledged pretraining architectures that learn from scratch and require magnitudes of more data than our approach. we show that it is possible to fine-tune pretrained models with fewer data and offer a general framework, applicable to a spectrum of pretrained models. further, unlike prior work, we demonstrate the benefits of combining be-based (contrastive) learning with cross-encoders for improved and efficient retrieval. 4 finally, visualsparta @xcite fine-tunes oscar, but at the level of token (text) and image-region embeddings. this enables the use of extremely fast lookup tables for efficient retrieval. however, this comes with a major disadvantage: the model disposes of wider context information. 5 our cooperative methods do leverage the finer-grained information at retrieval. ## methodology the predominant transformer-based multi-modal text-vision architecture is a single-stream encoder: it shares the majority of weights between the two modalities, including the multi-head crossattention @xcite @xcite @xcite . the transformer weights and text embeddings are typically initialized with weights of a pretrained lm (e.g., bert @xcite for english, xlm-r @xcite for multilingual models), where the corresponding vocabulary and tokenizer are utilized. images are preprocessed via object detection models such as faster r-cnn @xcite to extract feature representations for regions of interest @xcite . the image features are passed through an affine-transformation layer which learns to align the vision input with the pretrained transformer. the position of the region of interest (or in some models also the region's width and height) is used to generate positional embeddings. by combining these two representations, each object region is passed into the transformer separately. the cross-attention mechanism of the transformer attends over all text and image inputs at every layer, thus learning a joint representation of both modalities. similar to masked language modeling (mlm) in the text domain, multi-modal transformer models are trained with self-supervised objectives. for pretraining, image-caption datasets (i.e., mscoco @xcite , flickr30k @xcite , conceptual captions (cc) @xcite , and sbu @xcite ) are utilized. the pretrained multi-modal model is subsequently fine-tuned with multi-modal downstream task data. we focus on different fine-tuning strategies of the pretrained models for the downstream task of image-text retrieval. we illustrate these approaches in figure 1 and describe them in what follows. ## experimental setup our fine-tuning framework from §3 can be applied to any pretrained multi-modal transformer. in all the experiments, we opt for state-of-the-art pretrained multi-modal models for monolingual (english) and multilingual contexts: oscar @xcite and m3p @xcite , respectively. oscar is a single-stream multi-modal transformer @xcite , with its weights initialized with those of the pretrained bert base model, and then subsequently fine-tuned on multimodal data (see §3). unlike prior work, oscar additionally uses object labels of detected regions: those labels serve as anchors for visual grounding, with large improvements achieved over its prior work. m3p is a single-stream multilingual multi-modal transformer. its weights are initialized with those of pretrained xlm-r base and then fine-tuned on multi-modal data (see §3) as well as multilingual text-only data. training and test data. we primarily experiment with the english image-text retrieval benchmarks mscoco and flick30k. they comprise 123k and 31.8k images, respectively, with 5 captions describing each image. mscoco provides two test benchmarks of sizes 1k and 5k, where the smaller set is a subset of the 5k test set. the standard flickr30k test set consists of 1k images. in addition, we use the development set of conceptual captions (cc) @xcite for zero-shot evaluation, and also to construct a larger and more difficult test set (see later in §6). the original cc dev set contained 15.8k images, but currently, only 14k images are still available online. for multilingual experiments, we use the standard multi30k dataset @xcite @xcite , which extends flickr30k with 5 german and one french and czech caption per image. its test s et also comprises 1k images. the evaluation metric is the standard recallat-m (r@m): it reports the proportion of queries for which the relevant target item is present within the top-m retrieved items. training setup and hyperparameters. our setup largely follows @xcite and @xcite unless noted otherwise. 10 we experiment with learning rates [5e -5, 2e -5], and with the number of update steps between 25k and 125k. one batch contains 128 positive pairs plus 128 negative pairs with l ce . we use the adamw optimizer @xcite with a linear learning rate decay without warmup, and a weight decay of 0.05. we take model checkpoints every 5k steps and select the checkpoint with the best development set performance. ## results and discussion the main results on english-only monolingual datasets flickr30k and mscoco are summarized in as expected, all transformer-based approaches (groups g2 and g3) substantially outperform the pre-transformer (pt) models (g1). while this has already been established in prior work for ce methods, our findings confirm that the same holds also for the efficient be approach. this validates the effectiveness of transformer architectures pretrained on large corpora for the retrieval task. r@1 scores with be lag slightly behind the ce scores, but the respective r@10 scores are mostly on-par. this suggests that the be approach is ''coarser-grained'', and mostly relies on ''global'' interactions between the modalities. we investigate this conjecture further in §6. this is also illustrated by an example in figure 2 . when dealing with related target items, ce's cross-attention mechanism is able to explicitly attend over each token and image region, capturing additional (non-global) information relevant to the query. although the high-level ''global'' concept of a skiing person is present in (almost) every example, the additional important information related to what the person is wearing is not adequately represented in the embeddings. therefore, the be (sub)model does not rank this instance at the top position. the ce (sub)model then directly compares the instances, identifying that clothing is important and reranks the target examples accordingly. most importantly, the relative comparison of r@1 versus r@10 scores empirically hints at the necessity of the retrieve-and-rerank cooperative approach: the be approach efficiently retrieves 20 relevant examples, but the increased expressiveness of ce is required to refine the initially retrieved list. moreover, the results in the cooperative setup even without joint training (sep+coop oscar and sep+coop m3p ) demonstrate that the two models support each other: slight improvements are observed over the pure ce, while offering massive efficiency boosts over ce. our speculation is that the be model filters out false positives, which in turn makes the ce model more robust. the results of the joint+coop variant indicate that it is indeed possible to maintain retrieval efficiency with improved parameter efficiency: this approach performs on-par or even slightly outperforms the standard state-of-the-art ce models. the results verify that the two objective functions do not interfere with each other and that a single model is able to both embed and crossencode. we note that the joint+coop variant offers the best trade-off between parameter and retrieval efficiency, achieving the peak scores on ## further analysis we now discuss a series of additional experiments that further profile and analyze the proposed multi-modal retrieval approaches, focusing especially on the multiple efficiency aspects related to fine-tuning and retrieval stages. retrieval efficiency. we empirically validate the time efficiency of our cooperative approaches for retrieval in an image search scenario ( our measurements confirm the efficiency of bes in comparison to ces. the cooperative approaches, which only have to cross-encode a constant number of items invariant of the collection size, are close in retrieval latency to be for image search and remain feasible even for large datasets. larger benchmarks. the results in ## conclusion we proposed a novel framework that converts pretrained multi-modal transformers into effective and efficient cross-modal retrieval models. the framework is applicable to any pretrained model and combines the efficiency of bi-encoder (be) approaches with the accuracy of computationally more demanding cross-encoding (ce) approaches. their synergistic effect at retrieval is achieved through a cooperative retrieve-andrerank regime, where the initial retrieval from a large collection is performed via efficient be approaches, followed by another accuracy-driven step via a ce model. moreover, we introduced a parameter-efficient joint fine-tuning regime that blends be and ce into a single model with shared weights. our results with state-of-the-art pretrained models across a range of standard monolingual and multilingual cross-modal retrieval tasks and setups validated the strong performance of such cooperative and joint approaches. at the same time, we demonstrated their retrieval efficiency, which makes them viable in realistic retrieval scenarios with large collections. in future work, we will put more focus on zero-shot and few-shot retrieval scenarios, and expand the approach to more languages, modalities, and tasks.