Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval (2022)

Abstract: Image-text retrieval is a fundamental cross-modal task that takes image/text as a query to retrieve relevant data of another type. The large-scale two-stream pre-trained models like CLIP have achieved tremendous success in this area. They embed the images and texts into instance representations with two separate encoders, aligning them on the instance-level with contrastive learning. Beyond this, the following works adopt the fine-grained token-level interaction (Masked Language and Image Modeling) to boost performance further. However, the vanilla token-level objectives are not designed to aggregate the image-text alignment information into the instance representations, but the token representations, causing a gap between pre-training and application. To address this issue, we carefully design two novel conditioned token-level pre-training objectives, Conditioned Masked Language and Image Modeling (ConMLM and ConMIM), forcing models to aggregate the token-level alignment information into the instance representations. Combing with the instance-level contrastive learning, we propose our cross-modal dense retrieval framework, Conditioned Language-Image Pre-training (ConLIP). Experimental results on two popular cross-modal retrieval benchmarks (MSCOCO and Flickr30k) reveal the effectiveness of our methods.

## introduction image-text retrieval is an important task in the cross-modal community. recent years have witnessed the remarkable success of the large-scale language-image pre-trained models in this area. the existing works can be divided into singlestream and two-stream models. the former one as illustrated in figure 1a relies on the heavy transformer layers @xcite to fuse the cross-modal information (e.g. uniter (chen * * corresponding author @xcite , and oscar @xcite ). the intolerable drawback of these models is the slow inference speed. all possible query-candidate pairs need to be fed into the models to get the retrieval result of a query. for example, the average inference time of uniter 1 for a text query on mscoco @xcite is more than 30 seconds. therefore, these models are hard to be applied in real-life industrial applications. to overcome this limitation, recent works turn to the two-stream models as shown in figure 1b (e.g., clip @xcite ). these models embed images and texts into instance representations ( @xcite ) with two separate encoders, aligning them on the instance-level with contrastive learning @xcite ) and calculating the retrieval scores with a simple dot-product. decoupling the correlation of images and texts, the inference speed of these models is much faster. apart from the instance-level alignment, the fine-grained token-level tasks (masked language modeling, mlm @xcite and masked image modeling, mim @xcite ) are adopted to boost the performance further @xcite . nevertheless, the vanilla mlm and mim as illustrated in figure 1c are sub-optimal, which are designed to aggregate the token-level alignment information into the token representations, not instance representations. for example, the [cls] representation of the well-known masked language pre-trained model roberta @xcite performs poorly on the semantic textual similarity tasks without finetuning @xcite . therefore, it is necessary for us to come up with more suitable token-level pre-training tasks for image-text dense retrieval. in this work, we carefully design two novel conditioned token-level objectives, conditioned masked language and image modeling cross-modal transformer masked language modeling. ## conclusion in this work, we design two novel token-level pretraining tasks, conmlm and conmim for imagetext dense retrieval. combining with the instancelevel objective, we propose our language-image pre-training framework conlip. the experimental results and the cross-attention analysis reveal the effectiveness of our methods.