C ontext BLIP : Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions (2024)

Abstract: Image retrieval from contextual descriptions (IRCD) aims to identify an image within a set of minimally contrastive candidates based on linguistically complex text. Despite the success of VLMs, they still significantly lag behind human performance in IRCD. The main challenges lie in aligning key contextual cues in two modalities, where these subtle cues are concealed in tiny areas of multiple contrastive images and within the complex linguistics of textual descriptions. This motivates us to propose ContextBLIP, a simple yet effective method that relies on a doubly contextual alignment scheme for challenging IRCD. Specifically, 1) our model comprises a multi-scale adapter, a matching loss, and a text-guided masking loss. The adapter learns to capture fine-grained visual cues. The two losses enable iterative supervision for the adapter, gradually highlighting the focal patches of a single image to the key textual cues. We term such a way as intra-contextual alignment. 2) Then, ContextBLIP further employs an inter-context encoder to learn dependencies among candidates, facilitating alignment between the text to multiple images. We term this step as inter-contextual alignment. Consequently, the nuanced cues concealed in each modality can be effectively aligned. Experiments on two benchmarks show the superiority of our method. We observe that ContextBLIP can yield comparable results with GPT-4V, despite involving about 7,500 times fewer parameters.

## introduction text-to-image retrieval is a fundamental crossmodal task that aims to search images for textual queries. early studies relied on convolutional neural networks (cnn) @xcite and long short-term memory (lstm) @xcite to learn to extract image and text features and then proceeded to align the representations of two modalities for retrieval. recent proliferated large vision-language models (vlms), such as vision transformers (vits) @xcite , uniter @xcite and clip @xcite , which are trained on large-scale short text-image corpus, have made remarkable progress for retrieving images from sentences with few objects and simple linguistic. however, in the real world, natural languages are highly contextual @xcite with long utterances. context, including perceptual and temporal cues, plays a pivotal role in grounding the implication of a linguistically complex text @xcite . figure 1 demonstrates such a case that identifies an image from six very similar candidates with a grammatically complicated description. two major challenges of the retrieval are: 1) a model needs to understand nuanced textual cues, such as "hand is blurry", "eyes are almost shut", and "looking at..." across three grammatically complex sentences and align them with various context cues in each image. 2) long-range dependencies among candidate images need to be captured to perform cross-modal reasoning for further alignment. the above interesting and challenging task is known as image retrieval from contextual descriptions (ircd) @xcite . despite the great success of vlms, they can hardly tackle the above two challenges, and significantly lag behind human performance. existing vlms applied to text-image retrieval mainly include contrastive-based ones such as clip @xcite and align @xcite , randomly mask-based ones such as m3ae @xcite and maskvlm @xcite , and attention-based ones such as albef @xcite and blip @xcite . these models have some limitations. 1) the former two focus more on high-level semantic alignment, while the fine-grained contextual cues may be largely ignored. 2) existing mask-based ones randomly remove image patches, without specifically learning to concentrate on the key objects associated with text tokens, e.g., "hand" and "eyes" in figure 1 . 3) dependencies among images are not specifically considered. previous state-of-the-art ndcr @xcite ) proposed for ircd divided the complex alignment into multiple simple ones and then combined them for final retrieval. however, the performance is highly dependent on the candidate's distributions and is poor for fine-grained alignment on static images with a large variance. we observe that ndcr can hardly capture the key contextual cues in grammatically complex long sentences. further, it also lacks zero-shot capability. consequently, ndcr still suffers from the two challenges of ircd. details are discussed in to this end, we introduce contextblip, a novel doubly contextual alignment scheme for the challenging ircd task based on blip @xcite . 1) specifically, our contextblip comprises a multiscale adapter, a matching loss, and a text-guided masking loss. the learnable adapter, which is inserted into frozen blip, aims to capture higherlevel and lower-level visual features of the candidate images. the two losses enable iterative supervision during the training stage, gradually allowing the adapter to highlight the focal patches of a single image to the linguistically complex textual cues. such a way is termed as intra-contextual alignment that aims to tackle the first challenge issue of the ircd task. 2) then, we further fine-tune contextblip with a temporal transformer to learn dependencies among candidate images, facilitating alignment between text to multiple images. this step is termed as inter-contextual alignment that aims to address the second challenge issue of ircd. experiments on a public benchmark show the effectiveness of the proposed contextblip. the main contributions of this paper are listed as follows. we also evaluate our contextblip on a very recent benchmark mmvp-vlm and the results further confirm the effectiveness of the proposed method. masked image modeling (mim): mim, such as vibert @xcite and mae @xcite , refers to predicting the missing pixels in an image by using the surrounding pixels as context. mim has been applied for various tasks, such as robust learning @xcite and generation target @xcite . recent studies, such as maskvlm @xcite , m3ae @xcite , vl-beit @xcite extend mim for both image and language masking. the key difference between ours and existing ones is: we employ a text-guided masking scheme that learns to generate masks under iterative supervision with two losses, thus gradually highlighting the focal patches of a single image to the key textual cues. while the previous ones randomly remove image patches, without specifically learning to focus the key visual objects associated with text tokens. ## conclusion this paper presents contextblip, a simple yet effective doubly contextual alignment scheme for the challenging ircd. our model comprises a multiscale adapter, a matching loss, and a text-guided masking loss. the adapter learns to capture finegrained visual cues. the two losses enable iterative supervision for the adapter, gradually highlighting the focal patches of a single image to the key textual cues. then, contextblip further employs an inter-context encoder to learn dependencies among candidates, facilitating accurate alignment between text to multiple images. consequently, the nuanced cues concealed in textual and visual modalities can be effectively aligned. experiments on two benchmarks show the effectiveness of our method. we observe that our contextblip can yield comparable results with gpt-4v, despite involving about 7,500 times fewer parameters. in the future, we plan to extend our method to text-to-video retrieval. the backbone and the prediction head to enable contextual comparison and reasoning. during the first stage, we conduct full fine-tuning for 20 epochs. after that, we add a two-layer transformer as a context/temporal module, which is identical to krojer et al.. to optimize the module while keeping other parameters fixed, we freeze all parts of the model except the inserted module and train it alone for 5 epochs. we set the batch size to 360 images (36 batches of 10-image-set) and use the adam (kingma and ba, 2014) optimizer with learning rate of 2e-6 and weight decay of 0.01. we conduct the experiment on one nvidia geforce 3090 for about one day. all other baselines also adopted the same settings. hyperparameters max epoch 25 batch size 36 input resolution 224 2 augmentation randaug optimizer adam backbone learning rate 2e-6 head learning rate 1e-4 momentum (0.9, 0.999) learning rate scheduler exponentiallr(0.95) weight decay 1e-2 random seed 10