Exploring Compositional Image Retrieval with Hybrid Compositional Learning and Heuristic Negative Mining (2022)

Abstract: Compositional image retrieval (CIR) is a challenging retrieval task, where the query is composed of a reference image and a modification text, and the target is another image reflecting the modification to the reference image. Due to the great success of the pre-trained vision-and-language model CLIP and its favorable applicability to large-scale retrieval tasks, we propose a CIR model HyCoLe-HNM with CLIP as the backbone. In HyCoLe-HNM, we follow the contrastive pre-training method of CLIP to perform cross-modal representation learning. On this basis, we propose a hybrid compositional learning mechanism, which includes both image compositional learning and text compositional learning. In hybrid compositional learning, we borrow a gated fusion mechanism from a question answering model to perform compositional fusion, and propose a heuristic negative mining method to filter negative samples. Privileged information in the form of image-related texts is utilized in cross-modal representation learning and hybrid compositional learning. Experimental results show that HyCoLe-HNM achieves state-of-the-art performance on three CIR datasets, namely FashionIQ, Fashion200K, and MIT-States.

## introduction in this paper, we explore the task of compositional image retrieval (cir). as shown in figure 1 , cir is aimed at retrieving a target image slightly different from a reference image, where the difference is described by a modification text. the key to cir is to learn the cross-modal composition process from (reference image, modification text) pairs to target images. in the existing cir models, this is usually realized by fusing and matching image representations obtained from pre-trained vision models with text representations obtained from pre-trained language models @xcite @xcite @xcite @xcite . however, these pre-trained models are pre-trained on uni-modal data, which implies that visual concepts embodied in image representations are not aligned with semantic concepts embodied in text representations. as a result, applying these models to cir yields limited benefit, which necessitates the application of pre-trained vision-and-language (v&l) models. clip @xcite , a recently-proposed v&l model pre-trained on 400m image-text pairs, has exhibited strong zero-shot performance on image classification. empirical studies @xcite showed that it is suboptimal to apply clip in a zero-shot manner to complex v&l tasks requiring cross-modal reasoning, such as visual question answering (vqa), visual entailment, and v&l navigation. meanwhile, @xcite also showed that state-of-theart (sota) performance can be achieved on these tasks by integrating and fine-tuning clip. due to the requirement for cross-modal compositionality, we believe that cir is as complex as vqa. therefore, we use clip as the backbone of our proposed cir model, and fine-tune it together with the rest model components. there are indeed some other pre-trained v&l models than clip, such as al-bef @xcite and blip @xcite . these models use a single encoder to encode imagetext combinations through cross-modal attention mechanisms, which is computationally expensive during retrieval if there are many candidate images. however, clip uses two encoders to separately encode images and texts so that we can encode all candidate images in advance and calculate matching scores as simple dot products, which is applicable to large-scale retrieval tasks. based on clip, we propose a novel cir model named hycole-hnm, which features hybrid compositional learning and heuristic negative mining. on the one hand, unlike the existing cir models, which mostly focus on image compositional learning, we propose a hybrid compositional learning mechanism, which includes both image compositional learning and text compositional learning. specifically, we not only learn the compositional matching between reference images and target images, but also utilize image-related texts as privileged information to learn the compositional matching between reference texts and target texts. on the other hand, to facilitate the contrastive optimization of hybrid compositional learning, we also propose a heuristic negative mining method to filter negative samples so that only the negative samples relevant to the positive ones are retained. specifically, we enforce a heuristic rule to identify relevant negative samples, and thereby reduce the space complexity of negative samples from o(n 3 ) to o(n 2 ). compared with hard negative mining methods, the heuristic negative mining method is not only more efficient, but also achieves better performance in the ablation experiments. for optimal performance, when implementing hycole-hnm, we innovatively integrate some approaches originally aimed at other tasks. specifically, following the contrastive pre-training method of clip, we utilize the above mentioned privileged information to perform cross-modal representation learning. besides, we also borrow a gated fusion mechanism from a question answering (qa) model to perform compositional fusion. experimental re-sults show that by applying these approaches to hycole-hnm, we achieve sota performance on multiple cir datasets. ## model in this section, we propose our cir model hycole-hnm. first, we provide a task definition of cir. then, we present a cross-modal representation learning method. next, we propose a hybrid compositional learning mechanism and a heuristic negative mining method, from which hycole-hnm takes its name. finally, we describe the training and inference of hycole-hnm. ## limitations besides conducting experiments on fashioniq, fashion200k, and mit-states, which are all comprised of natural images, we also conduct experiments on another cir dataset css @xcite , which is comprised of synthetic images. however, on css, the performance of hycole-hnm is inferior to that of tirg (r@1: 67.3% vs 73.7%). since the backbone of hycole-hnm is clip, which is pre-trained on natural image-text pairs, we conjecture that the reason behind this under-performance is the domain shift between the natural images used to pre-train clip and the synthetic images in css used to train hycole-hnm. a appendix