SDA : Semantic Discrepancy Alignment for Text-conditioned Image Retrieval (2024)

Abstract: In the realm of text-conditioned image retrieval, models utilize a query composed of a reference image and modification text to retrieve corresponding images. Despite its significance, this task is fraught with challenges, including small-scale datasets due to labeling costs and the complexity of attributes in modification texts. These challenges often result in models learning a generalized representation of the query, thereby missing the semantic correlations of image and text attributes.In this paper, we introduce a general boosting framework designed to address these issues by employing semantic discrepancy alignment. Our framework first leverages the ChatGPT to augment text data by modifying the original modification text’s attributes. The augmented text is then combined with the original reference image to create an augmented composed query. Then we generate corresponding images using GPT-4 for the augmented composed query.We realize the cross-modal semantic discrepancy alignment by formulating distance consistency and neighbor consistency between the image and text domains. Through this novel approach, attribute in the text domain can be more effectively transferred to the image domain, enhancing retrieval performance. Extensive experiments on three prominent datasets validate the effectiveness of our approach, with state-of-the-art results on a majority of evaluation metrics compared to various baseline methods.

## methodology the text-conditioned image retrieval aims to return the relevant target images with an image and a modification text sentence as a composed query. let (i q , m, i t ) represents the reference image, the modification text and a candidate target image, respectively. our goal is learning a joint representation f (i q , m ) which is similar with the representation f target (i t ). in the following, we start by discussing composed query augmentation in sec. 3.1. then we introduce our framework in sec. 3.2 and the details of the semantic discrepancy alignment loss in sec. 3.3. finally, we elaborate the training and inference procedures in sec. 3.4. ## experiments to verify the effectiveness of our framework, we conduct experiments on three benchmarks including fashioniq @xcite , shoes @xcite and fashion200k @xcite . in this section, we will introduce the implementation details, the experimental results and ablation studies in sec. 4.1, sec. 4.2 and sec. 4.3, respectively. ## conclusions we propose a general boosting framework for the text-conditioned image retrieval task by exploring semantic discrepancy alignment. by leveraging the strong capability of chatgpt and gpt-4, we generate suitable modification texts to construct augmented composed queries with corresponding images. in our framework, we capture the semantic discrepancy alignment by introducing two novel losses: neighbour consistency and distance consistency. we leverage the well-defined feature distribution in text domain to improve the ability of the compositor and further ensure the first-order and higher-order alignment between composite domain and text domain. through extensive experiments, we demonstrate that our proposed method achieves a new state-of-the-art performance on most datasets.