CF - TCIR : A Compositor-Free Framework for Hierarchical Text-Conditioned Image Retrieval (2024)

Abstract: In text-conditioned image retrieval (TCIR), the combination of a reference image and modification text forms a query tuple, aiming to locate the most congruent target image within a dataset. The advantages of rich image semantic information and text flexibility are combined in this manner for more accurate retrieval. While traditional techniques often employ attention-driven compositors to craft a unified image-text representation, our paper introduces a compositor-free framework, CF-TCIR, which eschews the standard compositor. Compositor-based methods are designed to learn a joint representation of images and text, but they struggle to directly capture the correlations between attributes across the image and text modalities. Instead, we reformulate the retrieval process as a cross-modal interaction between a synthesized image feature and its corresponding text descriptor. This novel methodology offers advantages in terms of computational efficiency, scalability, and superior performance. To optimize the retrieval performance, we advocate a tiered retrieval mechanism, blending both coarse-grain and fine-grain paradigms. Moreover, to enrich the contextual relationship within the query tuple, we integrate a generative cross-modal alignment technique, ensuring synchronization of sequential attributes between image and text data.

## experiments to verify the effectiveness of our method, we conduct experiments on three benchmarks on fashion domain including fashioniq @xcite , shoes @xcite and fashion200k @xcite . in this section, we introduce the implementation details, the experimental results on different datasets and ablation studies in sec. 4.1, sec. 4.2 and sec. 4.3, respectively. ## conclusions we propose a compositor-free framework for hierarchical text-conditioned image retrieval (cf-tcir). in contrast to previous works that design an attention-based compositor to learn the joint image-text representation, we discard the compositor module and reformulate a more natural and efficient retrieval pipeline. in our framework, we utilize the semantic correlation of the query image and the target image and then perform crossmodal interaction with the modification text feature. through extensive experiments, it is demonstrated that our method achieves competitive performance, which can greatly reduce the model parameters and also increase retrieval efficiency.