DEMO : A Statistical Perspective for Efficient Image-Text Matching (2024)

Abstract: Image-text matching has been a long-standing problem, which seeks to connect vision and language through semantic understanding. Due to the capability to manage large-scale raw data, unsupervised hashing-based approaches have gained prominence recently. They typically construct a semantic similarity structure using the natural distance, which subsequently guides the optimization of the hashing network. However, the similarity structure could be biased at the boundaries of semantic distributions, causing error accumulation during sequential optimization. To tackle this, we introduce a novel hashing approach termed Distribution-based Structure Mining with Consistency Learning (DEMO) for efficient image-text matching. From a statistical view, DEMO characterizes each image using multiple augmented views, which are considered as samples drawn from its intrinsic semantic distribution. Then, we employ a non-parametric distribution divergence to ensure a robust and precise similarity structure. In addition, we introduce collaborative consistency learning which not only preserves the similarity structure in the Hamming space but also encourages consistency between retrieval distribution from different directions in a self-supervised manner. Extensive experiments on several widely used datasets demonstrate that DEMO achieves superior performance compared with various state-of-the-art methods.

## introduction image-text matching @xcite @xcite @xcite is a pivotal task in both computer vision and natural language processing, which bridges data across heterogeneous modalities. the objective is to return images correlated with a given † corresponding author. ## conclusion in this paper, we investigate the problem of imagetext matching and propose a novel deep unsupervised hashing-based approach termed demo. the crux of our demo is to explore the latent semantic distributions of each sample for effective semantics structure mining. specifically, we characterize each image with multiple augmented views, which are regarded as samples from its intrinsic semantic distribution. then, a non-parametric distribution divergence is employed to ensure a robust and accurate similarity structure in the process of similarity generation, which serves as guidance for the optimization. extensive experimental results across multiple datasets substantiate the efficacy of demo.