Joint Similarity Guidance Hash Coding Based on Adaptive Weight Mixing Strategy For Cross-Modal Retrieval (2024)

Abstract: “There is a continuous and explosive growth of multimodal data. Efficient cross-modal hash-ing retrieval is of significant importance in conserving computational resources.To further en-hance the attention to informative data within modalities and capture the semantic correlationsin cross-modal data, we propose an enhanced deep Joint-Semantics Reconstructing Hashing al-gorithm, which is the Joint Similarity Guidance Hash Coding Based on Adaptive Weight MixingStrategy(JSGHCA). The algorithm focuses on delving deeper into the correlations of the data incross-modal. We introduce the adaptive weight mixing strategy to construct the semantic affinitymatrix, so that the matrix can identify each modal data with specific weight in each batch. Atthe same time, in the process of the hash code generation, we introduce collaborative attentionmechanism. It helps the model to pay more attention to the local information of each modality,thereby capturing the semantic features within each modality more accurately. Additionally, itenables the model to jointly process the attention across different modalities and extract sharedsemantic features more precisely. Experimental results show that the proposed model is signifi-cantly better than the deep joint semantic reconstruction hash algorithm on multiple benchmarkdatasets.”

## introduction with the rapid development of digital technology, multimodal information is omnipresent in our lives. on the internet, various modalities of data such as text, images, videos, and audio provide us with a rich information experience. the diverse structures of different modalities of data give rise to data heterogeneity. this heterogeneity poses a new challenge: how to effectively establish semantic associations and perform retrieval across different modalities? cross-modal retrieval is divided into cross-modal real-value retrieval and cross-modal hash retrieval. in cross-modal real-value retrieval, the dimensionality of real-valued vectors is relatively high, requiring large-scale matrix operations and high computational complexity. cross-modal hash retrieval aims to eliminate differences between modalities and project them into a low-dimensional common hamming space for retrieval tasks. in this regard, deep joint-semantics reconstructing hashing (djsrh @xcite ) marks a significant milestone. it optimizes cross-modal semantic relationships by introducing a joint semantic affinity matrix. however, there is interference from irrelevant information of the two modalities during the synthesis of the joint semantic affinity matrix. additionally, there are challenges in capturing intra-modal local information and utilizing cross-modal related information during hash code generation. considering these limitations, we propose a joint similarity guidance hash coding based on adaptive weight mixing strategy (jsghca) algorithm. the traditional fixed weight allocation strategy may be affected by irrelevant information when processing multimodal data, leading to a decrease in retrieval accuracy. to this end, we propose a dynamic weight allocation method that can dynamically adjust weights based on the importance of modal information within the data batch. through this method, we can more accurately identify and emphasize useful modal information when constructing a joint semantic affinity matrix, while filtering out useless information, thereby improving the accuracy and robustness of retrieval. when dealing with the local information within a single modality and capturing the complex semantic correlation between different modalities, the existing methods often have shortcomings. to address this issue, we introduce an enhanced local attention mechanism (single attention mechanism, si-attention) that focuses more on key information within the modality during the feature extraction stage. meanwhile, we also designed a cross modal semantic capture module (joint attention mechanism, jo-attention) that can effectively capture and utilize the semantic associations between different modalities during the hash code generation process. through these improvements, our model can generate richer and more accurate hash codes, thereby improving the performance of cross modal retrieval. the main contributions of this research are as follows: cross-modal retrieval, an extension of single-modal retrieval, tackles the intricate relationships within multimodal data. recently, cross-modal hashing learning has been extensively employed to reduce feature storage and simplify retrieval.cross-modal hashing learning can be divided into supervised and unsupervised methods based on the need for semantic supervision.