Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks (2023)

Abstract: In this work, we present a post-processing solution to address the hubness problem in cross-modal retrieval, a phenomenon where a small number of gallery data points are frequently retrieved, resulting in a decline in retrieval performance. We first theoretically demonstrate the necessity of incorporating both the gallery and query data for addressing hubness as hubs always exhibit high similarity with gallery and query data. Second, building on our theoretical results, we propose a novel framework, Dual Bank Normalization (DBNorm). While previous work has attempted to alleviate hubness by only utilizing the query samples, DBNorm leverages two banks constructed from the query and gallery samples to reduce the occurrence of hubs during inference. Next, to complement DBNorm, we introduce two novel methods, dual inverted softmax and dual dynamic inverted softmax, for normalizing similarity based on the two banks. Specifically, our proposed methods reduce the similarity between hubs and queries while improving the similarity between non-hubs and queries. Finally, we present extensive experimental results on diverse language-grounded benchmarks, including text-image, text-video, and text-audio, demonstrating the superior performance of our approaches compared to previous methods in addressing hubness and boosting retrieval performance.

## introduction cross-modal retrieval (cmr) facilitates flexible information retrieval across various modalities, including images, videos, audio, and text, by extracting discriminative features and summarizing information from multiple modalities. recently, supported by the development of pre-training multimodal models @xcite @xcite hub hub figure 1 : top: the hubness problem @xcite in cross-modal retrieval where queries q 1 , q 2 , and q 3 are associated with their respective galleries g 1 , g 2 , and g 3 . the hub (g 1 ) is the nearest neighbor to multiple queries (q 1 , q 2 , and q 3 ), resulting to suboptimal retrieval performance. middle: previous methods employ a query bank to normalize similarities. bottom: our dbnorm tackles the hubness problem by utilizing both gallery and query banks. it reduces the similarity between the hub g 1 and all queries while simultaneously enhancing the similarity between non-hubs g 2 and g 3 and all queries, resulting in improved performance. @xcite @xcite , innovative retrieval methods @xcite @xcite , and cross-modal benchmarks @xcite , significant advances have been made in image-text retrieval @xcite @xcite , video-text retrieval @xcite @xcite @xcite @xcite @xcite , and audio-text retrieval @xcite , achieving satisfactory retrieval performance. in the most popular cmr paradigm, deep neural networks are employed to project data from diverse modalities into a shared high-dimensional vector space, enabling direct comparison through distance measures such as cosine similarity and ℓ 2 similarity. however, within this paradigm, the occurrence of hubness @xcite poses a challenge. hubness refers to the phenomenon where certain data points, known as hubs, frequently emerge as nearest neighbors of other points, resulting in a decline in retrieval performance. to empirically illustrate the existence of hubness, we visualize the distribution of the frequency at which each gallery data point is retrieved by queries, as shown in figure 3 and figure 6 in the appendix. we observe that hubness is prevalent in video-text, image-text, and audio-text retrieval, significantly impacting the performance of retrieval systems @xcite . prior research on mitigating hubness can be roughly categorized into training methods @xcite and post-processing methods @xcite @xcite @xcite , with particular attention in zero-shot learning @xcite and bilingual word translation @xcite @xcite . due to the limitation of space, detailed related works are presented in appendix a. in this paper, we focus on addressing hubness in a post-processing manner. however, hubness in the context of cmr received limited attention until the introduction of qbnorm @xcite , which is a pioneering method specifically designed for cmr and incorporates a novel technique called dis. it addresses hubness by selectively reducing the similarity between queries and hubs, thereby mitigating the impact of hubs. it follows the principle that hubs exhibit high similarity with query data to identify hubs. this prompts us to investigate whether hubs also demonstrate high similarity with gallery data. therefore, the first research question is: ## our methods in this section, we first address rq1 through a theoretical demonstration of the universality of hubness and the necessity of incorporating both query and gallery data to address hubness. next, to answer rq2, we propose a novel method called dbnorm, which builds upon our theoretical analysis, along with two novel similarity normalization techniques: dual inverted softmax and dynamic dual inverted softmax. these methods effectively utilize both query and gallery data to mitigate hubness. ## experiments in this section, we first conduct experiments to demonstrate that dbnorm can efficiently improve the retrieval performance as a by-product. next, our ablation studies indicate that dbnorm effectively mitigates hubness by reducing skewness and exhibiting robustness to hyperparameters. for clarity, we use is, dis, dualis, and dualdis to represent qbnorm (is), qbnorm (dis), dbnorm (dualis), and dbnorm (dualdis), respectively. following qbnorm, queries remain independent of each other as in practice, queries do not always occur at the same time. detailed comparison with gc and csls is deferred to appendix b.6. ## conclusion in this work, we addressed the issue of hubness in cross-modal retrieval through a post-processing approach. first, we theoretically proved that hubs exhibit high similarity with data from both the query and gallery modalities. then, motivated by our theoretical results, we proposed a novel post-processing method, dual bank normalization, along with two novel methods, i.e., dual inverted softmax and dual dynamic inverted softmax, which leveraged gallery and query banks to reduce similarities with hubs and improve similarities with non-hubs gallery data. finally, with extensive experiments across a range of tasks, models, and benchmarks, we demonstrated the superiority of our proposed methods over previous methods in addressing hubness and improving performance.