I nv GC : Robust Cross-Modal Retrieval by Inverse Graph Convolution (2023)

Abstract: Over recent decades, significant advancements in cross-modal retrieval is mainly driven by breakthroughs in visual and linguistic modeling. However, a recent study shows that multi-modal data representations tend to cluster within a limited convex cone (as representation degeneration problem), which hinders retrieval performance due to the inseparability of these representations. In our study, we first empirically validate the presence of the representation degeneration problem across multiple cross-modal benchmarks and methods. Next, to address it, we introduce a novel method, called InvGC, a post-processing technique inspired by graph convolution and average pooling. Specifically, InvGC defines the graph topology within the datasets and then applies graph convolution in a subtractive manner. This method effectively separates representations by increasing the distances between data points. To improve the efficiency and effectiveness of InvGC, we propose an advanced graph topology, LocalAdj, which only aims to increase the distances between each data point and its nearest neighbors. To understand why InvGC works, we present a detailed theoretical analysis, proving that the lower bound of recall will be improved after deploying InvGC. Extensive empirical results show that InvGC and InvGC w/LocalAdj significantly mitigate the representation degeneration problem, thereby enhancing retrieval performance.

## introduction cross-modal retrieval (cmr) @xcite @xcite , which aims to enable flexible retrieval across different modalities, e.g., images, videos, audio, and text, has attracted significant research interest in the last few decades. the goal of cmr is to learn a pair of encoders that figure 1 : a 2d visualization of the representation of text data uniformly sampled from the mscoco dataset generated by clip @xcite and then reduced by pca. as shown in figure 1a , the representation degeneration problem can be observed as all the representations are concentrated in a small convex cone, while after employing invgc, representations are better separated. µ sim , σ sim , and min sim are the mean, standard deviation, and minimum of the similarity. ∆ deg is the score of representation degeneration problem defined in equation ( 2 ) . however, @xcite demonstrate that the current common learning paradigm of cmr leads to the representation degeneration problem, which concentrates all the representations in a small (convex) cone @xcite in image-text retrieval and the cosine similarity between any two points is positive, as shown in figure 1a . consequently, retrieval performance will be significantly affected @xcite . due to the limitation of space, detailed related works are presented in appendix a. in this paper, to step forward in addressing representation degeneration problem and further improve the retrieval performance, we first empirically test whether it is prevalent across various cross-modal retrieval models and datasets, including video, image, text, and audio. we found that the representations of mscoco generated by clip are gathered in a very narrow cone in the embedding space, proving the existence of representation degeneration problem, as shown in figure 1a . this case does not stand alone since similar observations are observed across several cross-modal retrieval models and datasets as shown in appendix c.3. next, to model how severe the representation degeneration problem is and the relationship between this problem and the retrieval performance, drawing inspiration from previous work @xcite @xcite @xcite , we define it in cmr as the average similarity between each point and its nearest neighbor in the gallery set, as shown in equation (2). we observe that the scores are very high across different datasets and methods. they are able to model this problem as a high score always leads to more concentrated data distribution as shown in appendix c.3. while cmr has suffered from this problem, on the other side, the graph convolution @xcite and average pooling @xcite , which are widely employed in graph neural networks @xcite @xcite and deep neural networks @xcite , respectively, are designed to move the representations closer to each other if they are semantically similar @xcite . it might lead to the emergence of representation degeneration problem. drawing inspiration from the graph convolution and average pooling, we propose a novel method, invgc, which separates representations by performing the graph convolution inversely to separate representations with a bigger margin as shown in figure 1a . specifically, different from the vanilla graph convolution, considering one modality, in-vgc separates representations by subtracting the representation of the neighboring nodes from each node, instead of aggregating them as, to evaluate the effectiveness of invgc and in-vgc w/localadj, we conducted experiments on eight cross-modal benchmarks @xcite @xcite @xcite @xcite . experimental results show that invgc alleviates representation degeneration problem across different datasets and methods and improves retrieval performance as a by-product. in summary, our contributions are as follows foot_0 : it is shown to be more effective and efficient. ## invgc to alleviate the representation degeneration problem and further boost the retrieval performance, we design a post-processing method, called invgc, which does not need any additional training. our idea is generally based on the mathematical principles laid out in equation (1) and figure 1 , where the inverse version of graph convolution is able to decrease the similarity between data points and their neighbors. for the sake of clear representation, we use the cosine similarity as the similarity metric in this section following @xcite , as it is the most common practice in cross-modal retrieval foot_1 . the formal definition of invgc is presented in section 3.3. ## experiments we conduct a series of experiments to demonstrate that invgc can efficiently alleviate the representation degeneration problem in a post-hoc manner. we compare invgc and invgc w/localadj with the baseline performance produced by the representation model adopted. we also introduce the inverse version of average pooling, namely avg-pool, as another baseline. a series of ablation studies indicate that invgc addresses the representation degeneration problem by reducing the similarity between points in the gallery set and is not sensitive to hyperparameters and the amount of training data. ## conclusion this paper addressed representation degeneration problem in cross-modal retrieval, which led to a decrease in retrieval performance. the representation degeneration problem was validated across multiple benchmarks and methods. to alleviate this issue, we proposed a novel method called invgc, inspired by graph convolution and average pooling. the method established a graph topology structure within the datasets and applied graph convolution in an inverse form with subtraction over the neighborhood. additionally, we designed the adjacency matrix, localadj, that only leveraged the nearest neighbors of each data point rather than the entire dataset, resulting in a more effective and efficient method, invgc w/localadj. both invgc and invgc w/localadj were validated through theoretical analysis and demonstrated their ability to separate representations. finally, extensive experi-ments on various cross-modal benchmarks showed that both of our methods successfully alleviated the problem of representation degeneration and, as a result, improved retrieval performance.