Paper: GeRA: Label-Efficient Geometrically Regularized Alignment (ICLR-2024)

Reviewer comment (Evaluation):
While the paper reports performance improvements, it lacks clarity on their statistical significance. For instance, when a 3% improvement in performance is highlighted, it becomes crucial to understand the variance arising from different initialization seeds. Reporting standard deviations would have offered a clearer picture of the model's robustness and reliability.

Author response:
We conducted 5 runs with 100K paired points for both GeRA and unregularized contrastive learning, of the experiment presented in Figure 3. We report the mean and standard deviation below.

GeRA:
MEAN:  0.3507
STD: 0.0063

Contrastive Loss:
MEAN: 0.3105
STD: 0.0048

ASIF is not probabilistic and returns constant results.

This indicates that the performance of GeRA is insensitive to initialization and the improvement over vanilla contrastive loss is significant. Obtaining standard deviations for all experiments takes time and is not feasible within the discussion period, but we will add standard deviations to our figures in the final draft.

Reviewer comment (Experiments):
The manuscript introduces additional hyperparameters, yet it does not provide sufficient insight into their impact on downstream performance. Without guiding intuition, the only approach seems to be extensive hyperparameter tuning, which might be computationally expensive and time-consuming.

Author response:
The main additional hyperparameters in comparison to vanilla contrastive loss that may require some tuning are $\alpha$ (balancing the contrastive loss term and the geometric loss term) and kernel hyperparameter $\epsilon$. The choice of $\alpha$ signifies the degree to which we focus on preserving local geometry at the expense of potentially reducing the alignment quality. The kernel scale $\epsilon$ defines the notion of neighborhood size in the kernel. We set it to equal to the mean of the pairwise distances in the neighborhood, multiplied by a constant, which is the tuned hyper-parameter. Setting a lower constant puts more weight on similarities of closer neighbors in the kernel construction, decreasing the effective neighborhood size in the kernel (see equation 4) and focusing on more localized features. 

All other introduced hyperparameters have clear trends in terms of their effect on performance as demonstrated by our ablation studies, presented in Figure 7. We summarize the results below:

* *Number of Neighbors*: Most significant is the size of the neighborhood. The intuition here is the more samples we have in our neighborhood, i.e., the larger the kernel matrix $W$, the better we can capture the local geometry and hence preserve it. Figure 7 (left) clearly shows that the number of neighbors strongly correlates with the downstream alignment performance. However, the marginal increase in performance seems to diminish with larger neighborhoods, indicating already good performance using relatively small numbers of neighbors.

* *Sampling Technique*: We examined different ways of sampling the neighbors and found that the sampling technique has an effect on performance. We aim to select samples that best represent the local geometry. Hence, selecting only the closest neighbors preserves locality best. However, to obtain better continuity of the embedding space, and increase the amount of information gathered from the neighbors in different epochs, we subsample the neighbors from a larger neighborhood. Figure 7 (middle) demonstrates that sampling neighborhood points with a bias towards closest neighbors, i.e., higher probability of sampling closer neighbors while still including some information about further points, performs best overall.

* *Kernel Type*: Figure 7 (right) shows that the choice of the kernel type, i.e., the neighborhood encoding method affects performance as well. This plot demonstrates that the heat kernel is the best performing kernel, indicating its superiority in capturing the local geometry. This is also underlined by the theoretical properties of the heat kernel that can capture the geometry of the manifold and hence meets our expectations.

We added these clarifications and details to Appendix A.2.2 in the paper as well.

Reviewer comment (Reproducibility):
There is no publicly available code to reproduce the work.

Author response:
We will publish the code on acceptance

Reviewer comment (Theory):
The manuscript does not discuss the interplay between the density of the latent space and the preservation of its geometry. In particular for spaces where the points density changes in different regions, it may be problematic to use the same neighbors selection strategy for the whole space.

Author response:
Thank you for pointing this out. We added more details about how we compute the kernel scale $\epsilon$ to Appendix A.2.2, which addresses this point. In order to allow better adaptability of the kernel to neighborhoods of different sizes and densities, we compute the kernel scale $\epsilon$ as follows: $\epsilon = \sigma \times\mathrm{mean}(\mathrm{pairwise\  euclidean\  distances})$, where $\sigma$ is the tuned hyperparameter (constant).

Hence, by including the average distances within each neighborhood in the kernel normalization, the kernel adapts to the local scales and characteristics of these neighborhoods.

Reviewer comment (Evaluation):
The paper presents notably low performance figures for ASIF in the domain of speech-text alignment. How do these figures relate to the findings of [2] when using ASIF? An explanation or a comparative insight would help in contextualizing the reported results better and understanding potential disparities or improvements.

Author response:
Thank you for pointing out this relevant paper. Our results and the results presented in [2] using ASIF highlight the main advantage of our approach over ASIF. When the uni-modal embedding spaces are different, as suggested by [2] for the speech and text modalities, we hypothesize that ASIF needs a lot more paired samples to properly align the spaces, since extrapolation from a limited number of pairs is likely to be inaccurate. Moreover, the performance obtained by ASIF in [2] is using encoders that were trained on data that included paired points from the two modalities. In contrast, in our experiments, we use encoders that were trained on purely uni-modal data, and that may be the source of the performance gap. Our method for alignment of the uni-modal models combines the strengths of contrastive alignment with paired data, to match the uni-modal embedding spaces, and preserving the geometry of the respective spaces, thus it is more robust to the differences in uni-modal embedding spaces. For instance, in Figure 6, we see that vanilla contrastive loss performs quite well in aligning uni-modal models, significantly outperforming ASIF (unlike the text-image experiment in Figure 3), while our method further improves the performance of contrastive loss.

We added this discussion and citation of [2] to Appendix A.2.3 in the paper

Reviewer comment (Experiments):
One of the major motivations in the introduction for the method is to use unpaired data for training. However, I cannot find any experiment in section 5 that trains on a mixture of unpaired and paired data where paired data is small. If so, please name the dataset used in section 5. Is unpaired data referring to the data used for pretraining the models? If no experiments are done that use unpaired data during the alignment, at least the following sentence in the abstract should be corrected: “Our method leverages the manifold geometry of unpaired (unlabeled) data to improve alignment performance.” and the following sentence in Section 3: “...we propose to leverage unlabeled (unpaired) points from each modality to preserve the rich geometric structure of their original embedding spaces.”

Author response:
Thank you for pointing out this source of confusion. We added a clarification in Section 5.1 in the paper (paragraph Unpaired points) and in Section 5.4 (paragraph Dataset) that in all experiments we make use of both paired points, in the contrastive loss, and unpaired points, in the geometric loss when training the alignment neural networks: 

Section 5.1: “To preserve the local geometry of the pretrained unimodal models, we use unpaired points from each modality to compute the geometric regularization in equation 1. For the image and text experiments, we discard the pairing information of $6\times 10^6$ data points from CC12M and treat them as unpaired points used in the geometric regularization.”

Section 5.4: “In this experiment, we used up to $10^5$ paired points in the contrastive loss, and additional $10^5$ unpaired points in the geometric regularization.”

In addition, we add the following clarification in Appendix A.2.1: “To give a bit more detail on the use of unpaired data in our experiments, in the image-text experiments, the dataset we used for training is CC12M which is a paired dataset. However, during training we only consider the pairings for a small number of samples and use the (fraction of) remaining samples as unpaired data to simulate a scenario where there are limited amounts of paired data and many unpaired data points. More concretely, we take $M$ paired samples (used for contrastive loss) and include $N \gg M$ unpaired samples (distinct from the paired points used in the contrastive loss), where the unpaired points for each modality are chosen randomly and independently of the other modality. We leverage the unpaired data in the neighborhoods of each paired datapoint, and construct the kernels in the geometric regularization based on these neighboring unpaired points. Note that for a pair $(x,y)$, the neighborhood for $x$ is in general not the same as the neighborhood for $y$, i.e., the neighbors do not have to be pairs themselves.”

Reviewer comment (Evaluation):
The effectiveness of the method is limited to the low-data regime with paired data fewer than 10^5 samples and the performance is significantly lower than a model trained with just one order of magnitude more paired samples. So at least for image-text and speech-text modalities where available paired data is significantly more than 10^5, it is not clear how the proposed method can be helpful. In other words, when would it be useful to use GeRA instead of ASIF or the standard contrastive loss? Would it be for training other multi-modal models where the paired data is few? If so, do authors believe that the two examples of image-text and speech-text can be extrapolated to other multi-modal models?

Author response:
The method presented in this work is model-agnostic and can therefore be applied to any combination of modalities, given good pre-trained unimodal models. GeRA is especially useful in modality pairs for which there are limited amounts of paired data points, as demonstrated by Figures 3 through 6, for relatively few paired data points. 

We demonstrate our approach on the image-text and speech-text modality pairs due to their clear ground truth, but it can be applied to any modality pair. Hence, as pointed out correctly by the reviewer GeRA will be especially useful for modalities where the amount of paired data is low.
Comparing to ASIF, there are two limiting factors:

1. In the original paper, ASIF's evaluation was limited to the ViT image encoder - a very specific image encoder - that was partially trained on ImageNet images during pretraining. In Fig 10 we use different pretrained encoders. These results show that, compared to GeRA, ASIF struggles much more with other choices of encoders as well as when it comes to retrieving CC12M images and captions (See Figure 4). 

2. ASIF has much higher inference time than GeRA.

Reviewer comment (Evaluation):
Figure 1: This figure is interesting and shows that the nearest neighbors remain relatively the same. Is there a standard evaluation metric that is improved because the geometry of modalities remains almost the same? Most of section 5 considers evaluation metrics specific to multi-modal models. Can we also evaluate these models for unimodal metrics such as unimodal retrieval such as image-image text-text retrieval?

Author response:
To address this question, we conducted a new experiment using an alignment model trained with GeRA and the unregularized model trained only with the contrastive loss. Both models were trained using $10^6$ paired points. Note that these are the same models used to create Figure 3.
The unimodal metric that we compute is the image-to-image kNN accuracy on ImageNet. More concretely, we took the validation set of ImageNet which consists of 50 samples per class. We randomly selected 10 samples from each class as labeled training data. We embed the training images and the remaining images using our models. Each image was then assigned to a class based on the majority vote among its nearest 5 neighbors (results with 7 and 9 neighbors were similar). We then compared the performance of GeRA against the unregularized contrastive learning model and the original embedding space generated by ViT before applying our transformation. 
| Method | $k=5$ |
|-------------------------|---------|
| ViT only | $0.76$ |
| No Geom. Regularization | $0.67$ |
|  Geom. Regularization | $0.75$ |

This experiment demonstrates that GeRA preserves the geometry of the image space obtained by the ViT model pre-trained on the image domain, while alignment with vanilla contrastive loss disturbs it.
We added the description and results of this experiment to Appendix A.1 in the paper.

Reviewer comment (Evaluation):
Is Figure 1 related to any model trained and evaluated in Section 5? Can we confirm that the model trained with GeRA is also a strong model according to zero-shot metrics?

Author response:
The metrics reported in Figure 1(b) are computed based on the same models used for comparing zero-shot performance in Figure 3. Specifically, in Figure 1 we are using the unregularized model with only the contrastive loss (labeled ’’No Geom. Regularization’’) and the GeRA model with $K=150$ neighbors (labeled ``Geom. Regularization’’). Both models were trained with $10^6$ paired points. Figure 3 confirms that GeRA performs well according to the zero-shot metric as well. We added more detail on the creation of Figure 1(b) in Appendix A.1 of the paper.

Reviewer comment (Theory):
Eq. 3: What is the dimensionality of W? Is it MxM or N_k x N_k? Is the loss meaningful even if the nearest neighbors in the original and the new space are different?

Author response:
The dimensionality of $W$ is $N_k \times N_k$, as it computes the local neighborhood structure for each sample involved in the contrastive loss based on its k-nearest neighbors, which include unpaired data. 
The aim of the geometric regularization in the loss is to enforce that the nearest neighbors in the original space and in the new space remain similar (as depicted by Figure 1). For this purpose, we compute the neighbors based on the original space only, and use the *same set of neighbors* in the geometric loss for both the original and the new space, thus encouraging an alignment that preserves the local neighborhood structure.

Reviewer comment (Novelty):
This paper has limited novelty. Adding a geometrically regularization term is too conventional in manifold learning.

Author response:
Our objective in this paper is to align multiple modalities given limited amounts of paired data. From this perspective, the main novelty of our paper is not the geometric regularization term itself, but rather the entire framework. 
Our work combines two concepts that have not previously been studied together: preserving uni-modal geometry using a geometric loss term and multi-modal alignment based on uni-modal models. When the amount of paired data is limited, a multi-modal model cannot be sufficiently trained from scratch. In addition, we demonstrate that aligning embedding spaces of pre-trained unimodal models without taking into account the local geometry degrades performance and distorts neighborhood structures. We thus propose a new semi-supervised framework for aligning pre-trained unimodal models that preserves the local neighborhood structures. This resultes in improved performance in multi-modal zero-shot and retrieval tasks.

Secondarily, geometric regularization is indeed a pillar of machine learning theory and practice, and the design of effective geometric regularizers for different learning problems is far from trivial. The use of this time-tested and effective strategy for a new problem in machine learning should not be grounds for rejection, especially since our particular regularization technique is new.

Reviewer comment (Novelty):
The motivation of this paper needs further discussion. There are millions or even billions of paired speech-text and image-text data, why do we need a label-efficient semi-supervised method?

Author response:
Our approach is model-agnostic and can therefore be applied to any combination of modalities, given good pre-trained unimodal models. GeRA is indeed especially useful in modality pairs for which there are limited amounts of paired data points, as demonstrated by Figures 3 through 6, for relatively few paired data points. In this paper, we demonstrate our approach on relatively small datasets of the image-text and speech-text modality pairs due to their availability and clear ground truth, however, it can be applied to any modality pair.

Reviewer comment (Evaluation):
How long does it take to get the nearest neighbor information?

Author response:
Using Faiss [*] for computing 800 nearest neighbors for 6 millions vectors in a 768 dimensional space it takes 45 to 55 minutes on a NVIDIA GeForce RTX 3090. We added these details to Appendix A.2.1 in the paper as well. 

[*] Johnson, Jeff, Matthijs Douze, and Hervé Jégou. 2017. “Billion-Scale Similarity Search with GPUs.”, IEEE Transactions on Big Data.