Paper: Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR-2024)

Reviewer comment (Evaluation):
The abstract and introduction emphasize the importance of the modality gap of the feature space, however, the results in the experiments seem show that the key component is the alignment noise. For instance, in Table 2, C_2^2 (only use connect and corrupt) can achieve a similar performance as the C^3. Compared to C^1, only adding collapse boost 16.3 but 41 when only adding alignment noise.

Author response:
**Why corrupt ($C_2^2$) seems more effective than collapse ($C_1^2$)?**

> The abstract and introduction emphasize the importance of the modality gap of the feature space, however, the results in the experiments seem show that the key component is the alignment noise. For instance, in Table 2, $C_2^2$ (only use connect and corrupt) can achieve a similar performance as the $C^3$. Compared to $C^1$, only adding collapse boost 16.3 but 41 when only adding alignment noise.
> 

Thanks for the question. While $C_2^2$ (corrupt) seems to be more effective than $C_1^2$ (collapse) in Table 2 (image captioning on CLIP embedding space), they seem to be similarly effective in Table 3 (text-to-image generation on CLIP embedding space) and Table 4 (image / video / audio captioning on ImageBind embedding space).

We hypothesize that the greater effectiveness of $C_2^2$ is because $C_2^2$ injects random noise in the embedding space during training, including the direction of the modality gap, which potentially reduces the model's sensitivity to this gap. Nevertheless, given the substantial size of the modality gap, $C_1^2$ remains necessary to diminish this gap, thereby enhancing the overall performance of $C^3$. 

It is also worth noting that $C^3$ adds near-zero additional computational cost over baselines. As our experiments demonstrate its effectiveness across a wide range of tasks, it can be a template for future work.

Reviewer comment (Evaluation):
In section 3.3, it provides some statistic values to show the space geometry. I think such statistics may not actually reveal the space geometry since it averages all possible values. Is that better to provide a histogram of such statistics to demonstrate the space geometry? Furthermore, based on the constant modality gap analysis, is that the E_{i,j}[cos(d_i,d_j)] should have a value close to 1 since they should be parallel except for the noise effect.

Author response:
**Histogram of statistics revealing the space geometry**

> In section 3.3, it provides some statistic values to show the space geometry. I think such statistics may not actually reveal the space geometry since it averages all possible values. Is that better to provide a histogram of such statistics to demonstrate the space geometry? Furthermore, based on the constant modality gap analysis, is that the $E_{i,j}[\cos(d_i,d_j)]$ should have a value close to 1 since they should be parallel except for the noise effect.

Thank you for the suggestion and question. We have now included the histogram, as well as a detailed explanation of how to interpret these statistics in Appendix C.

For $E_{i,j}[\cos(d_i,d_j)]$, because there is a noise entangled in $d_i = c_\perp + \epsilon_i$ and $d_j = c_\perp + \epsilon_j$, the cosine similarity will not be 1. However, if we compute the group-level $E_{i,j}[\cos(d_{c_i},d_{c_j})]$, where $c_i$ and $c_j$ are 100 randomly sampled images, the value reaches 0.99. We provide a detailed explanation in Appendix C.

Reviewer comment (Evaluation):
The experiments are conducted on generation tasks, the quantitative performance is similar for the method and baseline, while in the qualitative examples, I can not tell which method is better based on three examples. It will be better to provide more qualitative examples.

Author response:
**Significance of the quantitative improvement and more qualitative examples**

> The experiments are conducted on generation tasks, the quantitative performance is similar for the method and baseline, while in the qualitative examples, I can not tell which method is better based on three examples. It will be better to provide more qualitative examples.
> 

Thank you for the suggestion. We have added more qualitative examples for image captioning and text-to-image generation in Appendix Figure 10 and Figure 11. We hope these examples provide additional evidence of our method’s effectiveness.

We also emphasize that the improvement of our method is significant. Per Reviewer bNKP’s suggestion, we have run $C^3$ in Table 2 three times with random seeds 1234, 5678, and 910 and reported the numbers below. There is a very small variance across different runs, and $C^3$ consistently outperforms all the baselines.

| Seed | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDER | SPICE |
| --- | --- | --- | --- | --- | --- | --- |
| 1234 | 71.0 | 27.6 | 25.0 | 52.0 | 93.2 | 18.3 |
| 5678 | 71.1 | 27.6 | 24.9 | 52.0 | 93.6 | 18.4 |
| 910 | 70.9 | 27.8 | 25.0 | 52.1 | 93.0 | 18.2 |
| Avg | 71.0 | 27.7 | 25.0 | 52.0 | 93.3 | 18.3 |
| Std | 0.1 | 0.1 | 0.0 | 0.0 | 0.3 | 0.1 |

Reviewer comment (Theory):
In the analysis of the space geometry of contrastive features, the paper proposes that the modality gap is a constant vector and orthogonal to each modality. But it can not be derived easily that why the modality gap is the constant vector from the dimension collapse as mentioned in section 3.1. Are there any formal propositions for the constant modality gap vector in the initialization stage?

Author response:
**Why modality gap vector is constant?**

> In the analysis of the space geometry of contrastive features, the paper proposes that the modality gap is a constant vector and orthogonal to each modality. But it can not be derived easily that why the modality gap is the constant vector from the dimension collapse as mentioned in section 3.1. Are there any formal propositions for the constant modality gap vector in the initialization stage?
> 

Thanks for the question. The constant vector of the modality gap *does not result from the initialization stage only*, but results from the joint effect of model initialization and optimization:

- During initialization, due to the dimensional collapse phenomenon (Definition 1), some dimensions of image and text embeddings are near constant, which we term ineffective dimensions.
- During optimization, the gradients will never be propagated into the shared ineffective dimensions between image and text embeddings (Lemma 1), causing these dimensions to remain the same constant.
- Finally, effective dimensions are aligned, and ineffective dimensions remain at initial values, resulting in a constant modality gap orthogonal to each embedding span.

Here we provide an example to help understand the process:

Suppose there are two image-text pairs in 3D space; only the 1st dimension of the image embeddings is effective, only the 2nd dimension of the text embeddings is effective, and all the other dimensions are ineffective.

During initialization, let us denote these embeddings as $x_1, x_2, y_1, y_2 \in \mathbb{R}^3$:

$x_1 = [0.7, 0.5, -0.5]$ (the 2nd, 3rd dimensions are constant across $x_i$)

$x_2 = [-0.7, 0.5, -0.5]$ (the 2nd, 3rd dimensions are constant across $x_i$)

$y_1 = [0.5, 0.7, 0.5]$ (the 1st, 3rd dimensions are constant across $y_i$)

$y_2 = [0.5, -0.7, 0.5]$ (the 1st, 3rd dimensions are constant across $y_i$)

During optimization, only the union of effective dimensions will be aligned, and the intersection of ineffective dimensions will remain constant:

$x_1 = [0.6, 0.6, -0.5]$ (the 3rd dimension remains constant because of no gradient)

$x_2 = [-0.6, -0.6, -0.5]$ (the 3rd dimension remains constant because of no gradient)

$y_1 = [0.6, 0.6, 0.5]$ (the 3rd dimension remains constant because of no gradient)

$y_2 = [-0.6, -0.6, 0.5]$ (the 3rd dimension remains constant because of no gradient)

Therefore, we have a constant gap with a distance of 1.0, which is also orthogonal to the embedding spans.

Reviewer comment (Experiments):
The paper aims to mitigate the modality gap between feature space of different modalities, but in the generation task or encoder/decoder-based architecture, is that the closeness of the features from different modalities indicate a better generation performance? I think it would be better to refer to previous works or conduct this kind of experiment to show the assumption of this paper is true in real applications.

Author response:
**Is closeness of features from different modalities indicate a better generation performance?**

> The paper aims to mitigate the modality gap between feature space of different modalities, but in the generation task or encoder/decoder-based architecture, is that the closeness of the features from different modalities indicate a better generation performance? I think it would be better to refer to previous works or conduct this kind of experiment to show the assumption of this paper is true in real applications.
> 

Thank you for your suggestion. We added an experiment to verify this assumption. We train a GPT-2 text generator (image captioner) upon CLIP’s text embeddings $x$. During inference, we manually shift all the $x$ to $x+c$ to simulate the modality gap (a constant vector orthogonal to original spans). We report the captioning performance in terms of gap distance $\|c|\$ in the table below. We observe there are substantial performance drops when the gap grows, showing the need to align embeddings and close the gap. We have now included this discussion in Appendix G.

| Gap Distance $\|c\|$ | ROUGE-1 | ROUGE-L | METEOR |
| --- | --- | --- | --- |
| 0.0 | 85.5 | 81.2 | 83.1 |
| 0.2 | 76.3 | 71.5 | 73.8 |
| 0.4 | 55.8 | 50.6 | 52.5 |
| 0.6 | 40.6 | 35.9 | 36.6 |
| 0.8 | 30.5 | 26.7 | 26.5 |
| 1.0 | 24.1 | 21.0 | 20.3 |
| 1.5 | 16.6 | 14.4 | 13.8 |
| 2.0 | 13.8 | 12.1 | 11.6 |

Reviewer comment (Novelty):
The proposed C3 algorithm has limited novelty on its own given that each step has been studied in previous work. However, the combination of these steps is new and well-motivated by the theoretical framework developed in this paper, which mitigates the lack-of-novelty issue.

Author response:
Thank you for the question. Yes, as you mentioned, while existing works have empirically proposed collapse and corrupt operation separately, we differ from them in three aspects:

1. Our method is built on the theoretical understanding of multi-modal contrastive representation space geometry.
2. We unified these insights and showed the combination of them leads to superior performance than each of them.
3. We proved the effectiveness of our method in a wide range of tasks, modalities, data, and embedding spaces, which can become a standard recipe for all future works built on multi-modal contrastive embedding space.

Reviewer comment (Reproducibility):
How is the "Collapse" step implemented (i.e., computing e_x' and e_y')? Is it the same as batchnorm?

Author response:
Thanks for the question. The “collapse” step can be viewed as a distribution norm, where the image embedding mean $\bar{e}_x$ and text embedding mean $\bar{e}_y$ are pre-computed on the entire training set and subtracted during training and inference. The batch norm is an approximation with a larger variance. We provided an algorithm formulation of $C^3$ in Appendix C for further clarification.

Reviewer comment (Evaluation):
The discussion on the alignment noise could be improved: in particular, the results in Table 1 are left for the reader to infer. The same statistics could also be easily computed upon any other modality combination in the appendix, it would be useful to see if it still applies.

Author response:
Thank you for your suggestion. We have now included a detailed explanation of how to interpret these statistics in Appendix C. We also included statistics of embeddings from different modalities in Appendix C. We hope these edits will help readers easily understand Table 1.

Reviewer comment (Theory):
Since the modality gap is due to the dimensional collapse, would reducing the dimensionality to the effective one help overcoming the issue?

Author response:
Thanks for the question. To understand how dimensionality affects the gap, we initialized CLIP with different dimensions. We find that, in lower dimensions, the ratio of effective dimensions increases for both encoders (but cannot reach 100%) and the modality gap decreases.

| Dimension | Effective Image Dimension | Effective Text Dimension | Init Gap Distance |
| --- | --- | --- | --- |
| 64 | 18 | 60 | 0.386 |
| 128 | 20 | 110 | 0.554 |
| 256 | 23 | 176 | 0.803 |
| 512 | 25 | 230 | 1.136 |

Nonetheless, we hope to clarify that while the relationship between the dimensionality and the gap is an interesting question, this gap can be well addressed by the collapse operation based on our theoretical analysis. Therefore, the gap actually does not matter.

Reviewer comment (Theory):
Is there any relation between the decomposition of the modality gap with the content-style-modality specific decomposition assumed e.g. in [1]? Briefly, each latent vector in a multi-modal contrastive learning space is there assumed to have a part that is shared across modalities, i.e. the content, one that is shared but with some distortion, i.e. the style, and one that is not shared at all, i.e. the modality-specific component. Is it possible that the modality specific component in [1] is just the constant component caused by the different initializations seen in this work?

Author response:
Thank you for pointing out this interesting work. The modality-specific component in [1] appears to be more closely related to the "alignment noise" than the "modality gap". For instance, [1] identifies "object rotation" as a modality-specific component in the visual domain. This suggests that changes like rotating an image would solely affect image embeddings, with no parallel effect in text embeddings. Therefore, this phenomenon seems to align more with alignment noise, where image and text embeddings are not perfectly aligned in the shared representation space.

Reviewer comment (Theory):
This paper proposes to align representations from different modalities but without convincing justification. In fact, it remains uncertain what effects are relevant to aligning modalities. [1] reports that making the modality gap too small or too large harms performance. [5] proves that strictly aligning modality representations is suboptimal. Therefore, I suggest adding reasons for aligning modalities.

Author response:
Thank you for the suggestion. If embeddings from different modalities are aligned, we can train a model on one modality and then infer on another modality, enabling us to build cross-modal applications with only uni-modal data. This is an emerging field that lacks principled approaches to be easily applied without requiring more empirical tuning.

[1,5] did not find aligning embeddings to be helpful because they use zero-shot retrieval settings where no model further consumes these embeddings. In contrast, [2,3,4] has a classifier, text generator, and image generator trained on these embeddings. Therefore, switching modality embeddings in [2,3,4] without aligning them will suffer from a significant performance drop.

We added an experiment to explain this further. We train a text generator (image captioner) over CLIP’s text embeddings $x$. During inference, we manually shift all the $x$ to $x+c$ to simulate the modality gap (a constant vector orthogonal to original spans). We report the captioning performance in terms of gap distance $\|c\|$ in the table below. We observe substantial performance drops when the gap grows, showing the need to align embeddings. We have now included this discussion in Appendix G.

| Gap Distance $\|c\|$ | ROUGE-1 | ROUGE-L | METEOR |
| --- | --- | --- | --- |
| 0.0 | 85.5 | 81.2 | 83.1 |
| 0.2 | 76.3 | 71.5 | 73.8 |
| 0.4 | 55.8 | 50.6 | 52.5 |
| 0.6 | 40.6 | 35.9 | 36.6 |
| 0.8 | 30.5 | 26.7 | 26.5 |
| 1.0 | 24.1 | 21.0 | 20.3 |
| 1.5 | 16.6 | 14.4 | 13.8 |
| 2.0 | 13.8 | 12.1 | 11.6 |

Reviewer comment (Evaluation):
Despite that the authors have conducted experiments on various tasks, the comparison with existing methods is limited. Most comparisons in this paper are ablating over different components in C^3. Tab.2 shows marginal improvement over CapDec without reporting std over independent runs, which is not convincing. Tab.3, Tab.4 and Tab.5 report few comparisons with SOTA methods.

Author response:
Thank you for the question. We have run our method $C^3$ in Table 2 three times with random seeds 1234, 5678, and 910 and reported the numbers below. We observe a very small variance across different runs, and $C^3$ consistently outperforms all the baselines. We have included these results in Table 2.

In terms of baselines, we have included the most comparable baseline Lafite in Table 3. To the best of our knowledge, there is no similar zero-shot audio/video captioning baseline in Table 4. 

We hope to emphasize that Table 3/4 is mainly used to show the generalization of our method across different tasks, data, modalities, and embedding spaces, where we can see $C^3$ consistently outperforms all the other baselines by a large margin, showing the importance to align embeddings.

| Seed | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDER | SPICE |
| --- | --- | --- | --- | --- | --- | --- |
| 1234 | 71.0 | 27.6 | 25.0 | 52.0 | 93.2 | 18.3 |
| 5678 | 71.1 | 27.6 | 24.9 | 52.0 | 93.6 | 18.4 |
| 910 | 70.9 | 27.8 | 25.0 | 52.1 | 93.0 | 18.2 |
| Avg | 71.0 | 27.7 | 25.0 | 52.0 | 93.3 | 18.3 |
| Std | 0.1 | 0.1 | 0.0 | 0.0 | 0.3 | 0.1 |

Reviewer comment (Experiments):
Section 3.2 mentions the effect of temperature, but no discussion is given in experiments. What are the effects of modifying temperature in stage 1 and std of Gaussian in stage 3?

Author response:
Thanks for the interesting question. We leave this to future work as there are no open-sourced CLIP models trained with different temperatures.

Thank you again for your feedback, which is very helpful in improving the paper. We hope the above responses and changes to our manuscript adequately address your concerns, and that you may be willing to improve your rating as a result. Please let us know if you have further questions or concerns!