Paper: Bridging Vision and Language Spaces with Assignment Prediction (ICLR-2024)

Reviewer comment (Experiments):
The lack of ablations and feature space visualizations to show the effectiveness of the proposed loss is a major concern, and the contribution over existing works like MAGMA is not enough.

Author response:
**[W1, W2, Q1] Ablation for the loss function**

We present the ablation analysis corresponding to the learning objectives in the general response.
Please refer to them.

Reviewer comment (Experiments):
The paper is an extension of MAGMA (Merullo et. al. Linearly mapping from image to text space. In ICLR, 2023.). While MAGMA is discussed in the paper, the difference between this paper and MAGMA is the proposed assignment prediction loss. However, the effectiveness of the proposed loss is not shown clearly in the paper.

Author response:
**[W1, W2, Q1] Ablation for the loss function**

We present the ablation analysis corresponding to the learning objectives in the general response.
Please refer to them.

Reviewer comment (Experiments):
No ablation results are provided to show the effectiveness of the proposed loss — this is related to Point 2. Since the major contribution lies in this loss, an ablation to show the contribution of this loss in the final results is very critical.

Author response:
**[W3, Q3] Difference between VLAP and the previous linear transformation-based approaches**

Thanks for the reference.
Similar to the image captioning objective of LiMBeR, LLaVa learns the linear projection with the language generation objective, which aims to provide target answers from given transformed visual and language instruction representations.
Although LiMBeR and LLaVa have shown two unimodal models can be bridged through the language modeling objective, they are insufficient to reduce the modality gap, as shown in our previous response.
Our proposed objective not only improves overall performance in the linear transformation-based methods, but also significantly facilitates the connection of the visual encoder even if it is not trained with language guidance (i.e., CLIP encoder).
In addition, our objective can be an alternative to the conventional objectives for cross-modal alignment, such as the image-text matching and image-text contrastive objectives.
We add the missing reference LLaVa in the main paper.

Reviewer comment (Presentation):
The paper would be easier to read if the method names (abbreviations) in the results tables come with citations next to them, or are described in the text to show which is which.

Author response:
**[W5] Citation.** We added references along with the model names to the table captions.

Reviewer comment (Evaluation):
Could visualizations (e.g., t-SNE) over the feature space with and without the assignment loss be provided, to show its effects in aligning the features?

Author response:
**[W4, Q2] Visualization.** We presented the UMAP visualization corresponding to visual arithmetic and ablation for the learning objectives. We can verify the effectiveness of the proposed VLAP in the feature space.

Reviewer comment (Novelty):
The difference/contribution over LLaVA or MAGMA can be more clearly discussed.

Author response:
**[W3, Q3, W4] Difference between VLAP and the previous linear transformation-based approaches.** We included the missing reference (i.e., LLaVA). In addition, we extensively revised the paper to cover the reviewer's concerns about the novelty and to remove potentially misleading terms in Introduction.

Reviewer comment (Experiments):
A better alignment (reducing the gap) in multi-modality is the essential contribution of this work. However, it lacks studies or results, apart from the overall performance, to validate that the gap reduction is achieved by the current predicted assignments rather than the linear layers from previous works.

Author response:
**[W1] Validation for the modality gap reduction**

| $\mathcal{L}\_\text{cap}$ | $\mathcal{L}\_\text{ITM}$ | $\mathcal{L}\_\text{ITC}$ | $\mathcal{L}\_\text{map}$ | CIDEr-D | CLIP-S | Ref-S | $\Delta$ |
|---------------------------|---------------------------|---------------------------|---------------------------|---------|--------|-------|--------------|
| v                         |                           |                           |                           | 51.3    | 72.4   | 76.6  | 0.972        |
| v                         | v                         | v                         |                           | 61.9    | 79.7   | 82.8  | 0.870        |
| v                         | v                         | v                         | v                         | 65.2    | 82.8   | 86.0  | 0.840        |
| v                         |                           |                           | v                         | 69.4    | 87.6   | 92.0  | 0.725        |

To provide a better understanding in terms of modality gap reduction, we measure the modality gap distance $||\Delta||$ according to the learning objective.
The modality gap $\Delta$ is defined as the difference between the center of visual and text representations [1]:

$\begin{equation}
        \Delta = \frac{1}{n}\sum\_{i=1}^{n} \bar{\mathbf{z}}^{v}\_i - \frac{1}{n}\sum\_{i=1}^n \bar{\mathbf{z}}^t\_i,
    \end{equation}$

where $n$ is the total number of image-text pairs, $\bar{\mathbf{z}}\_i^v$, and $\bar{\mathbf{z}}\_i^t$ are the $i$-th averaged visual and text representations.
The default gap distance between CLIP ViT-B/32 and T5$\_\text{Base}$ before training is $||\Delta|| = 1.378$.
The image captioning objective $\mathcal{L}\_\text{cap}$ reduces the modality gap distance to $||\Delta|| = 0.972$ and the additional objectives $\mathcal{L}\_\text{ITM}$ and $\mathcal{L}\_\text{ITC}$ further reduce the gap distance to $||\Delta|| = 0.870$.
The comparison between VLAP with all objectives and with the original objective (i.e., $\mathcal{L}\_\text{cap}+\mathcal{L}\_\text{map}$) shows the ITM and ITC objectives have a limitation to reduce the modality gap.
In addition, since the overall performance is inversely proportional to the modality gap, reducing the gap is an important factor in training VLMs.
We also present UMAP visualization [2] and the additional analysis according to the learning objective in Figure 8 and Appendix. D.2 of the revised paper.

    [1] W. Liang et al., ``Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning," NeurIPS'22
    [2] T. Sainburg et al., ``Parametric UMAP embeddings for representation and semi-supervised learning," Neural Computation'21

Reviewer comment (Evaluation):
The authors mentioned, ``Mapping visual data to LLM’s word embeddings results in learned visual representations that hold a semantic taxonomy of LLMs.'' However, there's a lack of quantitative/qualitative results to validate that this allows visual representations to inherit a semantic taxonomy of LLMs.

Author response:
**[W2] Additional analysis for visual arithmetic**

|         | Building$\rightarrow$Country |       |        | Country$\rightarrow$Capital |      |        | CEO$\rightarrow$Company |       |        | Food$\rightarrow$Country |      |        |
|---------|------------------------------|-------|--------|-----------------------------|------|--------|-------------------------|-------|--------|--------------------------|------|--------|
| Method  | B@1                          | R@5   | CLIP-S | B@1                         | R@5  | CLIP-S | B@1                     | R@5   | CLIP-S | B@1                      | R@5  | CLIP-S |
| ClipCap | 0.003                        | 0.035 | 0.24   | 0.0                         | 0.0  | 0.22   | 0.004                   | 0.005 | 0.18   | 0.0                      | 0.0  | 0.24   |
| ZeroCap | 0.1                          | 0.32  | 0.7    | 0.14                        | 0.32 | 0.68   | 0.1                     | 0.3   | 0.64   | 0.03                     | 0.33 | 0.66   |
| VLAP    | 0.3                          | 0.53  | 0.87   | 0.46                        | 0.62 | 0.89   | 0.25                    | 0.5   | 0.75   | 0.17                     | 0.5  | 0.81   |

To prove the visual representation that inherited the semantic taxonomy of LLM, we analyze that word analogies can often be solved with visual vector arithmetic. 
In addition to existing experiments on visual semantic arithmetic, we supplement an experiment on the visual relation (VR) benchmark [3].
The VR benchmark consists of a total of 320 relations, including building$\rightarrow$country, country$\rightarrow$capital, CEO$\rightarrow$company, food$\rightarrow$country, and leader$\rightarrow$country.
Since the relation of leader$\rightarrow$country includes out-of-date information (e.g. 'Obama'-'USA'), we exclude it from this experiment.
    In the above table, we compare the performance between VLAP, ClipCap [4], and ZeroCap [3] in terms of BLEU-1, Recall@5, and CLIP-Score, following [3].
For fair comparisons, we identically use CLIP ViT-B/32 as the image encoder and GPT-2 as the LLM.
The results show that VLAP outperforms the previous methods by large margins.
In particular, we attain correlations of over 75\% in all relations on the semantic distance-based metric, i.e., CLIP-Score.
We include the additional analysis with visualization in Figure 7 and Appendix. C.2 of the revised paper.

    [3] Y. Tewel et al., ``ZeroCap: Zero-shot image-to-text generation for visual-semantic arithmetic," CVPR'22
    [4] R. Mokady et al., ``ClipCap: Clip prefix for image captioning," arXiv'21

Reviewer comment (Experiments):
The final objectives are influenced by the assignment prediction loss and captioning loss. However, there's a lack of study on these hyperparameters. Also, which part contributes more to the learning remains a question.

Author response:
**[W3] Ablation for the balancing parameters**

We present the ablation results corresponding to the balancing parameters in equation (7) in the general response.
Please refer to them.

Reviewer comment (Writing):
For the probability that the corresponding modality data belongs to each word, $P_{nk}$, what does $P_{nk}^{v}$ in the visual modality signify? Does this ``word'' refer to the single word token in the class label of that visual region?

Author response:
**[W4] Meaning of the probability in equation (5)**

We first explain the probability $\mathbf{P}\_{nk}^{t}$ for the text representation.
Since LLMs provide text representations through sophisticated operations between word embeddings, text representations and word embeddings have high correlations with each other.
Therefore, we can obtain the probability of the $k$-th word attending to the $n$-th text representation without any correspondence between language captions and word embeddings as in equation (5).
Meanwhile, our assignment prediction objective enforces visual and text representations to contain the same information.
Simply put, the probability $\mathbf{P}\_{nk}^{v}$, which represents the $k$-th word attending to the $n$-th visual representation, is learned to be similar to a pre-calculated probability $\mathbf{P}\_{nk}^t$ by predicting the assignment of text data without any image-word correspondence.

Reviewer comment (Writing):
There's a lack of formal definitions for the terms/operations appearing in equations, i.e., $Tr(\cdot)$, $[prefix]$.

Author response:
**[W5] Clarity**

We denote the trace matrix as $\text{Tr}(\cdot)$ in equation (2) and the input text prompt (i.e., "$\texttt{A photo of}$") as $\texttt{[prefix]}$ in equation (6).
We clarify them in the revised paper.

Reviewer comment (Novelty):
The main concern of this work is the methodology is relatively incremental without new concepts or findings. Concept-wise and architecture-wise, it is similar to Asano et al. (2020) Selavi, which performs optimal transport across modalities with similar pipelines. Mathematics using the Sinkhorn clustering Swav as Caron et al. (2020).

Author response:
**[W1, W2] Novelty and ablation for assignment prediction**

Thanks for your valuable comments.
As the reviewer mentioned, our VLAP differs from the previous optimal transport-based (or clustering-based) approaches [1, 2, 3, 4] in the following aspects:

(1) defining a fixed central space instead of a learnable central space (i.e., prototypes) and
(2) defining the transport polytope with the word distribution of the training dataset instead of the equipartition assumption.

The previous works aim to learn visual representations [1, 2] or multimodal representations [3, 4] by \textbf{training the whole networks}, including backbone networks and the central space, in an end-to-end manner.
However, we need to connect two \textbf{frozen} backbone networks using visual features as LLMs input.
In our setup, we empirically found that learnable space is heavily sensitive to the hyperparameters (e.g. $\epsilon$ in equation (4)) and provides poor performance.
Therefore, the conventional optimal transport-based approaches [1, 2, 3, 4] have been limited to be directly applied to our work.

|       |                      |      | $\epsilon$-MSCOCO (CIDEr-D) |      |       |
|-------|----------------------|------|-----------------------------|------|-------|
|       | Method               | 0.1  | 0.05                        | 0.01 | 0.005 |
| (i)   | VLAP w/o word embed. | 20.4 | 43.8                        | 38.9 | -     |
| (ii)  | VLAP w/o word dist.  | 50.7 | 57.9                        | 57.1 | 46.2  |
| (iii) | VLAP                 | 61.8 | 68.0                        | 69.9 | 67.7  |

To verify our claim, we conduct additional experiments for the components in assignment prediction.
As shown in the above table, we evaluate each model for the zero-shot image captioning performance (CIDEr-D) on MSCOCO corresponding to various $\epsilon$ values in equation (4).
In (i) "VLAP w/o word embed.'', we first train VLAP with learnable prototypes as in [1, 2, 3, 4]. 
In this setting, we define 3K prototypes following [1] and apply the equipartition assumption in equation (3).
In (ii) "VLAP w/o word dist.'', we also verify the effectiveness of the word distribution.
In this setting, we use the word embeddings as a fixed central space and perform the optimal transport with the equipartition assumption.
The comparison between (i) and (ii) shows that the word embedding achieves substantial performance improvement, demonstrating the effectiveness of the word embedding.
The comparison between (ii) and (iii) original VLAP shows that the transportation polytope with the word distribution achieves additional performance improvement and provides more robust performance regarding $\epsilon$.

    [1] M. Caron et al., ``Unsupervised learning of visual features by contrasting cluster assignments," NeurIPS'20
    [2] Y. M. Asano et al., ``Self-labeling via simultaneous clustering and representation learning," ICLR'20
    [3] Y. M. Asano et al., ``Labelling unlabelled videos from scratch with multi-modal self-supervision," NeurIPS'20
    [4] J. Duan et al., ``Multi-modal alignment using representation codebook," CVPR'22

Reviewer comment (Experiments):
The main difference lies in 3 parts: word embedding as fixed center space, different distribution assumptions (polytope), and LLM application. The first two are the most interesting part, which will be different from previous Sinkhorn-based work. However, there is no ablation study on these two components, which leads the readers to question whether borrowing existing Selavi and Swav will also work.

Author response:
**[W1, W2] Novelty and ablation for assignment prediction**

Thanks for your valuable comments.
As the reviewer mentioned, our VLAP differs from the previous optimal transport-based (or clustering-based) approaches [1, 2, 3, 4] in the following aspects:

(1) defining a fixed central space instead of a learnable central space (i.e., prototypes) and
(2) defining the transport polytope with the word distribution of the training dataset instead of the equipartition assumption.

The previous works aim to learn visual representations [1, 2] or multimodal representations [3, 4] by \textbf{training the whole networks}, including backbone networks and the central space, in an end-to-end manner.
However, we need to connect two \textbf{frozen} backbone networks using visual features as LLMs input.
In our setup, we empirically found that learnable space is heavily sensitive to the hyperparameters (e.g. $\epsilon$ in equation (4)) and provides poor performance.
Therefore, the conventional optimal transport-based approaches [1, 2, 3, 4] have been limited to be directly applied to our work.

|       |                      |      | $\epsilon$-MSCOCO (CIDEr-D) |      |       |
|-------|----------------------|------|-----------------------------|------|-------|
|       | Method               | 0.1  | 0.05                        | 0.01 | 0.005 |
| (i)   | VLAP w/o word embed. | 20.4 | 43.8                        | 38.9 | -     |
| (ii)  | VLAP w/o word dist.  | 50.7 | 57.9                        | 57.1 | 46.2  |
| (iii) | VLAP                 | 61.8 | 68.0                        | 69.9 | 67.7  |

To verify our claim, we conduct additional experiments for the components in assignment prediction.
As shown in the above table, we evaluate each model for the zero-shot image captioning performance (CIDEr-D) on MSCOCO corresponding to various $\epsilon$ values in equation (4).
In (i) "VLAP w/o word embed.'', we first train VLAP with learnable prototypes as in [1, 2, 3, 4]. 
In this setting, we define 3K prototypes following [1] and apply the equipartition assumption in equation (3).
In (ii) "VLAP w/o word dist.'', we also verify the effectiveness of the word distribution.
In this setting, we use the word embeddings as a fixed central space and perform the optimal transport with the equipartition assumption.
The comparison between (i) and (ii) shows that the word embedding achieves substantial performance improvement, demonstrating the effectiveness of the word embedding.
The comparison between (ii) and (iii) original VLAP shows that the transportation polytope with the word distribution achieves additional performance improvement and provides more robust performance regarding $\epsilon$.

    [1] M. Caron et al., ``Unsupervised learning of visual features by contrasting cluster assignments," NeurIPS'20
    [2] Y. M. Asano et al., ``Self-labeling via simultaneous clustering and representation learning," ICLR'20
    [3] Y. M. Asano et al., ``Labelling unlabelled videos from scratch with multi-modal self-supervision," NeurIPS'20
    [4] J. Duan et al., ``Multi-modal alignment using representation codebook," CVPR'22

Reviewer comment (Experiments):
There is no ablation on different objectives, such as existing next-word prediction on learning visual projection on LLM.

Author response:
**[W3] Ablation for the loss function**

Our image captioning objective is basically defined as the next-word generation prediction, i.e., generating a word from the previous information.
Instead, we provide the results corresponding to the additional objectives for cross-modal alignment in the general responses.
Please refer to them.

Reviewer comment (Experiments):
Comprehensive ablation of w/ and wo/ assignment prediction on the same vision/language backbones is missing.

Author response:
**[W1] Ablation for assignment prediction**

We present the ablation results corresponding to the balancing parameters in equation (7) in the general response. Please refer to that section for more details.
The result without the assignment prediction objective corresponds to $\lambda\_\text{map}:\lambda\_\text{cap}=0:1$.
To briefly sum up, the assignment prediction objective combined with $\mathcal{L}\_\text{cap}$ improves the performance.

Reviewer comment (Experiments):
Comparison with other baselines that are designed for alignment is missing. For example, contrastive alignment in ALBEF, BLIP, and the first-stage alignment by BLIP2 which includes image-text matching, and image-grounded text generation.

Author response:
**[W2] VLAP with other cross-modal alignment**

Thanks for the constructive suggestion.
We present the results with the additional objectives, including the image-text matching (ITM) and image-text contrastive (ITC) objectives, in the general response.
Briefly, those two objectives show less performance improvement than our proposed assignment prediction.

Reviewer comment (Experiments):
In experiments, the pre-training data is CC3M which is too small in terms of scale. Whether this method can be generalized to larger scale is not validated.

Author response:
**[W3] Scale of the pretraining dataset**

Thank you for your valuable suggestion.
As previous works for pretraining VLMs have proven, a larger pretraining dataset should bring better results on zero-shot vision-language tasks.
To verify the impact of the scale of the training dataset on zero-shot performance, we will train VLAP with a large-scale dataset (e.g. CC12M) and will include results in the revised paper.
However, we would like to emphasize that our method mainly focuses on presenting an efficient methodology (i.e., linear transformation, using pretrained encoders, small-scale training) to make VLMs.

Reviewer comment (Experiments):
In Tab1,2,3, when compared with previous works, the vision/language backbone is always different. I wonder if using the same backbones as previous works, will the proposed method still outperform them?

Author response:
**[W4] VLAP with other backbone**

| Method | Vis. Encoder  | Lang. Model | NoCaps-In | NoCaps-Out | NoCaps-Near | NoCaps-All | NoCaps-CLIP-S | NoCaps-Ref-S | MSCOCO-CIDEr-D | MSCOCO-CLIP-S | MSCOCO-Ref-S |
|--------|---------------|-------------|-----------|------------|-------------|------------|---------------|--------------|----------------|---------------|--------------|
| MAGMA  | CLIP RN50x16  | GPT-J       | 30.4      | 43.4       | 36.7        | 38.7       | 74.3          | 78.7         | 47.5           | 75.3          | 79.6         |
| LiMBeR | CLIP RN50x16  | GPT-J       | 34.3      | 48.4       | 41.6        | 43.9       | 74.7          | 79.4         | 54.9           | 76.2          | 80.4         |
| VLAP   | CLIP ViT-B/32 | OPT         | 48.2      | 62.7       | 59.3        | 61.3       | 84.8          | 88.5         | 69.9           | 86.7          | 91.8         |
| VLAP   | CLIP ViT-B/32 | T5          | 48.3      | 62.7       | 59.6        | 61.6       | 85.1          | 88.7         | 69.4           | 87.6          | 92.0         |
| VLAP   | CLIP RN50x16  | GPT-J       | 53.8      | 67.5       | 65.7        | 64.5       | 88.3          | 90.1         | 75.3           | 90.6          | 72.2         |

While we used the transformer-based vision models (i.e., CLIP ViT-B/32 and BEiT) and the relatively small scale of LLMs (i.e., OPT$\_\text{1.3B}$ and T5$\_\text{Base}$) in the main paper, VLAP can be applied to all publicly available vision models and LLMs.
Following MAGMA and LiMBeR, we additionally evaluate VLAP with the CNNs-based vision model (i.e., CLIP RN50x16) and the larger LLM (i.e., GPT-J, which has 6B parameters) for zero-shot image captioning on NoCaps and MSCOCO.
Not surprisingly, VLAP with the larger LLM achieves better performance than with the smaller LLMs, significantly outperforming MAGMA and LiMBeR by large margins across all metrics with the same backbones.
We add the results to Table. 9 in Appendix.

Reviewer comment (Experiments):
The motivation for restricting the learned parameter space to just linear layers is unclear -- it would have been more interesting to see more analysis around different learned parameter space including non-linear layers.

Author response:
**[W1] VLAP with non-linear space**

We would like to emphasize that our method diverges from traditional objectives in linear transformation-based approaches by introducing an alternative objective that avoids direct representation comparisons.
In addition, although our method dramatically boosted the performance of VLMs based on linear transformation, but nothing is specifically designed for this purpose only.
As shown in the general response, our assignment prediction objective outperforms other objectives used in the modular-based methods.
The results suggest that the proposed assignment prediction will contribute to the modular-based methods.
To address the reviewer's suggestion, we are conducting additional experiments for the non-linear space, such as multiple linear layers with activation functions and an adapter, which is a trainable bottleneck module [1].
We will include the analysis in the revised paper.

    [1] J. He et al., ``Towards a unified view of parameter-efficient transfer learning," ICLR'22

Reviewer comment (Experiments):
Can the authors show ablation studies for the L_map and L_cap objectives to develop better understanding of each component?

Author response:
**[Q1] Ablation for the balancing parameters**

Thanks for the question. We present the ablation results corresponding to the balancing parameters in equation (7) in the general response.
Please refer to them.