Paper: IMP: Benchmarking Image Polysemy in Vision-Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The claim for having higher diversity in captions in this work is slightly problematic because diversity is never clearly defined. Based on their method, it seems to refer specifically to whatever euclidean distance can be associated with in sentenceBERT's embedding representation, which I am not entirely sure how to interpret.

Author response:
We thank reviewer $\textcolor{blue}{R4}$ for critical question on MPL2D and diversity maximization using sentenceBERT. We first apologize for the confusion we have made in the paper, and any potential misunderstanding of reviewer's question.

> If the used embeddings correspond to CLIP's normalized inner product space

We indeed used the normalized version for computing further statistics. We appreciate the concern when applying euclidean analysis with normalized inner product space. The aim of this metric is to measure how well each caption is paired with its image. We directly use the image enbedding as the pseudo center of its caption embeddings. This is because we argue that an image can have (potentially) infinitely many captions, which the collected captions are just a small subset of them.

> whatever euclidean distance can be associated with in sentenceBERT's embedding representation

To clarify the clustering process during caption selection we add further details about what is described in appendix A. We first extract the text embedding of all captions (including noisy captions) using sentenceBERT. We then proceed all the text embeddings to UMAP to learn a low-dimentional projection of the text embedding while maximally maintaining the structure of those embeddings in the original high-dimentional space. Noisy captions are also included to have more samples for better UMAP learning. 

After we obtain the UMAP projection, for each caption we select those captions which passed human-evaluation, and performing clustering in the low-dimentional 2D space. This is the commonly used visualization pipeline being used with different encoders and dimentionality reduction techniques. The results are also checked by human-annotators with randomly selected images. For image with more than 100 captions, we can see that each cluster represent different meanings. While for those with fewer captions (still larger than 10), the captions are still selected with maximal potential diversity.

Reviewer comment (Novelty):
While this work certainly provides a useful benchmark for polysemy, I don't think the contributions of this work sufficiently shed light on any new aspects of the problem that the community was already aware of.

Author response:
We thank reviewer $\textcolor{blue}{R4}$ for comments on our contributions.

> I don't think the contributions of this work sufficiently shed light on any new aspects of the problem that the community was already aware of.

To the best of out knowledge there are no works that address image polysemy in-depth. Our aim is to not only make the community aware of the issue, but to also make it possible to be able to quantify to what extent it is a problem. Our results show that polysemy has the potential to heavily degrade retrieval performance, which affects VLM applied to real-world data at large.

Reviewer comment (Evaluation):
The MPL2D metric measures the mean euclidean distance between image and caption embeddings, meant to capture the semantic diversity to justify the aforementioned diversity claims. There may be some concerns regarding this formulation which I would like to give the authors a chance to verify any possible misconceptions on my end: If these are the embeddings used in the CLIP contrastive cosine distance loss, then it should be specified whether they are normalized inner products. The CLIP contrastive loss encodes distances only in the angular difference of the embeddings, so the embedding norms should not contribute to the analysis unless there's a good reason to.

Author response:
We thank reviewer $\textcolor{blue}{R4}$ for critical question on MPL2D and diversity maximization using sentenceBERT. We first apologize for the confusion we have made in the paper, and any potential misunderstanding of reviewer's question.

> If the used embeddings correspond to CLIP's normalized inner product space

We indeed used the normalized version for computing further statistics. We appreciate the concern when applying euclidean analysis with normalized inner product space. The aim of this metric is to measure how well each caption is paired with its image. We directly use the image enbedding as the pseudo center of its caption embeddings. This is because we argue that an image can have (potentially) infinitely many captions, which the collected captions are just a small subset of them.

Reviewer comment (Evaluation):
The MPL2D metric measures the mean euclidean distance between image and caption embeddings, meant to capture the semantic diversity to justify the aforementioned diversity claims. There may be some concerns regarding this formulation which I would like to give the authors a chance to verify any possible misconceptions on my end: If the used embeddings correspond to CLIP's normalized inner product space, then we also have to be careful about applying any sort of euclidean analysis, because the clustering from the learned representation probably lies on the surface of the N-dimensional normalized sphere.

Author response:
We are grateful, and thank reviewer $\textcolor{blue}{R4}$, for the additional comments and suggestions. We are happy to provide further clarification:

> I'm not sure if squared or unsquared euclidean is being used here

For the calculation of MPL2D we have used squared euclidean distance. 

> A more principled approach - intrinsic means and variances

We recognise the importance of using a more principled metric such as geodesic distance [1,2] as suggested by $\textcolor{blue}{R4}$. Since the vector is normalized, we focus on the geodesic distance on a hypersphere and compute the mean and variance. We report the result (CLIP-B/32) in table below:

| Dataset | Geodesics mean| Geodesics var (scaled by $10^3$)|
| -------- | -------- | -----
| COCO     | 1.257       |2.189 |
|FLickr30k | 1.247| 1.418 |
|IMP-5k| 1.314 |2.837 |

We will add this table to the paper and expand it to include the other datasets.

Reviewer comment (Experiments):
Is it possible to verify that there is no dataset leakage in the zero-shot analysis? Based on my understanding of the dataset collection pipeline, there should be existing captions from previous datasets such as MSCOCO and FLICKR30K included in this dataset, with possibly different image pairings. Can we confirm that none of the pre-trained models have been exposed to MSCOCO or FLICKR30k?

Author response:
We thank reviewer $\textcolor{blue}{R4}$ for valuable concern on the data leakage problem.
We would like to clarify first that we did not include captions from MSCOCO or Flickr30k in IMP.
We can only verify that image-caption pair in IMP does not exist in CC3M and CC12M.
We observe that there exists handful cases of very similar captions (but not identical) existing in IMP and MSCOCO, but the image is not identical.
We are not able to test data leakage of whether pre-trained models have been exposed to MSCOCO or Flickr30k, either due to the private dataset, or the massive public dataset which might need a lot of time to do the leakage test.

> it's still important to take note of when leakage might be an issue

We fully agree, and we are glad we could confirm that there is no data leakage between IMP and CC3M or CC12M, nor with MSCOCO or Flickr. If desirable we could explore this further on even larger datasets for the camera-ready version. However, given the uniqueness of our data pipeline we suspect that there will be limited to no leakage even with webscale datasets.

Reviewer comment (Presentation):
I didn't feel like both Figure 1 and Figure 3 needed to exist in the main body of the paper because it feels like they are duplicating information.

Author response:
We thank the reviewer $\textcolor{green}{R3}$ for suggetion on better organizing the main text.
We follow the reviewer's suggestion, now Figure 1 has been moved to appendix so there is no duplicated information.

Reviewer comment (Experiments):
Not entirely clear why the dataset needed to be based on "high-quality stock photography from Unsplash".

Author response:
We thank the reviewer $\textcolor{green}{R3}$ for insightful question on database we select.
We would like to give two main reasons we chose Unsplash:

1.   These images from Unsplash, due to their nature (high resolution, less noise, aesthetic value), are more likely to be used as wallpapers or figures in blogs, forums, news, etc. This allows us to search for the use of this exact image over the internet and see how real humans use the image. On the other hand, images from other datasets are mostly likely to only appear once on the internet.
2.   We would like to make our dataset public, and Unsplash can be openly shared. So researchers can easily use the dataset for experiments or extend the dataset. For each image we have at least 10 perfect-matched image found by google-vision web entity search.

Reviewer comment (Experiments):
Which steps did you take to ensure that none of the models had pretrained on the photos that you used from Unsplash?

Author response:
We thank the reviewer $\textcolor{green}{R3}$ for valuable comment on potential data leakage.
We can confirm that in CC3M and CC12M there is no image-text pair exists from IMP-5k. Since many models either pre-trained on private dataset or massive public dataset, we cannot test the data leakage for all these models. However, since our pipeline can get new image and captions from the web, the leakage problem should be at most a handful number.

Reviewer comment (Experiments):
Why did you use captions that described other images as the "human-authored" versions of the captions? Is CLIP similarity really good enough for this?

Author response:
We thank the reviewer $\textcolor{green}{R3}$ for critical comment on details of our caption gather process. 
While CLIP similarity is not sufficient to find semantically identical images fully, it did enable us to select candidate captions which we could then use within the further pipeline, including human annotation. Due to our pipeline we do finally end up with high quality captions. Our aim with this procedure was to increase the number of conceptual captions in our dataset, i.e., if a caption can be transferred from one image to a similar image then it is presumably less descriptive.

Reviewer comment (Evaluation):
Why is RSUM a meaningful measure to report? It doesn't seem like it gives a better understanding of model performance than the original set-based retrieval measures.

Author response:
We thank the reviewer $\textcolor{green}{R3}$ for valuable comment on evaluation metric. 
We are interested in learning how well the model can retrieve all its corresponding captions, which means the ability to deal with image polysemy. We largely agree with the reviewer that RSUM can not fully reflect this aspect. We have added medr and meanr for more analysis and more insights into the experiments, which is included in the shared response and in Appendix C.

Reviewer comment (Evaluation):
Analysis of this dataset can be improved significantly from what is reported in Table 1. My major concern is in the characterization of polysemy, which is not much beyond “non-descriptive”. For instance, does any kind of image-text correspondence ontology emerge? Perhaps this needs human classification of (a subset) captions into a pre-defined categories, or automatically clustering captions, etc. This is especially important since quantitatively MPL2D cannot distinguish noise from diversity. Finally, I would also like to see further statistics beyond word lengths (word clouds, token/type ratio, etc. that would lead us to have a clearer picture of how this dataset is more diverse than existing ones.

Author response:
## Characterisation of polysemy 

We purposefully do not try to strictly define polysemy as any such definition would quickly exclude valid examples. In essence, image polysemy emerges when considering images in real-world scenarios: each person who observes an image may have a different interpretation, the existence of these different interpretations (or meanings) is what defines image polysemy. Polysemy does not necessarily have to be non-descriptive - however, we found that existing captions for images are mainly descriptive and by adding non-descriptive captions we were able to increase the diversity of our dataset. We do not that descriptive versus non-descriptive is not a binary, but rather a spectrum, and we did not find that any natural ontology emerges from the captions.

Distinguishing between noise from diversity is indeed an open challenge, and MPL2D cannot do this in a straightforward manner indeed. However, both MSCOCO and IMP/5K consist of human annotated or human verified captions, which significantly lowers the noise present in these datasets. For a comparison between MSCOCO and IMP/5K we can thus conclude that differences in MPL2D are due to differences in diversity.

For a large-scale webscraped dataset like Conceptual Captions the MPL2D score is most likely strongly influenced by noise - yet it gives us an understanding of the range of MPL2D and what a possible upper-bound may be.

Reviewer comment (Experiments):
The tasks that perform on this dataset are standard image- and text- retrieval tasks with zero-shot and fine-tuning evaluation. Further, an adaptation of models that address image polysemy does not lead to improved performance (see, e.g., SE models in Table 4). These experiments are a good start but the paper would be stronger with additional tasks that focus on measuring model’s capability in addressing polysemy such as image captioning generation given a “caption sense”.

Author response:
## Experiments and future work

We thank the reviewer $\textcolor{orange}{R2}$ for critical comments on the cross-modal retrieval task, evaluation, and suggestions.

The intuition of having these experiments is given in the shared reply. We would like to answer further regarding:

>   Further, an adaptation of models that address image polysemy does not lead to improved performance (see, e.g., SE models in Table 4).

The SetEmbedding (SE) model uses slot attention to have different slots focusing on different regions of the given input. According to the SetEmbedding paper, it can be treated more as a multi-view approach and can achieve SOTA performance on MSCOCO and Flickr30k. Due to the nature of object-centric learning, slot-attention modules would enable the model to link different combinations of objects in the image to different captions. However, we have performed small-scale experiments of how the SE model learns to utilise the slots. For example, when given four descriptive captions and one conceptual caption, it would learn to link the conceptual caption to the background noise (one slot always appears to be dedicated to 'noise'). Increasing the number of slots can somehow mitigate the problem, but we argue that an image can potentially have infinitely many different meanings. So this would need further modification on the SE model to deal with image polysemy, as well as moving beyond the multi-view focus of current SE models, which is out of the scope of this paper.

>   These experiments are a good start but the paper would be stronger with additional tasks that focus on measuring model’s capability in addressing polysemy such as image captioning generation given a “caption sense”.

We appreciate the reviewers recognition of the benefits our current experiments. The central question in our work is to show that current contrastive learning-based models fail to deal with the polysemy challenge. Whilst testing other types of models may give further insight into the polysemy challenge, it also adds complexity due to the difference in how these models are trained. Retrieval directly maps onto contrastive-learning and the alignment between images and text representations - and we believe that if the model cannot overcome this challenge in image-to-text retrieval then image captioning would suffer a similar fate. Additional "sense" prompts may be used during generation, however, as we cannot determine the set of possible prompts apriori this would leave the polysemy challenge unsolved still.

Reviewer comment (Experiments):
Besides improving the analysis in W1 above, the discussion of image-text datasets can also be further expanded to include additional datasets in the analysis, including RedCaps and SBU captions.

Author response:
## Expand the discussion to additional datasets and Data Collection Details

We thank the reviewer $\textcolor{orange}{R2}$ for valuable suggestions on including more datasets for comparison and more details of diversity and quality control.
We have included SBUcaptions in the table, the analysis is based on the entire SBUcaptions (849k). And due to the nature of RedCaps, doing analysis on only one topic will be biased. Instead we select all topics of year 2020, and perform the analysis. We still need some time to get statistics and analysis for RedCaps, once we finished we will add them to the main text.

Reviewer comment (Experiments):
It seems that the “image polysemy” considered in this paper mainly means: the model should have the ability to match images to both “descriptive” and “conceptual” captions; previous datasets such as COCO contain mainly descriptive captions. However, I am not fully convinced that the collected datasets have more “conceptual” captions than the web image-text data such as CC3M, since many of the captions in the dataset come from web data. The MPL2D also does not indicate that the collected dataset is more “conceptual”.

Author response:
We thank reviewer $\textcolor{red}{R1}$ for highlighting the importance of conceptual captions and their distinction from descriptive captions in our dataset. The reviewer's points raise crucial aspects of our work, which we aim to clarify further in addition to the shared response:

## Definition of "Image Polysemy":

We thank reviewer $\textcolor{red}{R1}$ for valuable comments on "Image Polysemy".

>   It seems that the “image polysemy” considered in this paper mainly means: the model should have the ability to match images to both “descriptive” and “conceptual” captions

It’s correct that in our dataset, we can say captions are "descriptive" or "conceptual". A more structured way to classify captions is possible, and we would like to leave this to be discovered by models instead of human design. 

The concept of image polysemy in our work revolves around the inherent capacity of an image to convey multiple, diverse meanings. This multifaceted nature of images presents a significant challenge for current vision-language models, particularly those based on contrastive learning. 

As noted in our shared reply, the central aim of this paper is not to develop or expect models to match both descriptive and conceptual captions accurately. Instead, we focus on demonstrating that contrastive learning-based models struggle with the presence of multiple meanings expressed across captions.

>   However, I am not fully convinced that the collected datasets have more “conceptual” captions than the web image-text data such as CC3M. 
>
>   The MPL2D also does not indicate that the collected dataset is more “conceptual”.

We would like to clarify that we are not expecting more “conceptual” captions than CC3M. We would expect a mixture of captions with different conceptual levels. The use of MPL2D is to illustrate the level of diversity per dataset instead of more “conceptual”. Moreover, we emphasise that our 5K test set has gone through human annotation, whereas datasets like CC3M have not. Any diversity in our IMP/5K thus comes from polysemy, whereas in CC3M it may still be noise.

Reviewer comment (Experiments):
The only advantage of the dataset seems to be that it has multiple captions per image while CC3M/12M does not. But if the final goal is to teach a model to match an image to descriptive/conceptual captions, then it is not clear why it is necessary to have multiple captions per image for training; as long as CC3M has a lot of conceptual and descriptive captions, then the model can learn to retrieve both types of captions. For example, say there is a dataset A with 1K images each with 5 captions; suppose a dataset B with 5K images each with 1 caption. The captions in A and B are identical. Then I do not see the necessity of training on A if we have dataset B.

Author response:
We thank reviewer $\textcolor{red}{R1}$ for insightful questions on the structure of the dataset.

>   b) the importance of having multiple captions per image

We believe that multiple captions differs from having a larger dataset with single captions per image. We show in the paper that when a model is exposed to multiple interpretations of the exact same image, it fails to recognise the multifaceted nature of visual content. The model could learn to match only one caption to the image if we use a single caption dataset like CC3M. To get a deep understanding of the challenge of image polysemy we believe it is necessary to explicitly model it in the dataset. We see this is a crucial step in advancing vision-language models beyond their current limitations, and beyond the typical paradigm of associating a single image with a single meaning.

Training models on datasets with single captions per image, even if they include both descriptive and conceptual captions, does not provide the same learning opportunity. It does not explicitly teach the model that multiple equally valid captions can describe a single image, an essential aspect of understanding image polysemy.

Moreover, creating a dataset with multiple captions per image allows us to control the experimental variables, thereby bridging datasets like CC3M and MSCOCO. By choosing a multiple-caption style, we facilitate a more balanced and reasonable comparison of how well models can handle diverse challenges not adequately addressed by existing datasets.

In conclusion, the multiple captions per image in our dataset are not merely about increasing the quantity of data but are a strategic decision to advance the study of image polysemy. This approach provides a unique opportunity to understand how vision-language models handle real-world challenges such as image polysemy.

Reviewer comment (Experiments):
It would be better if the paper could illustrate either a) why/how the dataset has more diverse/conceptual captions than CC3M or b) the importance of having multiple captions per image.

Author response:
We thank reviewer $\textcolor{red}{R1}$ for insightful questions on the structure of the dataset.

>   b) the importance of having multiple captions per image

We believe that multiple captions differs from having a larger dataset with single captions per image. We show in the paper that when a model is exposed to multiple interpretations of the exact same image, it fails to recognise the multifaceted nature of visual content. The model could learn to match only one caption to the image if we use a single caption dataset like CC3M. To get a deep understanding of the challenge of image polysemy we believe it is necessary to explicitly model it in the dataset. We see this is a crucial step in advancing vision-language models beyond their current limitations, and beyond the typical paradigm of associating a single image with a single meaning.

Training models on datasets with single captions per image, even if they include both descriptive and conceptual captions, does not provide the same learning opportunity. It does not explicitly teach the model that multiple equally valid captions can describe a single image, an essential aspect of understanding image polysemy.

Moreover, creating a dataset with multiple captions per image allows us to control the experimental variables, thereby bridging datasets like CC3M and MSCOCO. By choosing a multiple-caption style, we facilitate a more balanced and reasonable comparison of how well models can handle diverse challenges not adequately addressed by existing datasets.

In conclusion, the multiple captions per image in our dataset are not merely about increasing the quantity of data but are a strategic decision to advance the study of image polysemy. This approach provides a unique opportunity to understand how vision-language models handle real-world challenges such as image polysemy.

Reviewer comment (Evaluation):
I would expect more analysis and discussions on why and how studying image polysemy could benefit future vision-language models and how to model such a phenomenon. For example, does explicitly modeling image polysemy benefit other tasks that could require high-level conceptual understanding (e.g., understanding actions, events, memes, etc)?

Author response:
We thank reviewer $\textcolor{red}{R1}$ for this insightful question regarding the rationale behind our focus on the retrieval task and the broader implications of studying image polysemy for future vision-language models. We touch upon the former point in the shared response, and expand on the latter in the following.

>   I would expect more analysis and discussions on why and how studying image polysemy could benefit future vision-language models and how to model such a phenomena.

1.  **Increasing Prevalence of Polysemy in Real-World Data:**
    -   As we move towards utilising more real-world data in AI applications – such as in understanding events, memes, news, and social media content – the presence and significance of polysemy become increasingly prominent. Unlike controlled datasets like MSCOCO, real-world data is not bound by predefined usage constraints, leading to a natural occurrence of polysemy.
    -   Recognising and accurately interpreting this polysemy is critical for effectively deploying vision-language models in these real-world scenarios.
2.  **Need for Models to Navigate Polysemy:**
    -   Our research underscores the necessity for vision-language models to handle image polysemy, especially in user-facing applications. While it might be feasible to focus on either descriptive or conceptual interpretations, our findings show that current models often fail to distinguish between these types effectively. This lack of distinction results in a degradation of model performance when faced with polysemous images.
    -   Addressing this challenge is crucial for advancing vision-language learning, ensuring that models are equipped to deal with real-world data's diverse and complex nature.
3.  **Highlighting a Key Challenge in Vision-Language Learning:**
    -   We hope that by demonstrating the limitations of current models in handling image polysemy, our work brings to light a significant issue that has been somewhat overlooked in the field. This challenge is not merely academic but has profound implications for the practical application of AI in diverse and dynamic real-world contexts.
    -   We would like to see our findings serve as a call to action for the research community, emphasising the need to develop more sophisticated models capable of understanding and interpreting the nuanced meanings of images as they are used in everyday life.

Reviewer comment (Evaluation):
For the image-to-text retrieval evaluation, how does this test the model’s ability to handle image polysemy? If I understand it correctly, as each image has 5 matching captions, as long as the model retrieves 1 of the matching captions, then it is counted as correctly retrieved. Then the evaluation protocol does not test whether the model handles polysemy; if for most images, at least 1 of the captions is “descriptive”, then a model that only “understands” descriptive captions will still score high on image-to-text retrieval.

Author response:
We thank the reviewer $\textcolor{red}{R1}$ for raising this critical point about our evaluation methodology. We have added more evaluation using MeanR and MedR in the paper in Appendix C, as mentioned in the shared reply. We agree that if the model finds the descriptive one from five captions, that would be counted, but if this were fully the case, then the recall performance should be much higher than the results shown in Tables 2, 3, and 4. As such the mere presence of diverse captions makes the task more challenging.

Reviewer comment (Experiments):
For comparing the dataset with CC3M/CC12M on MPL2D, why not treat the collected dataset as a single caption dataset (either downsample to one caption per image or just “duplicate” the images)?

Author response:
We thank the reviewer $\textcolor{red}{R1}$ for valuable suggestions on comparing IMP with CC3M on MPL2D. We would like to further add this way of computation to table 1 for fair comparison.


| Dataset   | MPL2D mean | MPL2D std |
| --------- | ---------- | --------- |
| COCO      | 1.175      | 0.038     |
| FLickr30k | 1.168      | 0.031     |
| IMP-5k    | 1.221      | 0.042     |


By comparing the MPL2D scores of IMP and CC3M, we aim to understand how pre-trained models treat the representation of image-text pairs, and MPL2D offers a direct measure of this. A higher MPL2D score, in our context, suggests a richer, more nuanced representation of the image-text pair, aligning with our objective to challenge models with polysemous images.

We also acknowledge that a high MPL2D score could potentially arise from noise within the dataset. For our human annotated test split IMP/5K this is not an issue, however, for other datasets such as CC3M this may indeed explain the higher score. Yet, the high score on this metric for the not-noisy IMP/5K does confirm the notion that this is a diverse set.

Nonetheless, it remains challenging to distinguish noise from diversity, which is why we believe that approaches which try to reduce noise will ultimately not work, instead a more structural approach to image polysemy is needed.

Reviewer comment (Evaluation):
I am not sure about the takeaway message by comparing MPL2D: is higher MPL2D score better or lower score better? On the one hand, if there are more diverse/conceptual captions, the score is higher; on the other hand, if the captions are noisier, the score is also higher. Thus, a lower/higher score could be attributed to these two possible factors and we cannot make a conclusion.

Author response:
By comparing the MPL2D scores of IMP and CC3M, we aim to understand how pre-trained models treat the representation of image-text pairs, and MPL2D offers a direct measure of this. A higher MPL2D score, in our context, suggests a richer, more nuanced representation of the image-text pair, aligning with our objective to challenge models with polysemous images.

We also acknowledge that a high MPL2D score could potentially arise from noise within the dataset. For our human annotated test split IMP/5K this is not an issue, however, for other datasets such as CC3M this may indeed explain the higher score. Yet, the high score on this metric for the not-noisy IMP/5K does confirm the notion that this is a diverse set.

Nonetheless, it remains challenging to distinguish noise from diversity, which is why we believe that approaches which try to reduce noise will ultimately not work, instead a more structural approach to image polysemy is needed.